PRDV151: Bitcoin for Everybody

STEPHAN LIVERA PODCAST: SLP124 JON ATACK – BITCOIN CORE CONTRIBUTION

November 13, 2019

Jon Atack joins me to talk about his experience with open source software development, and to talk about the process of Bitcoin Core development and code review. We talk:

• Becoming a Bitcoin Core contributor
• How to best contribute
• Communication channels for development
• Priorities in Bitcoin development
• Funding for development

 

 

Jon Atack Links:

• Twitter: @jonatack (https://twitter.com/jonatack) 
• Github: @jonatack (https://github.com/jonatack) 
• Bitcoin Core PR Review Club: https://bitcoincore.reviews and https://twitter.com/BitcoinCorePRs
• How to Review Bitcoin Core PRs: https://github.com/jonatack/bitcoin-development/blob/master/how-to-review-bitcoin-core-prs.md
• How to Make Bitcoin Core PRs: https://github.com/jonatack/bitcoin-development/blob/master/how-to-make-bitcoin-core-prs.md
• Bitcoin Core Peer Review process: https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#peer-review
• "On Consensus" by the Internet Engineering Task Force (IETF): https://tools.ietf.org/html/rfc7282

Podcast Transcript:

Stephan Livera: Jon, welcome to the show.

Jon Atack: Hi, Stephan. Thanks for having me. I'm pleased to be here.

Stephan Livera: So, Jon, I had the pleasure of meeting you at the lightning conference recently and I know you are working on Bitcoin core in a part time capacity to do some review and development work. I was keen to discuss with you and help break that down a little bit for the listeners, but let's start with some background on yourself. Where did you come from before all this Bitcoin stuff?

Jon Atack: Right. So I was born in United States, which one can hear, I think with my accent, but I've spent most of my life or traveling and living around Europe. Germany the former Soviet union now France, and I currently spend most of my time between Biarritz in the Southwest of France, Paris, and sometimes San Francisco and New York.

Stephan Livera: Awesome. And tell us a little bit about your career before Bitcoin and software development review, et cetera.

Jon Atack: Sure. Let's see. For the past decade, I've been a freelance software developer doing software engineering and CTO roles for large multinational companies, often in Europe. I studied computer science and physics at Harvey Mudd college in Claremont, California and business at INSEAD in France, which is an international MBA. I speak English, French, German and some Russian computer languages wise. I began with basic a very long time ago when I was nine years old. And fortunately the teacher in my math class took me over to the admin offices of our grade school and there was a terminal printer. There was no screen back then and I learned basic. And then I taught myself assembly language for the 6502 microprocessor and from age 12 to 17, alongside school and athletic activities, I was a professional freelance games creator and programmer. And during my teenage years I wrote three games that were released for Atari and Commodore computers back in the 1980s. So that gives you some idea. I first started Bitcoin back in 2013.

Stephan Livera: And I know you were doing some work in the Ruby ecosystem. Tell us a little bit about that.

Jon Atack: Correct. While I was doing, like after my studies, I did freelancing as I mentioned for large companies and it was very often using Ruby as well as various databases. So that exposed me to open source. We were using entirely open source software, which I think is a very good thing. And the theme of this podcast clearly and around 2013 I began contributing to open source. I saw some issues with a search engine in Ruby and I began contributing to it and very quickly the lead maintainer gave me the commit bit and I became the lead maintainer rather quickly. And that became the leading database search engine in the Ruby ecosystem. It's called ransack and following, which I got my feet wet and I started getting a bit more confidence in how the whole open source process works because it is a social process. And I began contributing to the Ruby on rails with web framework. I became a top 100 contributor out of around six or 7,000 people now who have contributed and as well began co-editing the newsletter This week in Rails. So that really was about three or four years of quite heavy open source contributing. And it was a very interesting way to begin.

Stephan Livera: Yeah, that's great. And I think what we can do for this episode is really break that down in terms of what does open source development look like for those people who came to Bitcoin more from the economics or finance angle. Maybe they're curious to understand a little bit more about how open source development works. What are the conventions, if you will, how does it, how does a new proposal come up? How does it get debated? How did, how does that flow work? So maybe we could just start with a little bit around how do you get started working on Bitcoin core? What are some good resources? What are some good ways to attack that problem?

Jon Atack: Sure. if you don't mind, I'll backtrack a bit and relate my own journey to getting started and then I'll start providing resources for other people. Clearly as a developer what I think about day in and day out is probably of more interest to developers looking to getting involved rather than, but I originally was interested in Bitcoin as well from an investor point of view in the econ finance point of view. So it's been a whole journey. I first heard of Bitcoin in 2013 I was very skeptical and that was a foolish idea, but I didn't have bandwidth for it. Let's say I was busy with freelance missions. I started following Bitcoin more closely in 2016 and fell down the rabbit hole, the story we all know.

Jon Atack: And I began saving money from my freelance missions to begin spending time, hopefully in the future learning to contribute. So I saved money for about two and a half years. And in March of this year, in 2019 I decided to stop accepting missions and to try to apply myself a bit towards Bitcoin as a developer. At the same time, I applied to Chaincode labs. In late February for the residency this summer. And I hadn't done anything yet. And so no surprise, I was told I needed to begin showing some proof of work in this space. At least I got through the first interview though, which I was pleased to have some feedback on that. So within two weeks I began applying myself to Bitcoin core to learn how to contribute, which involved basically spending a lot of time on the github repository where the development work happens, reading the conversations around issues and pull requests for your viewers, I don't know if pull requests is a familiar term?

Stephan Livera: Yeah. So let's break some of that down. So what is what, what, what do we use get hub for? What are some of the common terms that they might need to learn?

Jon Atack: Right. so github is an online version of the, of the good software with a bit of a social aspect to it to facilitate interaction around code development online in an open source fashion. Usually get help can be used in a closed source way, but usually it's open source, which means anyone can, can view, anyone can follow along, anyone can participate within reason if they don't get themselves banned for bad behavior. Like I said, it's a social process and generally discussion takes place on get hub where the code base is stored around what are called issues and pull requests. Issues are issues, problems that people have found in our reporting on topics to discuss and pull requests or proposals to contribute some changes to the code base or to the documentation. These are online. Anyone can see them and participate.

Stephan Livera: Right. And I think to an outsider, this might look really crazy, right? Because they might be coming from a more top down authoritarian view where they think, "Oh, there's this guy in charge". And he says, "everyone just work on this". But open source is a different beast altogether. Can you explain a little bit around why it's different and how, how does how do different things get prioritized?

Jon Atack: That's a really great point. And, and often a source of misunderstanding. People come from corporate backgrounds or job backgrounds where as you said it is, it does happen that way. Whereas open source is, is more of an ad hoc scratch your own itch sort of process where people come in, they have issues, they try to fix them. People can work on what they want when they want. They, they can fade in, they can fade out, everything is possible. Like I said, it's ad hoc and it's, I think that's the way it should be. And I'm fairly confident that almost any open source developer would agree with that. Nobody wants that it to be structured in waterfall and scrum managers come in and start ordering things around and, following issues up on, on issue trackers like Trello or whatever. It's a very free and free wheeling and anyone can participate thing. And I love it. I think that's the way it should be.

Stephan Livera: Right? And again, just playing devil's advocate, right? So again, I'm a fan of open source development myself, but somebody who's an outsider might look at that and think, Oh, but what if something urgent comes up? And we really need to work on this particular thing. And everyone's just putting their hands up and saying, well, I don't report to anyone. This is open source.

Jon Atack: That's true. And it's good to steelman in the process with, with questions like that. In fact, there are a fair number of us, very small number, but there are people who work on it pretty much constantly who are, who are day in, day out present working on the Bitcoin core code base and handling issues and discussing, reviewing pull requests. There are not nearly enough of them and even fewer who are experienced, but there is a small number who are working full time and who are more or less mostly supported by sponsors and funding or even have a full time position to do that. And so these people are very reactive.

Stephan Livera: Awesome. so look, let's break that down a little bit in terms of, as you were mentioning, there's this code base and then part of it is a matter of coming to terms with it or coming to grips with it. What are some good ways for a developer to learn about that and then start contributing changes or potential changes such as a pull request to it?

Jon Atack: Right. Well that highlights a common, I think error that many new people do make in that they come and they want to change things and that would be absolutely, probably the worst step to take is to arrive and start proposing to change things. What is best is to first take, take some time to observe the process, to see how review takes place, to see how people interact, what kinds of things are desired, what kind of review at what kind of time, some kinds of review are desired at certain points in the process and not in others. And to watch how people interact in a positive fashion, giving positive feedback to help construct things in the right direction. The worst possible thing would be to arrive and start saying, Oh, I think we should change this and we should change that and here's my pull requests.

Jon Atack: And unfortunately there are quite a few people who do do that. And don't take the time to learn the social process behind contributing a, I've been compiling since last March, I've been a part time Bitcoin core developer for about eight months now. And I've been compiling what I've been learning and observing into a document, which we can put in the show notes if you're, you're keen to do so. Sure. Yeah. One is how to make Bitcoin core PRs and another one is how to review Bitcoin core PRs. And the second one is the first one people should begin with. And there is a list of other articles that people could begin with by Jimmy Song, John Newbery, Pierre Rochard, Jeremy Rubin and my articles build on those and go deeply into the process as I've been observing it since eight months.

Stephan Livera: So you mentioned taking the time to understand the community and the social aspect of the development. And I've noticed there are different communications channels, if you will. So you've got IRC, you've got the get hub comments itself. There is the Bitcoin and lightning, they've each got a mailing list as well and sometimes just from private emails and conferences and so on. Can you just outline a little bit about, about the communication there and how a person can learn a bit more from monitoring those?

Jon Atack: Absolutely. I would say the main three vectors of communication are IRC, which is internet relay chat, which we can do find for your listeners if needed because it's an old school communications channel that many people aren't aware of nowadays. It's very heavily used for Bitcoin development as well as lightning. There is the git hub where the main conversation happens around specific issues and pull requests. And then there are the mailing lists which people can subscribe to freely. And that is where more in depth discussion takes place. And finally, there are the BIPs, Bitcoin improvement proposals, which are proposals that can be adopted, may or may not be on an ad hoc basis in terms of if people believe that they are valid.

Stephan Livera: And I've, in my own experience, I've noticed it's quite difficult to stay on top of all of these different channels. How do you do it?

Jon Atack: It's crazy. It's a full time job. To be honest, as a part time, I'm a part time Bitcoin core developer who aspires to hopefully one day work on it full time. Depending on if I find the support and funding grants or position doing so because there's not nearly enough people who are working on full time, but really following things along is a bit of a fulltime job. And if you're not on it really pretty much night and day, it's pretty difficult to keep up. There's a lot happening.

Stephan Livera: Yeah. And I've found one good resource and I know you have contributed to that as well as Bitcoin Optech, which is a newsletter which spells out kind of the digest version, the weekly digest of what was important over the last week or so.

Jon Atack: Yes. Bitcoin Optech is a, is a fabulous initiative. I believe was founded, created by John Newberry, a chaincode labs. John, someone who has really, really been helpful to me. I could go back and relate how instrumental he's been in my, in, in what I've been doing in the last this year. And I just, every week I review the newsletter before it is released to make sure that everything is as best as it can be. Bitcoin Optech, I don't know if your viewers are familiar, they provide educational resources about scaling and efficiency and how to best work with Bitcoin for industry. Who sponsors their work. They have the newsletter, but they also have workshops and they have a scalable Bitcoin scaling book and other, other resources online for people to learn. It's a really good project and I believe they're open to ideas from people for taking it to the next level.

Stephan Livera: That's great. I'd definitely, I recommend listeners if you're interested to subscribe to Bitcoin Optech. Let's jump into a little bit more detail on the process around what gets merged in, right? Because it's not just anything that gets merged in and, but I guess maybe if we spoke from a life cycle point of view, let's say somebody has been reviewing, they've been lurking the communications channels for a little bit of time and now they're a little bit more comfortable with how things are going and now they've come up with an idea, improve Bitcoin right now they might typically float that idea and get feedback on IRC or on get hub. Can you talk through that process a little bit? What typically happens there?

Jon Atack: Sure. Would you mind if I mentioned though I think a very important, completely underappreciated point is that way before getting to that point it's better to spend time reviewing and testing.

Stephan Livera: Sure. Yeah.

Jon Atack: The main thing that Bitcoin is lacking is not proposals for improvements. By far the greatest resource constraint and bottleneck is experienced review and testing. Bitcoin longterm Bitcoin core developers cite these as not only the most needed resource, but also the best and most helpful way to begin contributing and learning reputation in the community of developers. The developer community is a very different, very different space from the Crypto Twitter community or the finance econ investment community around Bitcoin, which was one of the things that really surprised me when I started making the shift towards the start of this year. Certain topics that everyone else talks about are absolutely not discussed at all.

Jon Atack: For example, price, you'll almost never hear price discussed in Bitcoin development circles. It's in fact maybe a good way to not be appreciated. It's not that it isn't important. I think this is an interesting point for people to understand because it's not that the developers don't care about price. I'm sure they do, like everyone else does. But I think that if you allow price to take up the bandwidth that it takes up on Twitter and in the rest of people's mind space, I think it represents an obstacle to contributing and following the very indepth discussions. There's so much context, so much history, so many things to learn and follow in terms of Bitcoin core development, enlightening development, that there's just no bandwidth to usefully give to price. Sorry for the deviation. I thought that's something that might your listeners.

Stephan Livera: Show it. Let's let's jump into the review component of that as well. So let's say you are joining, you want to join Bitcoin and you want to be a contributor and you want to stop by reviewing. What does that look like? Are you commenting on different proposals? Are you testing them? Are you providing a concept ACK testing you know, can you spell out what that looks like?

Jon Atack: Sure. I think at a high level, the goal as a newcomer is to try and add value with while being friendly and humble and learning as much as possible. Because no matter what you're going to make missteps, you're going to say stupid things. I still do and I don't think there's an end to that. But a good approach is to not make it about yourself or as many people do, arrive hoping to get a commit in and, and wow, all their friends and local community. But the better approach I think is how can I best serve? It's really challenging because the codebase has a great deal of breadth and complexities and all the technology surrounding it. It's not enough to be a good coder. It's not enough to be a good cryptographer it's not enough without having all the history and the context.

Jon Atack: And so it's important I think as a new person to be aware of everything that you don't know. The long term people have years of experience and there's a deep collective wealth of knowledge that has been pulled up by the community. So you have to keep in mind that some of your new ideas have most likely already been proposed or considered several times in the past. That's why it's important to not just show up in like a bull in a China shop. At least that's my opinion. Also you have to keep in mind that contributor and maintainer resources are limited. There's not many people, so you have to ask for their help carefully and respectfully and to try to give more than you take ideally, and help more than you hinder as you're getting up to speed. So it's important to try to figure things out on your own and respect other people's time.

Jon Atack: This begins with following the IRC, Bitcoin core dev IRC channel, subscribing to the Bitcoin dev mailing lists and following everything on GitHub. And before you jump in, yes, you need to observe the process and the guidelines, read all the documentation, which I'm sure we'll get to later. It's critical to read the documentation in the code base, watch the interactions, get to know who is doing what and how they're giving feedback, and then we can get into the details of providing the review and review. The big picture is more important than its code style and spelling, which I still fall into. I still am tempted to say, Oh, this word is spelled wrong, this code style isn't nice. Oh, I see this little nitpicky thing. And that actually isn't very helpful compared to understanding the big picture, which is much harder and longer to do. So then let's

Stephan Livera: Take that a little bit further then. So an example might be you're providing some feedback on a PR and what would that look like? Would that, you know, what would you normally do as part of that? Would you say, you know pull that branch, test it and then provide feedback on it. Like what, what are some ideas there?

Jon Atack: Okay. Concretely what people do when they, when they review. And that's getting a bit more into the weeds where I was going to stay a bit high level, but the technical specifics are you need to get really good and comfortable with compiling Bitcoin core from source because you're going to do it for every single PR that nearly, that you review unless it's documentation only. And you might want to set up an automated system for compiling and running the unit tests and the functional tests, making sure that it all passes and reproducing everything you'll want to be reproducing, testing issues, and you'll be wanting to do the same for every PR that you review. So you launched that and that can take time. And while it's building and running all the tests, you'll want to start skimming through the code. Reading the comments so far and getting an idea of where in the review process we currently are.

Jon Atack: Is that the beginning, are we in the middle or are we getting towards the close? For example, ACKs depend on where we are. If it's a brand new PR in a fairly critical or large one. This is a bit more into the weeds, but we're talking now about concept ACKs and approach ACKs. ACKs are basically comments that approve a change and ACK means more or less. I've read this, I've reviewed it, I've tested it and I think it should be merged, but there are different kinds of ACKs. If concept ACK approach ACKs and then full stop ACKs which basically means, okay, it's ready to go, but you should give reasoning behind it. You could also disagree with the change and you could NACK it which is NACK. NACking is something as a new contributor that I would strongly avoid doing because it probably means you're lacking context. And I would first question if I understand everything I wouldn't NACK lightly. In fact, I don't think I have without giving very good justification and reasoning. You don't just say NACK, not if you're a new person at least should we go into what concept and approach are?

Stephan Livera: Yeah, sure. Do you want to just outline what they are?

Jon Atack: Sure. concept ACKs are things that are given towards the beginning of review, which means that you agree with the concept of the change, but you haven't confirmed that you've really looked at the code or or tested it or maybe the code isn't ready to be tested. Perhaps the contributor might have said, I'm just looking for conceptX and so that you would provide one or not. It's a valuable signal. It can be a valuable, valuable signal to an author of the PR to let them know that it has merit and is heading in the right direction and approach ACK is quite new. I think that was added in June at the last core dev meeting. Discussions and approach act basically means, I agree with the concept and the way it's being, the direction of how it's being done in the code base.

Stephan Livera: Could you just outline a little bit around this idea of the structure of a branch and the master because I think that's also something that an outsider of might not understand or appreciate as well. So you might have the master branch, which is let's say the main Bitcoin core code and then somebody might be creating their own offshoot if you will. Could you explain what that is? What is that branch and then that is what people are testing when they want to do this review work.

Jon Atack: Sure. The branch, as you said, there's master and then there are branches. There can be branches on the main repository and each individual contributor can create their own branches locally and then push them up when they're proposing changes. When these changes are merged, they become part of the master branch. But the git hub repository also contains release branches, which is most of what people download install actually are released branches for like 0.18, 0.19 which is coming up now and being released. So on the repository you have those master branches that people install the release branches you have the master branch with which once every release is merged into a release branch and which contains all the latest changes. And then you have the PR branches. When someone proposes code, they push up their local branch, they've made a branch of master, they've changed it, they've pushed it up in a pull request, and then reviewers will pull that pull request branch down into their local repository of Bitcoin core and test it locally. It's very important to test things locally and not trust github, don't trust verify. So review takes place locally and not github, tries to encourage review on its web interface, but it's generally discouraged, at least in Bitcoin core to do that. You pull it down in your view at locally.

Stephan Livera: Got it. And another thing just to clarify for our listeners who are not as familiar, can you just spell out what are some of the different roles here? Because we've got developers, we've got people who are doing review, we've got people who might be doing testing and then we've got the maintainer role and then maybe documentation as well. So could you just outline at a high level, what are these different pieces and what do they each do?

Jon Atack: Sure. At a high level you basically have a large number of contributors. You have some people who just report issues and then you have a core of four, and now five maintainers. It used to be four. And then in June there was a fifth one that was added. And maintainers are the people who have the commit bit and they have, they're the only people who have the right to actually merge changes in after peer review into the master branch, which at one point will be merged into a release. So you have five maintainers, you have a, fairly core group of, I would say 10 to 15 daily contributors who are contributing, reviewing, testing on an everyday basis. It's a very small group, surprisingly, and then you have other people who file issues as they come across them. Awesome.

Stephan Livera: And I think something that might be very interesting as well is around conflict when it comes to people not agreeing on what should be going into Bitcoin core. Can you talk a bit about how that is resolved and how that's then dealt with by a maintainer, for example?

Jon Atack: Well, I can't speak for the maintainers because I'm personally not one, but conflict happens. There hasn't been a whole lot of conflict this year. I would say if I had to point out one thing where I was surprised to see a tiny bit, but it certainly nothing compared to the conflicts we saw in 2017. There was, there was a bit of discussion say about introducing rust components into Bitcoin core, which is a new thing, injecting rust. Bitcoin core being in C++ with testing functional testing in Python at the moment. And basically discussion happens on IRC during the Bitcoin core dev meetings there's a weekly meeting or also a biweekly wallet, Bitcoin core wallet meeting as well as the weekly Bitcoin core PR review club meeting, which we should talk about. I believe anyone anyone interested in getting involved. And I lost my train of thought.

Stephan Livera: Oh, that's fine. But yeah, basically what I was getting at is that there's a certain conservatism about how Bitcoin Core is developed and that things aren't necessarily just merged in Willy nilly. Right. It sort of has to have that rough consensus idea. Can you articulate what that is?

Jon Atack: Yes. To be honest, as a developer, I'm a bit surprised how quickly things actually move. I, from the outside, Bitcoin has a reputation of being very slow and glacial in terms of progress. And maybe that's the case for very large changes like taproot and schnorr signatures and things like that, which do take time and rightly so. They should. But at a, at a code base level, things are happening actually rather quickly. If you're not a fulltime reviewer, sometimes things get merged in before as a part time. We've even had a chance to really go through it completely. It's very hard to stay on top and review everything. There's a lot happening, but basically the process is very collegial and there's, there's not that much conflict. Maybe at a very small micro level. Like one person will say, I don't agree with this change.

Jon Atack: And another person might say that, but more or less consensus is come to with the I think it reminds me of the what do you call it? The internet standards committee process of consensus. Are you familiar with that outline?

Stephan Livera: I'm not.

Jon Atack: Sure. So back in the 90s, there was the internet engineering task force and they came together. They brought together the idea, I believe it was Dave Clark back in 92 of rough consensus. And that's where you see the familiar quote, which you've probably seen quote, "We reject Kings, presidents and voting. We believe in rough consensus and running code". And there's, there's the whole basis of a lack of disagreement is more important than agreement. And that rough consensus is achieved when all issues are addressed but not necessarily accommodated.

Jon Atack: So consensus is a path, not necessarily a destination. You won't necessarily achieve it. And in fact it States that 100 people could be four and five people against and it might not be rough consensus or inversely five people could be four and 100 against, but it might still be rough consensus. And there's a document that I think people interested in knowing more about could read, I could send you a link if you like. It's the internet engineering task force, rough consensus guidelines. And I suspect that that's more or less how things are happening.

Stephan Livera: Right. And, and my understanding, as I've heard explained via some other people is that essentially for a change to come through in Bitcoin core, it sort of has to, it has to pretty much be a net win with very few losses in a kind of compatibility or security or otherwise people won't be in favor of that change. What's your perspective on that?

Jon Atack: I think that I completely admitted to mention that the Bitcoin core or pository actually describes the process of coming to consensus on peer review. It's in the root of the repository, a document called contributing.md and there's a section entitled peer review. It describes the whole process of review and for example, I quote from it maintained years reserve the right to weigh the opinions of peer reviewers using a common sense judgment and also may weight based on meritocracy. Those who have demonstrated a deeper commitment and understanding towards the project over time or we have clear domain expertise may naturally have more weight as one would expect in all walks of life. I think that's a pretty good summary. I apologize if I didn't answer your question.

Stephan Livera: No, I think that's that's a totally fair enough. Answer there. What about now the testing component of that? So as I understand there's different types of testing. Can you outline what the, what they are and what's the importance of that?

Jon Atack: Sure. I mean testing is vital and there are never enough people who are testing Bitcoin core releases during the release candidate process, which has just been in the last month or testing issues and pull requests as they come in. When when, when a change or an issue affects consensus, critical code, keep in mind that Bitcoin core's overarching mission is to maintain consensus. Nothing is important is as important as maintaining that and the robustness in censorship resistance So the bar on those sorts of issues is much higher in terms of discussion and peer review requirements because it could be just very costly to the whole community. So those things tend to see many more eyes and need more ACKs during the review process than a documentation change, which can be emerged quite quickly and easily.

Stephan Livera: What about the role of automated testing? Can you tell us a little bit about, what is automated testing and what's the importance of that?

Jon Atack: Automated testing is writing tests that cover that test. A certain area of the code base. It could be a unit test which is at a very micro level or it could be a functional tests which tests more on an end to end basis or a user view basis. There are both in Bitcoin core as well as well as other kinds of Newark testing that is being worked on. Like fuzzing fuzz testing is another way of saying fuzzing or property based testing, which I've been working on getting in along with Chris Stewart. Marco Falko the maintainer and practical Swift have been adding a bit of fuzzing lately and changes improvements to the testing framework. Whether making the coverage wider or better, adding missing coverage or improving the speed of these tests are very welcome in general. In terms of changes from the community.

Jon Atack: A good way to start with contributing when one does. So is to add to improve or add documentation or tests. Test speed is really important because how fast, how long it takes to run all the tests. I think it matters in terms of how many people are actually going to run them and it can be quite long. And so for example, my last commit to Bitcoin core was in fact speeding up a functional test which took 30 seconds to run. It was one of 120 130 tests in the functional test suite. I think it sped it up from around 30 seconds to around eight and those are the sorts of things that we need more of because faster tests means more tests being run by people.

Stephan Livera: Great. Can you tell us a little bit about what property based testing is?

Jon Atack: Oh, property based testing is, that's getting a bit into the weeds but basically your haven't got it top of mind even though I did contribute the contribution to it, but people who are interested in looking at it I encourage them to look into the repository doc, the doc folder with documents or documentation contained doc slash rapid check, rapid check because that's the name of the property based testing library that is being slowly introduced into Bitcoin core and there I added some documentation about that.

Stephan Livera: Right. What is an example of a test that you might run in Bitcoin core and what is it or what sort of era might it be catching?

Jon Atack: Well a unit test could be as small as testing just a one single function or one, one change in the code base at a very small level. Whereas the functional test I think is more easy to grok. Functional tests will test things like if you call an RPC or or wallet function, like say if you call getaddressinfo and it might test that the information corresponds to what should be returned and is actually what correspond what that information should be. Functional testing can be user review point of the codebase. So for example, you might test, if I call get address info, what kind of info am I receiving and is it the correct information?

Stephan Livera: Awesome. Let's talk about review again. You mentioned the weekly PR review. I think John Newbery has been leading this one and running a weekly IRC meeting. Can you tell us a little bit about this process? What is it?

Jon Atack: Yes, indeed. I've been actually quite involved in the review club. John started it in on May first, I believe this year. So the PR review club just celebrated its six months of exciting existence. Basically, the Bitcoin core PR review club proposes weekly meetings on IRC and has its own channel. The club provides questions and study notes to study notes to review and help you understand the PR. That will be the object of next week's meeting. And ideally you should spend time really going through it, reviewing it, going through the notes and questions and logs of the previous meetings as well. Take a deep dive. It can take hours or even days to really get the most from the weekly meeting. It's every Wednesday at 5:00 PM Greenwich time, which depends on your times on when that is for you. And I really encourage anyone who's interested in getting involved in the club to to participate. There are typically five to 10 participants per week, so it's a quite exclusive little little affair, but it's open to anyone and the website. We can put in the show notes if your listeners would be interested,

Stephan Livera: Tell us what that meeting looks like. Do you open it up and just say, okay, here's the pull request. What, what, what's the process then?

Jon Atack: Sure it's ad hoc. People can jump in at any time and there can be several conversations happening simultaneously on IRC, which is one of the wonderful things about it and that's all good, that's welcomed. But generally the host of the week and sometimes it's very often John sometimes it's other people who guest host. I, I hosted two of them last month. We'll open up the discussion by saying, okay, here we are. What did you think of this PR? Did you test it? Did you review? What did you think? Who, who made a review on github of the actual employee request, which we encourage people to do and then the discussion goes forward and we follow the questions. Sometimes we do, sometimes the conversation deviates and that's all good and it's an ad hoc. It's an hour long of ad hoc, simultaneous multithread discussion.

Stephan Livera: Okay, great. I'd also like to talk about the process then of documenting and documentation within Bitcoin core. Can you give us a some high level understanding on what does documentation in Bitcoin core look like?

Jon Atack: Well, there are two main types of documentation. There is pure documentation which in the Bitcoin core repository is in the folder called doc. Most of it is in that folder there, it is a bit spread out and that it's because there's so much of it and there's so many different kinds of information that it tends to be quite spread out over at least 40 30, 40 different files, if I recall correctly. You also have documentation in the test folder and in other places. So you have to really just explore in terms of getting started contributing to Bitcoin core, I would say there are three. The first three that one should look at are contributing, which is in the root, the top level. And then in doc you have the developer notes and productivity. Those are the really the first three they need.

Stephan Livera: And then you should look at the testing documentation. Again, anyone who adds helpful documentation or improves existing documentation, that tends to be well-received in terms of a new contribution. Let's talk a little bit about what directions you would like to see in terms of Bitcoin core code base. Are there any things that are particularly interesting to you?

Jon Atack: There's just so much to do. I would say that my personal priorities are in order. The, the robustness of the of the code base and of Bitcoin core is functioning and maintaining consensus followed very closely by maintaining, improving privacy and censorship resistance. Those would be my one and two. And after that features which are more fun to work on, they're a bit easier. And scalability and then UI, UX, usability for average users. I put them in my, in that order, not for what everyone should do, but that is what my personal priorities are.

Jon Atack: There are many, many projects going on at any one given time right now in Bitcoin core. One of the ones that interests me the most is improving the robustness and privacy of the P2P networking code and the testability. It's, a really hard area to understand and to simulate and to test. I think that we could, there's a lot more work that could be done if we really want to get into it. There's things like better testing tools and frameworks for evaluating how the behavior changes in complex state when people are making changes, their ideas like multiple P2P networks instead of just one. Currently there is a separation ongoing between transaction relay and block relay. That's being done by Suhas and AJ and others working on it. Some of these ideas are from fellow chain code residents like James Chiang who I really enjoyed me getting to know last summer.

Jon Atack: I forgot to mention that I wasn't, I ended up being invited to go to Chaincode for the seminars and met some, a really great, great bunch of people. Chaincode is an amazing place and I was actually mildly depressed after coming home from it because I missed the people and the excitement of being around those ideas and that stimulation. In general we could use a better framework for thinking about privacy systematically simulations for evaluating things like dandelion, P2P improvements, better test coverage. But also things like improving the testing. My property based testing, fuzz testing, improving the modularity of the code base, separating for example, the node from the wallet. There is multi-process work that's ongoing that's very slow and difficult to slowly untangle the various components that were all in one file originally called main dot CPP and to separate them out into modular things.

Jon Atack: Russ Yanofsky at chaincode labs has been working extensively on that and that's in longterm project I think has been doing it for a year or two already. At least there's the assume UTXO. So there's benchmarking work. Those would be both being done by James O'Beirne at Chaincode. There was Carl and Cory Fields have been working on improving builds and reproducibility with guix. There's constant interference from companies like Apple who are trying to make it harder to do that. For example, the new release Apple is forcing users to right click and say they accept Apple is always trying to close their garden down there is Matt who's working on adding Rust components into Bitcoin core along with others. So much going on. Of course, taproot and schnorr, which we had the large review, the BIP review that's just begun.

Jon Atack: That's a two month process with amazingly over 160 people participating, which is fabulous.

Stephan Livera: So let's, let's, let's jump into that a little bit. So can you give us a high level overview? What is the purpose of the Schnorr Taproot review project?

Jon Atack: Right. Well, I think we've often heard on crypto Twitter, people often outside of Bitcoin complain about how slow progress is in things and then they often like to cite taproot and schnorr as examples. Whereas of course these, these need review extensively and maybe there was just a need to organizes some more to organize it more systematically and extensively. And that's a fabulous initiative that's been taken up by I'm sure I'm going to miss a name, but people like Anthony Towns, his nickname is AJ on IRC. Mike Schmidt Steve Lee and others who I'm certainly forgetting and I'm sorry about that.

Jon Atack: They've basically organized this whole eight weeks of review by topic. There's a curriculum and people signed up for it and we've split off into around 10 study groups, maybe even more because there's a 160 people involved now and we meet once a week on a per group basis. And then there's, there are two, one hour question and answer sessions on IRC with people like Pieter Wuille and who I'm mispronouncing. Other experts like Andrew, Poelstra and Anthony Towns. And it's really good to be able to ask questions away and to, and then to go back home and to individually review and then, and then to propose changes if needed. I completely forgot. Gregory Maxwell, he's also involved present. He's been present at the question answer sessions and there's nothing more amazing than seeing people when you get together in the same IRC room.

Jon Atack: Gregory Maxwell and Pieter Wuille and Andrew Poelstra and Anthony towns roofing away on, on taproot and snore. It's an amazing thing.

Stephan Livera: That's awesome. Also, I'm keen to ask you more at a high level, what are some things that you see? Obviously you've got some open source experience even before Bitcoin. What are some unusual or exceptional things about Bitcoin core compared to other open source projects?

Jon Atack: That is actually a very good question. To my mind. One of the most impressive differences in my experience, which was working with web frameworks and databases and search engines mostly is that here we have a code base that is very critical and bugs can be so costly and so there's not only the criticality, but there is also the lack of experienced reviewers and we really could use more. And this leads to an interesting situation where maintaining ACKs has much greater importance than in other projects.

Jon Atack: Other projects, I've seen other projects, often things will not be merged until everything's just right. Test coverage is complete. All the nits have been handled and the linter smashes everything down to, to be just as it should be in Bitcoin core things are often merged in an incomplete state simply because if there are enough review, is there enough review behind something someone could find some things that still should be changed, but then we don't want to lose the ACKs that have already been, the review time that has already been invested in it. It might well be merged with a followup. Okay. As a follow up, we'll handle the missing test coverage or we'll handle the missing suggestions to improve the codebase style or this and that to make it more modular or more readable for developers. We'll try and add documentation. There are lots of little bits and pieces that are often not finished, but review is so critical and so difficult to come by.

Jon Atack: Once there's enough review, things get merged even if they're not complete. And that is quite, quite new to me and I've had to overcome a lot of instincts on that.

Stephan Livera: Right? Yeah. Another one is a, I've seen this acronym BDFL What is a BDFL?

Jon Atack: Ah, a benevolent dictator for life. Yes. I've actually got turned off from open source because many, so many projects out there are led by BDFL as well as programming languages and everything's fine for the best of everything until you find yourself in stark disagreement with the BDFL and you realize that the BDFL always wins. And for example, in the Ruby on rails framework there's for example DHH who was very well known at least in that ecosystem and he can basically override with an iron fist, the entire Ruby on rails core team if they disagree with him, if he feels strongly about something.

Jon Atack: One thing I found really nice about Bitcoin core and that personally was very important to me to becoming involved again in Bitcoin in open source was seeing the lack of centralized BDFL attitude, at least from what I can tell in the Bitcoin core maintainers. Yes, there are only five who have a commit bit, but it really does seem the consensual approach really does seem to be respected and there seems to be, the closer you get into Bitcoin core, I find the more humble people actually are. And I think that's amazing. I find it inspiring that sort of open source humble, how can I serve attitude rather than not, this is the way it's going to be. And I really liked that. I'm really drawn to that ad hoc approach and that's a really important reason along with the activism and principles behind Bitcoin. That's why I'm here. And I would say that many, many people, if there were to be a BDFL or if the ad hoc approach were to be lost, I think a lot of people would leave. At least I would.

Stephan Livera: I love the explanation there. There's a certain ethos around it and it is fair to say that Bitcoin core has a very, there's some history and specialized knowledge required. And that I think leads me into the next topic I was keen to discuss, which is around funding of developer time and developers who are working on this. Now we have some organizations in the space that are doing this so obviously Chaincode labs, Blockstream, I think Xapo is funding you know, AJ for example, there's MIT DCI, there is now Square Crypto as well. What's your view around funding for these developers and reviewers and contributors? What's your take?

Jon Atack: Sure. And to their credit Xapo also hired a, one of the chaincode residents last summer Amiti who I think is working with AJ. So yeah, that's two for Xapo, congrats. You also have grants. You have sources of grant funding and sponsorship. For example, some of the, some of the exchanges like BitMex research has been even bit, Maine has been sponsoring people in the past. I think, OKCoin has done a bit of funding granting and I hope that they will be involved more and more in the future. There's Kraken that's perhaps coming into this, they've hired Pierre Rochard to hopefully develop their open source program. Among other things, I think he's working on the strategic side of things for Kraken, which is a great exchange as you would agree with.

Jon Atack: There, there are some say that there has never been a better time to be funded and sponsored. And I would say that from what I can gather, most of the daily present Bitcoin core people are either working full time on it paid to do so or are funded in ad hoc ways. Receiving grants or sponsorships. Jeremy Rubin, I believe he received something from BitMex research or OKcoin. I'm not sure. I'm certainly mixing people in grants up. Square of crypto is hopefully looking into funding some individual developers, but they need to, they've got a lot going on with the team they've hired and they did of course support BTCPayServer. There's different things afoot. I personally would be interested in finding something like that for myself. I'm relatively new but this is something I'm inspiring to, to move sustainably from part time, a few hours a week to full time.

Stephan Livera: That's great. And do you believe more is needed? What would you like to see in this space?

Jon Atack: Yes, definitely there is more needed. There are not enough people who are present every day reviewing and testing issues. Keep in mind that at any given point of time, Bitcoin core alone has roughly 700 open issues to test 700 and roughly 300 open pull requests. They need to be tested and reviewed and, and all of these take time. Some of them are old and sitting there and rotting or need to be picked up by someone. But that's still a great number of things that need help. So yes, I very much encourage companies, industry who are not yet supporting open source to get behind open source and do their part. As for myself, I, I've been saying to myself that if I do not find funding before I run out of runway although I do do some freelance work, I do some lecturing at universities.

Jon Atack: I've been turning down full time hardcore missions, which is what I was doing for a decade. And just doing little things in order to preserve time for Bitcoin core, which has a very great opportunity cost. But I said to myself, if I run out of runway or just don't find any support, perhaps maybe what should be done is to make an industry organization that goes around and develops it, builds a developer fund from people in the space and angels and organizations, companies and maybe perhaps bring to bear social norms to encourage each other to up the ante and, and participate in being involved. Because if Bitcoin isn't solid and robust, how is that going to work out for these exchanges in different people in the space? And I know that Steve Lee's interested in Miles Suter from Square Crypto. And I think John Newbery and Adam Jonas and many others that are interested, and Pierre Rochard as well, and bringing some social norms to this so that everybody, so it becomes a norm to help and not the exception of course, people who've been, there are some angels who have been heavily solicited like the two chaincode labs, founders and I believe John Pfeffer, who I have not yet personally interacted yet with, but these people had been supporting individually for years.

Jon Atack: Bitcoin and congratulations to them.

Stephan Livera: Yeah, and that's a, that's great to say. And hopefully as a Bitcoin grows, we'll start to see more and more companies realize that they want to be building on a very solid foundation. And part of that involves funding development and education around Bitcoin and particularly Bitcoin core development and contribution, potentially even other Bitcoin whales, right. They should be thinking of it like, Hey, my, I want to protect my investment. I want to help ensure that Bitcoin stays resilient and stays strong.

Jon Atack: Absolutely. There are really good reasons to help Bitcoin protect your investments. Absolutely. visibility for your, for your company. Public relations, having a voice having a more credible voice. I believe not necessarily influencing developers directly because the developers, it's important to let them work on what they feel is important. But developers will probably be more amenable to listening to you and your concerns if you're helping the open source community. It just all makes sense.

Stephan Livera: Excellent. Well, look, I think we're coming close to time, but Jon, if you had any closing comments for the listeners, make sure you just leave them with some parting thoughts on Bitcoin core and development.

Jon Atack: Personally, I'd like to really personally thank John Newbery and Adam Jonas and Chaincode labs for all the help they've given me in the last eight months helping me ramp up to become a Bitcoin core developer. I want to thank the maintainers and long term contributors on Bitcoin core for all that they do. I think that people underestimate how difficult and it can be to be a maintainer. It takes a great deal of patience and generosity and time and and don't underestimate the difficulty of, it's not a power trip at all. And as people might imagine from the outside, it's really humble service and gardening, pulling weeds and trying to make the Bitcoin core garden as healthy as it can be. And I encourage everyone to help Bitcoin core, lightning open source, get involved and do what you can.

Stephan Livera: Fantastic. So look, before we let you go, Jon, make sure you tell the listeners where they can find you if they want to get in touch or if potentially, if one of the listeners would like to fund you, how can they get in touch with you and keep up with what you're doing?

Jon Atack: Thank you. Well, I have a github and a Twitter account and also a Mastodon account. And all three are jonatack. That's my first name and last name. That's my Twitter and my github. And I'll be putting a website up soon online. As a sort of a blog and what I am working on and proposals for funding grants and perhaps a position working full time on Bitcoin core.

Stephan Livera: Excellent. Well, that's gonna do it for today. Thank you very much for joining me, Jon.

Jon Atack: Thank you, Stephan.

---

This resource was provided to Saylor Academy with the permission of Stephan Livera. You can find the original work at https://stephanlivera.com/episode/124/.