In this article, Jameson Lopp delves into further detail in how Bitcoin Core (the reference implementation of Bitcoin) is developed and how changes are made. As you read, consider the following questions. Which parties may propose upgrades? Which parties may block upgrades or changes to code? What other controls exist to stop unwanted or malicious code from entering the Bitcoin Core codebase?
The question of who controls the ability to merge code changes into Bitcoin Core's GitHub repository tends to come up on a recurring basis. This has been cited as a "central point of control" of the Bitcoin protocol by various parties over the years, but I argue that the question itself is a red herring that stems from an authoritarian perspective – this model does not apply to Bitcoin. It's certainly not obvious to a layman as to why that is the case, thus the goal of this article is to explain how Bitcoin Core operates and, at a higher level, how the Bitcoin protocol itself evolves.
Bitcoin Core is a focal point for development of the Bitcoin protocol rather than a point of command and control. If it ceased to exist for any reason, a new focal point would emerge – the technical communications platform upon which it's based (currently the GitHub repository) is a matter of convenience rather than one of definition/project integrity. In fact, we have already seen Bitcoin's focal point for development change platforms and even names!
While there are a handful of GitHub "maintainer" accounts at the organization level that have the ability to merge code into the master branch, this is more of a janitorial function than a position of power. If anyone could merge into master it would very quickly turn into a "too many cooks in the kitchen" scenario. Bitcoin Core follows principles of least privilege that any power bestowed to individuals is easily subverted if it is abused.
Core is transparent about the list that matters: the PGP keys who can sign merge commits.– Peter Todd (@peterktodd) October 4, 2018
The lesson to be learned here is to not trust GitHub! Even Bitcoin Core doesn't know the full list of people who can change the repo, as that extends to probably dozens of GitHub employees.
From an adversarial perspective, GitHub can not be trusted. Any number of GitHub employees could use their administrative privileges to inject code into the repository without consent from the maintainers. But it's unlikely that a GitHub attacker would also be able to compromise the PGP key of a Bitcoin Core maintainer.
Rather than base the integrity of the code off of GitHub accounts, Bitcoin Core has a continuous integration system that performs checks of trusted PGP keys that must sign every merge commit. While these keys are tied to known identities, it's still not safe to assume that it will always be the case – a key could be compromised and we wouldn't know unless the original key owner notified the other maintainers. As such, the commit keys do not provide perfect security either, they just make it more difficult for an attacker to inject arbitrary code.
At time of writing, these are the trusted PGP fingerprints:
These keys are registered to:
Wladimir J. van der Laan <email@example.com>
Pieter Wuille <firstname.lastname@example.org>
Jonas Schnelli <email@example.com>
Marco Falke <firstname.lastname@example.org>
Samuel Dobson <email@example.com>
Does this mean that we are trusting these five people? Not quite. Keys are not a proof of identity – these keys could potentially fall into the hands of other people. What assurances do you really get if you run the verify-commits python script?
Using verify-commits data from bitcoin/contrib/verify-commits
All Tree-SHA512s matched up to 309bf16257b2395ce502017be627186b749ee749
There is a valid path from "HEAD" to 82bcf405f6db1d55b684a1f63a4aabad376cdad7 where all commits are signed!
The verify-commits script is an integrity check that any developer can run on their machine. When executed, it checks the PGP signature on every single merge commit since commit 82bcf405… in December 2015 – over 3,400 merges at time of writing. If the script completes successfully, it tells us that every line of code that has been changed since that point has passed through the Bitcoin Core development process and been "signed off" by someone with a maintainer key. While this is not a bulletproof guarantee that no one has injected malicious code (a maintainer could go rogue or have their keys stolen), it reduces the attack surface for doing so enormously. What are maintainers and how did they attain this role? We'll dig into that a bit later.
The integrity of Bitcoin Core's code must not rely solely upon a handful of cryptographic keys, which is why there are a multitude of other checks in place. There are many layers of security here to provide defense in depth:
Despite all of the technical security measures that are implemented by the Bitcoin Core project, none of them are perfect and any of them can theoretically be compromised. The last line of defense for the integrity of Bitcoin Core's code is the same as any other open source project – constant vigilance. The more eyes that are reviewing Bitcoin Core's code, the less likely that malicious or flawed code will make it into a release.
Bitcoin Core has a lot of testing code. There is an integration test suite that runs against every PR and an extended test suite that runs every night on master.
You can check the code coverage of the tests yourself by:
Alternatively, you can view the coverage report Marco Falke hosts here.
Code Coverage Report
Having such a high level of test coverage means that there is a higher level of certainty that the code functions as intended.
Testing is a big deal when it comes to consensus critical software. For particularly complex changes, developers sometimes perform painstaking mutation testing – that is, they test the tests by purposely breaking the code and seeing if the tests fail as expected. Greg Maxwell gave some insight into this process when he discussed the 0.15 release:
"The test is the test of the software, but what's the test of the test? The software. To test the test, you must break the software". – Greg Maxwell
BitMEX wrote a great article about the ecosystem of Bitcoin implementations. There are over a dozen different Bitcoin compatible implementations, and even more "competing network" implementations. This is the freedom of open source – anyone who is dissatisfied with the efforts of the Bitcoin Core project is free to start their own project. They can do so from scratch or they can fork the Core software.
At time of writing, 96% of reachable Bitcoin nodes are running some version of Bitcoin Core. Why is this the case? How can Bitcoin Core have near-monopoly status over the network of nodes if the effort required to switch to another software implementation is minimal? After all, many other implementations provide RPC APIs that are compatible with, or at least highly similar to Bitcoin Core.
I believe that this is a result of Bitcoin Core being a focal point for development. It has orders of magnitude more developer time and talent backing it, which means that the code produced by the Bitcoin Core project tends to be the most performant, robust, and secure. Node operators don't want to run the second best software when it comes to managing money. Also, given that this is consensus software and the Bitcoin protocol does not – and arguably can not – have a formal specification because no one has the authority to write one, it's somewhat safer to use the focal point implementation because you're more likely to be bug-for-bug compatible with most of the rest of the network. In this sense, the code of the development focal point is the closest thing to a specification that exists.
People who are unfamiliar with the Bitcoin Core development process may look at the project from the outside and consider Core to be a monolithic entity. This is far from the case! There are frequent disagreements between Core contributors and even the most prolific contributors have written plenty of code that has never been merged into the project. If you read the guidelines for contributing you may note that they are fairly loose – the process could be best described as "rough consensus".
Maintainers will take into consideration if a patch is in line with the general principles of the project; meets the minimum standards for inclusion; and will judge the general consensus of contributors.
Who are the Bitcoin Core maintainers? They are contributors who have built up sufficient social capital within the project by making quality contributions over a period of time. When the existing group of maintainers believes that it would be prudent to extend the role to a contributor who has exhibited competence, reliability, and motivation in a certain area, they can grant commit access to that person's GitHub account. The lead maintainer role is for someone who has oversight over all aspects of the project and is responsible for coordinating releases. It has been voluntarily passed along over the years:
Acting as a Bitcoin Core maintainer is often referred to as janitorial work because maintainers don't actually have the power to make decisions that run contrary to the consensus of contributors or of the users. However, the role can be quite taxing due to the extra attention from the ecosystem at large. For example, Gregory Maxwell gave up his maintainer role in 2017 for personal reasons, likely due to the public pressure he experienced during the scaling debate. Wladimir wrote a thoughtful post about the stress of being a Core maintainer and why it was appropriate to remove Gavin's commit access, which upset a lot of people.
Similarly, when Jeff Garzik was removed from the GitHub organization, he and others were upset about it, but he had not contributed to Core in two years. Leaving his GitHub account with write access to the repository was providing no benefit to the project – it was only creating a security risk and violated the principle of least privilege to which Wladimir referred in his post.
Others may look at Core and believe it to be a technocracy or ivory tower that makes it difficult for new entrants to join. But if you speak to contributors, you'll find that's not the case. While only a dozen people have had commit access over the years, hundreds of developers have made contributions. I myself have made a few small contributions; while I don't consider myself a "Core developer" I technically am one. No one can stop you from contributing!
In 2011, as a high school student who didn't understand what a pointer was, the @bitcoincoreorg developer community (especially people like Greg Maxwell, @pwuille, etc) worked with me to make my shitty patches worth merging and made it a great environment to learn in.
– Matt Corallo (@TheBlueMatt) November 18, 2018
In 2016, @TheBlueMatt organised a residency at @ChaincodeLabs. I'd been reading everything about Bitcoin I could lay my hands on but hadn't dared submit a PR. Matt, Alex and Suhas were extraordinarily generous with their time in teaching us about Bitcoin and how to contribute.
– John Newbery (@jfnewbery) November 18, 2018
– Jeff Rade (@jeffrade) November 19, 2018
One of the most difficult things for people to wrap their mind around seems to be that the focal point for Bitcoin development is not simply the structure that is defined by the Bitcoin Core GitHub account. While Bitcoin Core has some structure (it uses centralized communications channels in order to coordinate), the project itself is not subject to being controlled by any of its participants – even those who have escalated privileges on the GitHub repository. While it is technically possible for a maintainer-organized coup to hijack the GitHub repository, censor dissenting developers, and perhaps even maintain the brand name of "Bitcoin Core", the result would be that Bitcoin Core would stop being the development focal point. Developers who disagreed with the actions of the maintainers would simply fork the code and shift their work to a different repository over which the Bitcoin Core maintainers had no administrative privileges.
Even absent a "coup" per se, if a controversial change did somehow make it into Core, some developers would fork the software, remove the controversial change, and make it available for users. You could argue that this is exactly what happened when Amaury Sechet forked Bitcoin Core and removed the Segregated Witness functionality to create Bitcoin ABC. Alternatively, if Core rejects proposed changes that some people want, developers can fork it and add those changes. This has happened many times, such as when:
Forking the code is easy. Shifting the focal point of Bitcoin development is hard – you must convince contributors that their time is better spent contributing to a different project.
I hold allegiance to no man, to no dev team in Bitcoin. My intention is to run the code that I judge best protects my financial sovereignty.– Jameson Lopp (@lopp) March 18, 2017
It's also hard to convince many people that users do not blindly follow Bitcoin Core's changes – this may be a self reinforcing belief, because if users don't participate in the consensus process by staying aware of their options, they are ceding some of their power to developers. However, the power of the users was exercised during the UASF (User Activated Soft Fork) movement of 2017. An anonymous Bitcoin developer using the pseudonym shaolinfry proposed BIP 148, which would force miners to activate Segregated Witness functionality at a block height that would occur near August 1. However, BIP 148 proved to be too controversial to be adopted by Bitcoin Core, so shaolinfry forked Core and made " Bitcoin UASF" software available. This software implementation gained a nontrivial amount of traction and seemed to create sufficient pressure to convince miners to adopt BIP 91 to activate the fork before the BIP 148 deadline.
In my opinion the best Bitcoin Core contributors are those who practice extreme ownership. Case in point – while John Newbery did not write the code that contained this particular consensus bug, he feels responsible for not preventing it from being merged via careful review and for not finding it later while writing test cases.
I am responsible for the CVE-2018-17144 bug. https://t.co/BrPVivM296– John Newbery (@jfnewbery) September 24, 2018
We are all Satoshi.
It can feel daunting to start contributing to Core, though there are plenty of resources available to help aspiring developers. The guidelines for contributing can be found here though you may wish to start off with Jimmy Song's Gentle Introduction to Bitcoin Core Development.
Newcomers may also be interested in joining the Bitcoin Core PR Review Club to see how code reviews are performed.
A specific example may be helpful – while writing this article I encountered difficulties while trying to run the verify-commits.py script on my machine in order to audit the integrity of the GitHub commit history. In order to save future developers from having to deal with these issues, I opened a pull request to improve the documentation. As you can see from the PR history, 4 different developers chimed in with suggestions for how I could improve my pull request. This ranged from using different wiki markup to a simplified bash command to a new parameter that could be used in the verify-commits.py script. I agreed that all of the suggestions made sense, so I incorporated them into my code and pushed an updated version for my pull request. At that point, the developers who were participating in the review acknowledged that they found the PR to be acceptable, and maintainer Marco Falke tagged it for inclusion in the 0.18 release. After several more days went by with no objections from developers, the code was merged into Core by maintainer Samuel Dobson.
As I've argued extensively over the years, it's practically impossible to fully comprehend Bitcoin as a system. The definition (control) of Bitcoin the protocol is like the definition of a language. Languages emerge spontaneously; the consensus over the meaning of words is organic rather than dictated by dictionaries. Much as dictionaries describe the phenomenon of a language rather than define it, so do Bitcoin implementations describe the language of Bitcoin with code. No one is forced to agree with the definition of a given word in a dictionary, neither are they forced to agree with code in a given Bitcoin implementation by running it.
Languages are not governed by democracy and neither is Bitcoin; while you may hear people make references to miners, nodes, developers, or users "voting" there is no such mechanism that can enable a majority vote of any kind to coerce a minority of dissenters into accepting changes with which they disagree. Bitcoin is anarchy – without rulers, but not without rules. The rules are defined and enforced by individual participants on the network.
While changes to the Bitcoin protocol itself are usually made via the Bitcoin Improvement Proposal process, even this is only a recommended best practice and no one can be forced to follow it. It is merely a more formalized way of trying to guide a change through a process of peer review and consensus building.
As difficult as this is to explain and understand, it is a crucial aspect to Bitcoin's antifragility – if there was a single point of control, it would also be a single point of failure that would be exploited by powerful entities that are threatened by Bitcoin's success. Ultimately, each node operator governs themselves by ensuring that no one else on the network is breaking the rules to which they agree. This security model is the foundation for Bitcoin's bottom-up governance.
No one controls Bitcoin.
No one controls the focal point for Bitcoin development.