SRLabs research suggests that security vulnerabilities remain unpatched for many Ethereum blockchain participants for extended periods of time, putting the blockchain ecosystem at risk.
Crypto currencies provide a popular alternative to centralized payment systems, and promise transactions between mutually anonymous parties, often called “trustless” transactions. More specifically, blockchain participants rely on a majority of participants taking rational actions, rather than having to rely on a single banking institution or government. However, the required rational actions seem to extend beyond what many blockchain users are willing to do. In particular, we found early evidence that blockchain participants do not sufficiently patch and hence carry known vulnerabilities.
A month after its release, a critical security patch has not reached a third of Parity Ethereum nodes.
Ethereum is a cryptocurrency that comes to be through a peer-to-peer network. With a market capitalization in excess of USD 19 billion, Ethereum is a highly attractive target for hackers.
Each participant needs a software client to access the Ethereum network: The most common choices are Parity-Ethereum (Parity) and Go-Ethereum (Geth).
Ethereum relies on high availability to prevent double spending.
A hacker who controls more than 51% of the computational power in the network can double spend coins, enriching the hacker and undermining the trust in the ecosystem. If a hacker can crash a large number of nodes, controlling 51% of the network becomes easier. Hence, software crashes are a serious security concern for blockchain nodes (unlike in other pieces of software where the hacker does not usually benefit from a crash).
Hence, denial of service vulnerabilities hold a significantly high severity in cryptocurrency networks. They have the potential to extensively reduce the required computational power for executing a 51% attack and facilitating double-spending. Ethereum has to rely on the node software to be very hard to crash remotely. However, creating perfect software is near impossible and bugs that allow for remote crashes appear from time to time in blockchain clients.
Unpatched Parity Ethereum nodes can be remotely crashed
According to our collected data, only two thirds of nodes have been patched so far.
Shortly after we reported this vulnerability, Parity released a security alert, urging participants to update their nodes.
One month after this alert, we used data from Ethernodes.org to assess the security of the Ethereum node landscape and found that around 40% of all scanned Parity Ethereum nodes, making up 15% of all scanned nodes [1], remained unpatched and thus vulnerable to the mentioned attack [2]. Another patch that released on Mar 2, 2019 reached around 70% of Parity Ethereum nodes, still leaving another 30% outdated.
Figure 1 illustrates that the percentage of vulnerable nodes shrinks slowly over time. For Parity-Ethereum, we measured how many nodes are below version 2.2.10 (Feb 13, 2019, security-critical), and how many are below version 2.3.9 (Apr 2, 2019, not security critical). For comparison, we track Geth-based nodes that did not receive a consensus critical patch (below patch 1.8.21, released Dec 11, 2018). The data emphasizes that patching happens relatively slowly.
7% of active Parity Ethereum nodes have not been patched for nine months.
While using the same data, we also discovered that 7% of the Parity Ethereum nodes announce a version that is vulnerable to a critical consensus vulnerability, which the developers patched on July 5, 2018.
Breaking the backbone of the Ethereum network requires crashing only a handful of nodes. Unfortunately, the data from ethernodes.org does not include whether a node is a miner.
However, we know that currently the vast amount of hashing power concentrates on a few mining pools. Mining pools often share one node to communicate with the Ethereum network, and we can safely assume that those mining pools are very security aware and keep their nodes up-to-date. While this indeed means that the threat of a collapse due to a patch gap is currently low, this centralization carries a risk as well: Breaking the backbone of the Ethereum network requires crashing only a handful of nodes, which is easily doable considering a denial of service vulnerability like the one we found. If the network were more decentralized, it would prevent attacks that do not scale with more nodes from breaking the network.
To resolve this situation, we need more reliable update mechanisms.
It is therefore desirable (and in line with Ethereum core beliefs) to decentralize the hashing power – this decentralization however would only increase security if the new mining nodes would still be security aware. Our research indicates that this might be too much to ask and draws a pessimistic picture for a more decentralized future. Blockchain ecosystems should look for ways to decentralize the power in their network, without running the risk of relying on vulnerable nodes to maintain the infrastructure. One way to achieve this is an automated updating mechanism, which is indeed the way Parity chose for their Ethereum client.
Even if the miner nodes are secure for now, failure to close known vulnerabilities may lead to a collapse of the blockchain ecosystem if and when the hashing power becomes more decentralized. This failure to update could leave the blockchain ecosystem in a more vulnerable state by lowering the barrier for performing a 51% attack.
The Parity Ethereum has an automated update process – but it suffers from high complexity
Parity Ethereum does have an automated update mechanism, suggesting that the patch gap should be much smaller. However, the Parity Ethereum auto-update mechanism relies on multiple smart contracts on the blockchain. This blockchain reliance creates two possible problems:
First, not all users are in sync with the blockchain and hence will not receive information about the latest updates. Before Parity can auto-update for the first time, the client needs to download large parts of the blockchain and stay synchronized to receive future updates. Consequently, during the synchronization process, outdated nodes become vulnerable to attacks and may never catch up if they are only utilized for short durations. Relying on an on-chain smart contract also means that only clients connected to the canonical Ethereum chain get notifications about new updates. Clients connected to smaller, alternative chains that do not have above mentioned contracts on-chain are vulnerable.
Additionally, the update infrastructure is complex, as it involves not only keeping the contract data up-to-date but also incurring additional overhead during the release process. But the Parity maintainers must also ensure that the data those smart contracts point to – downloadable binaries – are always available from all nodes. This makes the automatic update process error-prone.
Complicating matters further, the Parity Ethereum client only downloads updates that are critical when using the default settings. The threshold for marking a patch as critical seems rather high: Although the Parity security alert urges users to install the patch, the mentioned denial of service bug did not make it to the critical release track.
Geth lacks an update process, by design.
The other popular choice of Ethereum client, Geth, does not include an auto-update feature, which appears to be an intentional design decision. Consequently, the patch gap for Geth clients is larger than for Parity clients. According to their announced headers, around 44% of the Geth nodes visible at ethernodes.org were below version v.1.8.20, a security-critical update, released two-month before our measurement.
Lack of basic patch hygiene undermines the security of the Ethereum ecosystem.
The lack of patch hygiene among Ethereum users suggests that more serious vulnerabilities might also survive for days, weeks, or months among a significant number of Ethereum users, putting their own security and the integrity of the Ethereum ecosystem at risk. If a popular client software were to contain a remote code execution vulnerability, the severity of the patch gap would have the most severe consequences.
Ethereum, as a peer-to-peer network, relies on rationally acting participants: The current patch gap violates this core assumption since leaving yourself and others exposed to hacking is not strictly rational.
Global trustless systems rely on software, and software has bugs. It’s our responsibility as blockchain users to immunize these large decentralized ecosystems from against infections by installing security updates as soon as they become available. This requires both more automated patching options and more patching hygiene from blockchain users.
(Whether moving from a world in which a few large banks control financial systems to one in which a few patch-signing keys wield similar power is progress, that is a topic for another day.)
Research by: Vincent Ulitzsch (@vinulium), Stephan Zeisberg (@stze), Martin Saar
For all of the technical details on the DoS-vulnerability and how we found it, check out our parallel blogpost here.
Footnotes
[1] Not all Parity nodes contribute the same computing power to the Ethereum network. In fact, the computing power appears highly concentrated among a small number of nodes. If major nodes are well maintained, the urgency to patch minor nodes is somewhat diminished to prevent a 51% attack.
However, other hacking risks grow due to the concentrated risk.
[2] Ethernodes.org provided the data on Mar 21, 2019, and the version of a node is determined by its announced header. The version of a node is identified based on its announced header. Ethernodes.org leverages the Ethereum network’s discovery protocol to scan the network and discovered almost 11.000 nodes at that time. Please note that Ethernodes.org data might miss nodes that are not constantly online. We assume that the missed nodes are less well-patched on average. Hence, the patch gap estimates are a lower bound. The real patch gap is likely larger.