Avalanche Brought Down by Bug Triggered by Unusually High Volume, Engineer Says

An Ava Labs engineer gave a rundown of the small code bug that severely crippled the Avalanche blockchain earlier this week. 

In a Sunday Medium post, blockchain engineer Patrick O’Grady wrote that increased congestion on the network triggered a “non-deterministic bug” related to how the high-throughput, proof-of-stake blockchain keeps track of transactions. 

Funds were never at risk, O’Grady notes, though the high-profile misstep has a valuable lesson for the blockchain industry.

Avalanche launched in September 2020 with the claim it could process 4,500 transactions per second. It’s backed by prominent cryptocurrency firms including Mike Novogratz’s Galaxy Digital, Bitmain and Initialized Capital. It also has an academic stamp of approval, having been designed by Emin Gün Sirer, a computer science professor at Cornell University.

The blockchain is usually grouped with other so-called “Ethereum killers,” or blockchains designed to solve the scalability problems that have plagued the second-largest blockchain since inception. While positioned to steal market share from Ethereum, Avalanche also has been billed as a way to complement and connect – rather than strictly compete  – with its forbear. 

Avalanche has three “default chains,” including the so-called “contract chain” that supports the Ethereum Virtual Machine and its Solidity coding language. It’s this chain that was part of this week’s issue. 

You can read a full accounting of the problem that arose here. But in short, in order to boost transaction throughput, Avalanche’s three chains remain separate and distinct from each other, each performing within a set range of transaction-types, up until the moment an asset has to hop over to another chain. That process was placed under an incredible strain, following the launch of a new decentralized money market called Pangolin. 

An atypical amount of users and volume created an atypical amount of blocks to be processed. This, O’Grady notes, triggered a bug that was creating false cross-chain “mints.” In O’Grady’s words: “This caused some validators to accept some invalid mint transactions, while the rest of the network refused to honor these transactions and stalled the [contract]-chain.”

Importantly, no double-spends occurred. “The bug did not affect regular transactions, coin transfers, asset transfers, coin destruction, or smart contract invocations. Avalanche never allowed any user to successfully send the same funds to two recipients,” O’Grady wrote.

A read of the issue was ready just hours after the initial issue, though a fix was harder to come by. Given Avalanche’s decentralized nature, it would be impossible to get all the nodes to collude and rollback problematic transactions. 

Instead, as O’Grady writes, a solution was found through incremental deployment of a patch – basically the way any software is updated. 

Blockchains are complex things, built by human beings, but run by machines. An issue that was small enough to bypass during an initial inspection can snowball as a network grows. In Avalanche’s case, the bug didn’t bring down the network but it did pour ice water over some of the boasts made about the network’s ability to handle high-throughput prior to launch.

AVAX, the blockchain’s token, is trading hands at around $41.20, down from $53 on Feb. 11 when the problem occured.