This post is a more a detailed account of the Liquid hard fork issue and its resolution. Readers can find the original statement from the Liquid Network Oversight Board here.
On Monday, October 4, during a planned hard fork upgrade to enable future deployment of the much-anticipated Dynamic Federations (DynaFed) feature to the Liquid Network, there was an issue with the consensus parameters resulting in functionaries failing to sign blocks. Funds remained unaffected and secure.
The discrepancy in the consensus parameters caused the blocksigner subsystem running on each functionary device to stop producing blocks until functionary operators could manually resolve the issue.
Within hours, Liquid's team of developers diagnosed the issue and implemented, tested, and delivered a patch to functionary operators. Due to the number of functionaries and their distribution around the globe, the patch took some coordination to deploy.
Note: Regular users running Liquid full nodes and other node operators do not need to upgrade Elements node software at this time. The issue only affected functionaries, as only they run the affected blocksigner subsystem.
For security and decentralization reasons, only functionary operators can upgrade nodes. By design, no single entity has remote upgrade capability, and there is no auto-upgrade functionality. The functionary software requires a manual upgrade, meaning a technician must be physically present in front of the hardware.
More centralized systems are easier to coordinate bug fixes and upgrades for, but blockchain security derives from operational decentralization and a defensive shield of many full node verifiers.
At 19:29:58 UTC on Tuesday, approximately 22 hours after the initial issue, the patch had been deployed in a sufficient number of functionaries, block signing started again, and normal operation resumed.
During the downtime, all Liquid transactions were queued in each functionary’s mempool, and have now been successfully settled in blocks.
Diagnosis and Patch
The root cause was determined to be a failing consistency check for the value of the consensus parameter
signblock_witness_limit. In the Liquid production network, this value is hardcoded in Elements and included in the headers of the first DynaFed block.
After DynaFed activation, the blocksigner calculates a value based on the size of the network and expects this value to match the value in the chain headers. The hardcoded value (1416) was inconsistent with the computed value (1325), causing six blocksigners to terminate on startup. The other nine functionaries, running an earlier version of the functionary software, reported errors but did not terminate.
The patch allowed the value of
signblock_witness_limit to be overridden in the production blocksigner configuration, and to set its value to that of the existing Liquid chain.
Monday's hard fork overhauls the block header for Liquid to enable functionaries to broadcast changes in the Federation parameters. The change allows for DynaFed deployment later on, which will introduce a range of improvements at the functionary level to increase network flexibility.
One of DynaFed’s fundamental changes to the Liquid Network is the ability for the Federation to add and remove functionaries without affecting network operations. Currently limited to 15, increasing the number of functionaries will promote decentralization and enable a more resilient sidechain.
See July’s progress report for a more thorough breakdown of what DynaFed brings.
Follow Official Liquid Channels
Transparency remains the highest priority for us, and the Federation's policy will continue to be the full disclosure of any issues relating to Liquid.