Ethereum Prysm Client Experiences Mainnet Incident: Resource Depletion Leads to Massive Block and Attestation Loss

Prysm team released a report on the mainnet incident, stating that on December 4th, during the Fusaka period of the Ethereum mainnet, almost all Prysm beacon nodes experienced resource depletion when processing specific attestations, resulting in the inability to respond to validator requests in a timely manner, leading to a significant loss of blocks and attestations. The incident affected epochs 411439 to 411480, totaling 42 epochs, with 248 blocks missing out of 1344 slots, a loss rate of approximately 18.5%; network participation rate dropped to 75% at one point, with validators losing around 382 ETH in attestation rewards. The root cause was Prysm receiving attestations from nodes that may have been out of sync with the mainnet, referencing the block root of the previous epoch. To verify their legitimacy, Prysm repeatedly replayed old epoch states and executed costly epoch transitions, causing nodes to experience resource depletion under high concurrency. The related defect originated from Prysm PR 15965, deployed to the testnet a month ago but did not trigger the same scenario. The official temporary solution is to enable the --disable-last-epoch-target parameter in version v7.0.0; subsequent releases v7.0.1 and v7.1.0 include long-term fixes by verifying attestations using head state to avoid replaying historical states. Prysm stated that the issue gradually eased after 4:45 UTC on December 4th, with network participation rate returning to over 95% by epoch 411480. The Prysm team emphasized the importance of client diversity, highlighting the risk of temporary inability to finalize if a single client accounts for over one-third, and the risk of invalid chains if over two-thirds are dominated by a single client. They also reflected on unclear communication of feature switches and the inability of the testing environment to simulate large-scale out-of-sync nodes, promising to improve testing strategies and configuration management in the future.

Related Articles