Technical Remediation — Massa incidents on October 4–5, 2025

Executive summary

On October 4, Massa experienced a network degradation that briefly self‑recovered; on October 5, a similar pattern escalated into a network‑wide outage driven by heavy forking. The root cause was a subtle interaction between autonomous smart‑contract execution under a rare, loop‑heavy pattern and the operational coupling between Execution, the Transaction Pool, and the Block Factory. A single slot occasionally took up to 60 seconds to execute (target is ≤300 ms), which blocked state access, stalled the pool, and led nodes to produce blocks with stale parents, creating widespread forks. We have deployed performance fixes (including increased gas costs for specific host calls and lock/contention reductions), restored the network, and are finalizing a 4.1 release that hardens inter‑module decoupling to prevent this class of back‑pressure cascade.

Impact and detection

On Saturday, approximately 40% of nodes dropped, CPU usage spiked, slot execution slowed markedly, and the network temporarily forked before stabilizing after restarts. Metrics showed a sharp dip in final slots per second, with a rise in “misses” (empty slots), and a clear correlation with spikes in the asynchronous message pool used by autonomous contracts. Sunday followed the same pattern but failed to recover: nodes repeatedly produced conflicting blocks, consensus was overwhelmed by competing cliques, and the network went down until actively managed recovery. Centralized exchanges and the bridge were impacted but were brought back online once the network was restored.

What happened technically

The investigation (≈50 GB of logs and targeted repos) showed autonomous smart contracts consistent with an on‑chain arbitrage strategy. Under a specific, deep‑loop pattern involving host (“ABI”) calls, slot execution ballooned to ~60 seconds. Execution held a long‑lived write lock on state during these slots. The Transaction Pool, which validates fee solvency by querying Execution, was forced to wait on that locked state. The Block Factory had already selected parents and then waited for the pool to return operations; by the time it received them, its parent choice was ~60 seconds old. Blocks were thus produced with valid contents but stale parents, effectively forking back in time. Because many nodes hit the same lock‑induced delay, they all emitted such blocks at once, amplifying forks until consensus could no longer converge. Massa’s consensus tolerates significant lateness, but this systematic, synchronized, minute‑scale lag across multiple threads exceeded its tolerance envelope.

Why a slot could take 60 seconds

Our original gas calibration used randomized contract generators and regressions on a reference machine with safety margins. It did not sufficiently cover an extreme adversarial‑style loop pattern of specific host calls. In that pattern, three effects interacted badly:

• Atomic locks on shared resources degraded CPU cache behavior and introduced cross‑core contention over many loop iterations.
• Rust HashMap protections (per‑process randomized salting) drained the RNG under heavy, repeated access, causing waits and increasing hashing costs.
• Wasmer/Cranelift gas metering for deterministic execution adds probes; when host calls cross the WASM boundary frequently in tight loops, the metering overhead compounded, producing up to ~200× slowdowns in this narrow case.

Individually, these mechanisms are sound; in combination, and only under this loop‑heavy shape, they created pathological latency.

Remediation applied

We optimized the Execution engine by removing or reducing several atomic locks, eliminating sensitive HashMap access patterns in hot paths, and improving gas access routines. We raised gas costs for selected host ABI calls (≈5×) to ensure slot‑time ceilings are respected even in worst‑case loops; builders using these calls should expect higher gas consumption and adjust accordingly. We also added targeted observability for anomalously long slot execution. These changes have been deployed; nodes, exchanges, and the bridge have restarted, and the network is stable.

Structural hardening (coming in 4.1)

We are finalizing a non‑breaking decoupling upgrade to eliminate the back‑pressure chain between Execution, the Pool, and the Block Factory. The pool will maintain a ready‑to‑read snapshot (double‑buffering) that the Block Factory can consult immediately; when the live pool updates after state checks, it atomically swaps the snapshot. The Block Factory will also stop waiting indefinitely: if the pool does not respond within ~1 second, it will still produce a block (potentially with fewer or no transactions) to avoid emission with stale parents. Additional warnings will fire when production overruns safe timing thresholds, with a configurable option to abstain from producing in extreme cases. This work is being thoroughly reviewed and tested and will ship in 4.1.

What went well and where we’ll improve

Rich telemetry and disciplined snapshot/log collection enabled a precise diagnosis of a genuinely intricate integration failure. Recovery steps brought the network and ecosystem services back online in a controlled way. Going forward, we are expanding stress and adversarial tests to include deep‑looping host‑call patterns, strengthening module isolation to prevent lock cascades across subsystem boundaries, and refining alerts specifically tied to slot latencies and inter‑module timeouts.

Current status and next steps

The network is operating normally on the remediated code. We recommend staying current and preparing to upgrade to 4.1 upon release to benefit from the structural decoupling and added safeguards. Builders should review their use of host calls and re‑assess gas budgets. We will continue to communicate timelines for 4.1 and any further improvements. Thank you for your patience and for the community’s support during an unusually challenging incident.