ExecutionPayloadEnvelopesByRange returns envelopes of payloads the chain did not build on

[nflaig]

Observed with teku/v26.8.0+82-g1ef46b9bb2 on glamsterdam-devnet-9, code references below are at that commit. The by-range handler and CombinedChainDataClient are unchanged on master as of 2026-09-07.

Observed

On glamsterdam-devnet-9 (2026-09-04, Lodestar nodes resyncing from genesis) Teku peers returned envelopes for slots 7392, 8000 and 8288 whose payloads were orphaned (the next block built on the PAYLOAD_STATUS_EMPTY variant). 3 in 380 batches served. For slot 7392 one Teku peer returned the envelope and another did not, so it depends on what the serving node holds, not on a policy. A syncing client that imports such an envelope adds a childless FULL variant that the zero-weight tiebreak picks as head (this parked 19 Lodestar nodes for hours, fixed on the Lodestar side by ChainSafe/lodestar#10005). Spec clarification: ethereum/consensus-specs#5608.

[Teku] 16:42:27.604 batch ready peer=16Uiu2HAmMmYRE6dMsMT1Pu67YFYcxLmbsFKqiYBncm8f4fFEB7Zn, blockSlots=[7392, 7395-7401, ...], envelopeSlots=[7392, 7395-7401, ...]
[Teku] 16:46:58.009 batch ready peer=16Uiu2HAmLiLQoNqEpfqzdBwHqrBRJAMLuGPnfciidL76n9JjyDnV, blockSlots=[7392, 7395-7401, ...], envelopeSlots=[7395-7401, ...]

Cause

The handler resolves each requested slot to the canonical block root and returns whatever envelope is stored under that root. The payload status of the node on the head chain is never consulted.

1. ExecutionPayloadEnvelopesByRangeMessageHandler.onIncomingMessage builds hotRoots with combinedChainDataClient.getAncestorRoots(start, 1, count) → RecentChainData.getAncestorRootsOnHeadChain → ForkChoiceUtil.getAncestors, a walk over blockParentRoot by root. The javadoc of ReadOnlyForkChoiceStrategy.getAncestorNode already warns that a block-root walk "would discard each ancestor's payload status".
2. RequestState.loadNextExecutionPayloadEnvelope calls getExecutionPayloadByBlockRoot(root) for hot slots and getBlockAtSlotExact(slot) + getExecutionPayloadByBlockRoot(block.getRoot()) for finalized slots. The only filter is executionPayload.getSlot().equals(slot).
3. getExecutionPayloadByBlockRoot reads Store.executionPayloads (keyed by block root, filled by StoreTransaction.putExecutionPayload from ForkChoiceUtilGloas.onExecutionPayload for every imported envelope) and falls back to the finalized DB (V4FinalizedKvStoreDao.getBlindedExecutionPayloadEnvelope, column blindedExecutionPayloadEnvelopesByRoot, unblinded via the EL).
4. Persistence has no status filter either: StoreTransactionUpdatesFactory.createBlindedExecutionPayloads() blinds every entry of hotExecutionPayloads and KvStoreDatabase.updateBlindedExecutionPayloads stores them, deleting only envelopes of pruned non-canonical block roots. The orphaned payload of a canonical block is kept for good and served for every finalized-range request.

The import side amplifies it. SyncSourceBatch.validateNewExecutionPayloads only checks the "child is FULL but the envelope is missing" direction (isParentBlockFull), and BatchImporter.handleBlockImportResult imports any envelope present for a block root. ForkChoiceModelGloas.onExecutionPayload then adds a FULL variant for any block that has a base node. A Teku node that receives an orphaned envelope by range imports it, persists it at finalization and serves it on.

Suggested fix

• Hot range: walk the head chain by ForkChoiceNode (getAncestorNode, payload-status aware) instead of by root, and return the envelope only when the node on the chain has PAYLOAD_STATUS_FULL. Same rule for the head slot.
• Finalized range: persist a blinded envelope at finalization only when the finalized chain went through the FULL variant (getBlockData(root, PAYLOAD_STATUS_FULL) on the canonical node, or child.bid.parent_block_hash == payload.block_hash). Then the existing by-root lookup is correct for finalized slots and the DB stops accumulating orphaned payloads. Checking the next canonical block's bid at serve time works too but costs an extra block read per envelope.
• Import (optional): in validateNewExecutionPayloads skip the envelope of block i when block i+1's bid parent_block_hash does not equal the envelope's block_hash, so orphaned envelopes received from other peers are neither imported nor re-served.