Teku rejects its own validator client's proposer preferences for the Gloas fork epoch

[ovidiu-ssv-labs]

Teku rejects its own validator client's proposer preferences for the Gloas fork epoch

Summary

The Teku beacon node rejects proposer preferences when the proposal slot is in the Gloas fork epoch. The error is invalid signature.

The Teku validator client and the Teku beacon node do not agree. The validator client signs the preferences with the Gloas fork version. The beacon node verifies them with the Fulu fork version. As a result, a Teku validator client cannot publish proposer preferences for the first Gloas epoch through its own beacon node.

Preferences for later epochs are correct. The problem occurs only when compute_epoch_at_slot(proposal_slot) == GLOAS_FORK_EPOCH.

Environment

We saw this on a local devnet. We started the devnet with the ethereum-package on Kurtosis.

Component	Version
Teku	teku/v26.8.0+82-g1ef46b9bb2 — image ethpandaops/teku:master@sha256:dd41d279c837b91f3927c1ab6e0e700582db813d8db6015f0358c5e73c9e8ed1
Lodestar	chainsafe/lodestar:v1.47.0@sha256:5593f6e97912d6aa47114f99850c516062902c7eafd502d79ea0acaa3d38ba30
Lighthouse	ethpandaops/lighthouse:unstable@sha256:e05f5c173e4ed01869ed7fba2a7cb977e722faf626c1adb75d9a9586ddaa49f4
Execution layer (all nodes)	ethpandaops/geth:master@sha256:d49060998ab8605a66f9c0614ba507a72954322b2e8c978a57f1201cf151f9d2
Genesis generator	ethpandaops/ethereum-genesis-generator:6.2.1

The devnet had four beacon nodes. Each beacon node had its own execution node. Teku ran its validators in the beacon node process.

Fork schedule, from GET /eth/v1/config/spec on the Teku node:

FULU_FORK_EPOCH     0
FULU_FORK_VERSION   0x70000038
GLOAS_FORK_EPOCH    4
GLOAS_FORK_VERSION  0x80000038

The Gloas fork epoch is 4. The first Gloas slot is 128. MIN_SEED_LOOKAHEAD is 1, SLOTS_PER_EPOCH is 32. All four beacon nodes gave the same fork schedule.

The devnet had 64 validators. It gave the validator indices 16 to 31 to the Teku node. Teku loaded these 16 keys in its own process (Loading 16 validator keys...). There was no separate validator client for Teku.

What happens

In epoch 3, the Teku validator client prepared proposer preferences for epoch 4. It sent them to the Teku beacon node. The beacon node refused all of them.

Teku wrote this to its log:

2026-09-11 10:06:18.292 ERROR - Validator *** Failed to publish proposer preferences for epoch 4
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException:
  Proposer preferences (validator index 31, proposal slot 131,
    dependent root 0xb4a9526aebb274c3e5302eb9ebbf9b3a5847ac98c6c5b95f136bc0da5599d17a): invalid signature;
  Proposer preferences (validator index 21, proposal slot 134,
    dependent root 0xb4a9526aebb274c3e5302eb9ebbf9b3a5847ac98c6c5b95f136bc0da5599d17a): invalid signature;
  Proposer preferences (validator index 23, proposal slot 140,
    dependent root 0xb4a9526aebb274c3e5302eb9ebbf9b3a5847ac98c6c5b95f136bc0da5599d17a): invalid signature

Three points are important:

1. The validators (21, 23 and 31) are in the range 16 to 31. This Teku node owns that range. Teku signed these messages itself. No other client was involved.
2. The proposal slots (131, 134 and 140) are all in epoch 4. Epoch 4 is the Gloas fork epoch.
3. The message Failed to publish proposer preferences for epoch N occurs for epoch 4 only. It does not occur for any later epoch.

The same Teku beacon node accepted proposer preferences for epoch 5 and epoch 6. A different validator client sent those preferences on the same devnet. So the beacon node is correct after the fork epoch.

Our analysis

We think that the validator client and the beacon node select a different fork version. Please correct us if this reading is wrong.

The validator client uses the fork schedule. ForkProvider.java:31-39:

public SafeFuture<ForkInfo> getForkInfo(final UInt64 slot) {
  return genesisDataProvider.getGenesisValidatorsRoot()
      .thenApply(gvr -> new ForkInfo(
          spec.getForkSchedule().getFork(spec.computeEpochAtSlot(slot)), gvr));
}

getForkSchedule().getFork(4) gives the Gloas fork. The signature therefore uses GLOAS_FORK_VERSION.

The beacon node uses a state. ProposerPreferencesGossipValidator.java:77-80 and 152-162 read a checkpoint state. The epoch of that state is proposal_epoch - MIN_SEED_LOOKAHEAD:

final UInt64 lookaheadEpoch = spec.computeEpochAtSlot(proposalSlot).minusMinZero(minSeedLookahead);
...
return recentChainData.retrieveCheckpointState(new Checkpoint(lookaheadEpoch, dependentRoot))

For proposal_epoch = 4 and MIN_SEED_LOOKAHEAD = 1, this state is in epoch 3. Epoch 3 is before the fork. The Fork of that state is the Fulu fork.

SigningRootUtil.java:145-155 then uses that Fork:

final Bytes32 domain =
    spec.getDomain(
        Domain.PROPOSER_PREFERENCES,
        spec.computeEpochAtSlot(slot),   // the proposal epoch — correct
        forkInfo.getFork(),              // the Fork of the epoch-3 state — Fulu
        forkInfo.getGenesisValidatorsRoot());

BeaconStateAccessors.java:361-371 selects fork.getCurrentVersion() because 4 >= FULU_FORK_EPOCH. The result is FULU_FORK_VERSION.

The two fork versions are different. The signature check fails.

Please note one more result of this design. The beacon node finds the state through dependent_root. That state does not change. Therefore we expect the beacon node to refuse an epoch-4 preference also after the fork. We did not test this.

Other clients

We looked at five other implementations. All of them take the fork version from the fork schedule at the proposal epoch. None of them takes it from a state.

Client	Source
Lodestar	packages/state-transition/src/signatureSets/proposerPreferences.ts — config.getDomain(preferences.proposalSlot, DOMAIN_PROPOSER_PREFERENCES)
Lighthouse	consensus/types/src/builder/proposer_preferences.rs:44-61, with the fork from beacon_node/beacon_chain/src/beacon_proposer_cache.rs:224-236
Prysm	beacon-chain/verification/signed_proposer_preferences.go:102-114 — params.Fork(slots.ToEpoch(msg.ProposalSlot))
Nimbus	beacon_chain/gossip_processing/gossip_validation.nim:2261 — dag.forkAtEpoch(preferences.proposal_slot.epoch)
Grandine	helper_functions/src/signing.rs:500-519 — config.version_at_epoch(epoch)

Two of these clients do this on purpose. Lighthouse has this comment:

// Use fork_at_epoch rather than the state's fork, because post-Fulu we may not have
// advanced the state completely into the new epoch.

Grandine gives ProposerPreferences a special signing-root function. That function does not use the state-based get_domain that its other message types use.

Lodestar and Lighthouse verify this signature. They do not accept it without a check. Both run the gossip validation routine before they publish the message.

Specification references

The specification is not clear on this point. We give the references so that you can judge it.

• specs/gloas/beacon-chain.md:564 gives DOMAIN_PROPOSER_PREFERENCES = DomainType('0x0D000000').
• specs/gloas/validator.md:163-186 signs with get_domain(state, DOMAIN_PROPOSER_PREFERENCES, proposal_epoch).
• specs/gloas/p2p-interface.md:1128-1141 verifies with the same call. The state comes from store.block_states[preferences.dependent_root].
• specs/phase0/beacon-chain.md:1538-1548 gives get_domain. It selects state.fork.previous_version or state.fork.current_version.
• specs/gloas/p2p-interface.md:1072-1074 tells proposers to send the preferences in the epoch before the fork.

If you read the pseudocode literally, the anchoring state is before the fork. The fork version is then the Fulu version. This agrees with the Teku beacon node.

However, the reference test test_gossip_proposer_preferences__valid_at_gloas_fork_epoch gives the Gloas version. The test builds its genesis state directly in the Gloas type (tests/core/pyspec/.../helpers/genesis.py:111-134 sets fork.epoch = 0 and current_version = GLOAS). Therefore the test does not go through a fork transition. There is no test for a true Fulu-to-Gloas transition.

We found no discussion of this question in the consensus-specs repository.

How to reproduce

You do not need any other client software.

1. Start a devnet with GLOAS_FORK_EPOCH = N, where N > 0.
2. Use a Teku beacon node. Give it validators, in the same process or in a separate validator client.
3. Wait for epoch N - 1.
4. Read the Teku log. Teku writes Failed to publish proposer preferences for epoch N with the text invalid signature.
5. Wait for epoch N + 1. The message does not occur again.

Questions

1. Do you agree that the validator client and the beacon node use a different fork version?
2. Which fork version do you think is correct for a proposal slot in the fork epoch?
3. If the fork schedule version is correct, can the beacon node use spec.getForkSchedule().getFork(proposal_epoch) in ProposerPreferencesGossipValidator? This would agree with the validator client and with the five other clients.

Related issues

We looked at #10847 and #10798. They are different. #10847 is about rejections after a usual epoch transition. #10798 is an InvalidCheckpointException about a checkpoint slot boundary. This report is about a fork-version difference at the Gloas fork epoch. The error is invalid signature.

Note on the effect

We found this problem during interoperation tests of a distributed validator client. The effect on that client is small. Each operator sends the same message to its own beacon node. If one beacon node refuses the message, the other beacon nodes still accept it and send it on the network. The problem is limited to one epoch.

The effect on a Teku validator client is larger, because that client has no second beacon node. It loses its proposer preferences for the first Gloas epoch.