Proposed block and mevboost timed out

[PanosChtz]

Describe the bug
I've been using nimbus + mevboost for some time and have proposed some blocks with mev normally. However a couple of days ago I noticed that mevboost timed out, fortunately nimbus proposed the block but without mev.
This happened after I had upgraded to nimbus 23.5.1.
v23.5.0 proposed blocks with mev with no isues.

Screenshots
Logs below

Jun 14 22:42:36 validator nimbus_beacon_node[2904275]: WRN 2023-06-14 22:42:36.990+00:00 Could not obtain blinded execution payload header topics="beacval" error="getBlindedExecutionPayload timed out" slot=6663211 validator_index=109040 head=a346460e:6663210

Jun 14 22:42:36 validator nimbus_beacon_node[2904275]: INF 2023-06-14 22:42:36.992+00:00 Payload builder error                      topics="beacval" slot=6663211 head=a346460e:6663210 validator=afe1ab14 err="getBlindedExecutionPayload timed out"

Jun 14 22:42:36 validator mev-boost[1063926]: time="2023-06-14T22:42:36Z" level=info msg="best bid" blockHash=0xdaee9f006ecc1b2442c303abbb0507c2373cc65a5b822fa7b4f95b44b881a2b4 blockNumber=17481214 method=getHeader parentHash=0xe5ff879d961c5d3ea5bfb455c65baebd92de7fa39c7df3e201d90b902cf180b2 pubkey=0xafe1ab147ad6b91cd77a79c4762bdf80a410c35e523a88b23082995b3a81cd633a05b08d987b6899b827ff9f50a30f41 relays=https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com/ slot=6663211 txRoot=0x3297a8df11f50691e7f7362be69739d6ccbf1d1c147b2f42f24a69937cbc7628 value=0.030212656458174002 version=v1.5.0

Jun 14 22:42:36 validator mev-boost[1063926]: time="2023-06-14T22:42:36Z" level=info msg="http: GET /eth/v1/builder/header/6663211/0xe5ff879d961c5d3ea5bfb455c65baebd92de7fa39c7df3e201d90b902cf180b2/0xafe1ab147ad6b91cd77a79c4762bdf80a410c35e523a88b23082995b3a81cd633a05b08d987b6899b827ff9f50a30f41 200" duration=0.142337 method=GET path=/eth/v1/builder/header/6663211/0xe5ff879d961c5d3ea5bfb455c65baebd92de7fa39c7df3e201d90b902cf180b2/0xafe1ab147ad6b91cd77a79c4762bdf80a410c35e523a88b23082995b3a81cd633a05b08d987b6899b827ff9f50a30f41 status=200 version=v1.5.0

Additional context
flashbots/mev-boost#532

[tersec]

nimbus-eth2/beacon_chain/spec/mev/bellatrix_mev.nim

Lines 70 to 72 in 11f5250

	# Spec is 1 second, but mev-boost indirection can induce delay when the relay
	# itself has already consumed the entire second.
	BUILDER_PROPOSAL_DELAY_TOLERANCE* = 1500.milliseconds

defines a getHeader timeout of 1.5 seconds.

nimbus-eth2/beacon_chain/validators/validator_duties.nim

Lines 465 to 483 in 11f5250

	proc getBlindedExecutionPayload[
	EPH: bellatrix.ExecutionPayloadHeader | capella.ExecutionPayloadHeader](
	node: BeaconNode, slot: Slot, executionBlockRoot: Eth2Digest,
	pubkey: ValidatorPubKey): Future[BlindedBlockResult[EPH]] {.async.} =
	if node.payloadBuilderRestClient.isNil:
	return err "getBlindedExecutionPayload: nil REST client"
	when EPH is capella.ExecutionPayloadHeader:
	let blindedHeader = awaitWithTimeout(
	node.payloadBuilderRestClient.getHeaderCapella(
	slot, executionBlockRoot, pubkey),
	BUILDER_PROPOSAL_DELAY_TOLERANCE):
	return err "Timeout when obtaining Capella blinded header from builder"
	elif EPH is bellatrix.ExecutionPayloadHeader:
	let blindedHeader = awaitWithTimeout(
	node.payloadBuilderRestClient.getHeaderBellatrix(
	slot, executionBlockRoot, pubkey),
	BUILDER_PROPOSAL_DELAY_TOLERANCE):
	return err "Timeout when obtaining Bellatrix blinded header from builder"

then uses this 1.5 second time out.

This 1.5 second timeout is longer than the official 1 second timeout, but eventually, as https://github.com/ethereum/builder-specs/blob/v0.3.0/specs/bellatrix/validator.md#relation-to-local-block-building notes

Honest proposers using the external builder network will give the builders a duration of BUILDER_PROPOSAL_DELAY_TOLERANCE to provide a SignedBuilderBid before the external builder is considered to have hit the deadline and the external builder flow must be aborted in favor of a local build process.

According to one client I checked logs of, 2023-06-14 22:42:35 is the timestamp of slot 6663211, and

WRN 2023-06-14 22:42:36.990 (timeout happened)

shows up in your logs, which is almost two seconds after the slot began. Thus, it appeared to exactly aborted the external builder flow in favor of a local build process according to the builder specs.