immutableValidators.len >= numValidators AssertionDefect from ncli_db

[jakubgs]

Describe the bug
ERA files generation failed on Mainnet with following error:

jakubgs@erigon-02.ih-eu-mda1.nimbus.mainnet:~ % j -u nimbus-era-files-update -o cat -n 18 | cat
Starting Update Nimbus beacon node ERA files...
.../ncli/ncli_db.nim(1225) ncli_db
.../ncli/ncli_db.nim(567) _ZN7ncli_db12cmdExportEraEN7ncli_db6DbConfEN7presets13RuntimeConfigE.constprop.0
.../beacon_chain/consensus_object_pools/blockchain_dag.nim(1113) _ZN14blockchain_dag4initE8typeDescI3refIN17block_pools_types27ChainDAGRefcolonObjectType_EEEN7presets13RuntimeConfigE3refIN15beacon_chain_db29BeaconChainDBcolonObjectType_EE3refIN17validator_monitor16ValidatorMonitorEE3setIN6extras10UpdateFlagEE6string4procIN5forks30ForkedTrustedSignedBeaconBlockEE4procIN17block_pools_types20HeadChangeInfoObjectEE4procIN17block_pools_types15ReorgInfoObjectEE4procI3refIN17block_pools_types27ChainDAGRefcolonObjectType_EEN17block_pools_types22FinalizationInfoObjectEEN11vanity_logs10VanityLogsEN30block_pools_types_light_client21LightClientDataConfigE.constprop.0
.../beacon_chain/consensus_object_pools/blockchain_dag.nim(753) _ZN14blockchain_dag13getState_s256E3refIN15beacon_chain_db29BeaconChainDBcolonObjectType_EEN7presets13RuntimeConfigE7MDigestIE5SliceIN9constants4SlotEE3varIN5forks23ForkedHashedBeaconStateEE4procIE
.../beacon_chain/beacon_chain_db.nim(1193) _ZN15beacon_chain_db13getState_s256E3refIN15beacon_chain_db29BeaconChainDBcolonObjectType_EEN5forks13ConsensusForkE7MDigestIE3varIN5forks23ForkedHashedBeaconStateEE4procIE
.../beacon_chain/beacon_chain_db.nim(1178) _ZN15beacon_chain_db13getState_s256E3refIN15beacon_chain_db29BeaconChainDBcolonObjectType_EE7MDigestIE3varIN7electra11BeaconStateEE4procIE.isra.0
.../beacon_chain/beacon_chain_db.nim(1117) _ZN15beacon_chain_db29getStateOnlyMutableValidatorsE9openArrayIN4base23ImmutableValidatorData2EE3refIN7kvstore26KvStoreRefcolonObjectType_EE9openArrayI5uInt8E3varIN7electra11BeaconStateEE4procIE
.../vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(41) _ZN10assertions16failedAssertImplE6string
.../vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(36) _ZN10assertions11raiseAssertE6string
.../vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) _ZN10assertions8sysFatalE8typeDescIN10exceptions15AssertionDefectEE6string
.../vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(349) _ZN6system18rawWriteStackTraceE3varI3seqIN6system15StackTraceEntryEEE
.../vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Error: unhandled exception: .../beacon_chain/beacon_chain_db.nim(1116, 5) `immutableValidators.len >= numValidators`  [AssertionDefect]
nimbus-era-files-update.service: Main process exited, code=exited, status=1/FAILURE
nimbus-era-files-update.service: Failed with result 'exit-code'.

To Reproduce
Steps to reproduce the behavior:

1. Platform details (OS, architecture): Ubuntu 22.04
2. Branch/commit used: testing / a946d61
3. Commands being executed: ncli_db --network=mainnet exportEra --db=/data/beacon-node-mainnet-testing/data/db
4. Relevant log lines: beacon_chain_db.nim(1116, 5) immutableValidators.len >= numValidators [AssertionDefect]

Additional context
Interestingly no such failure was recorded on other Mainnet hosts. For example the same commit was used on erigon-01.ih-eu-mda1.nimbus.mainnet but it completed successfully.

[arnetheduck]

Was the beacon node running while ncli_db was doing its thing?

[jakubgs]

Yes, we do not stop the BN while generating ERA files currently:
https://github.com/status-im/infra-nimbus/blob/23523445b2d5d6b9f90f7ff0b3c41212bde08680/ansible/roles/nimbus-era-files/tasks/linux.yml#L2-L18