Need to upgrade FC module to handle very long branch or non finalized branch

[jangko]

This feature is needed for #2995

• Algorithm changes in FC module
• Possibly need to serialize aristo txFrame
• Requires aristo: fork support via layers/txframes #2960

[jangko]

Copied from discord discussion:

basically, the first step that is needed is to keep track of the latest arrived fcu in forkedchain .. there are two cases:
we know the header -> we can store the "latest finalized block number" and use that in logic surrounding base updates and other things
we don't know the header -> we store the hash in a "pendingFcu" field - then, when we receive any header (either via newpayload or from the sync), can simply inform FC about it and FC can check if it's in the "pendingFCu" entry and promote it to to latestFinalizedNumber, then continue the above logic

in particular, there's no "long async queue" involved here.
once that's done, we can start reasoning about storing the "header chain" directly in the database instead of using the intermediate table that is used now
also - cc @mjfh - we need to get rid of those out-of-band persist calls - the syncer must not interfere with the base flow - if need be, it can even have its own column family which is a lot easier now the backend has been cleaned of excessive abstraction

the other critical change needed to FC is to remove all writes to baseTxFrame essentially, such as here:

nimbus-eth1/execution_chain/core/chain/forked_chain.nim

Line 333 in 9244cc9

c.baseTxFrame.setHead(head.branch.headHeader,

that head update, it should not be written to the base txframe but rather to the txframe of the block being selected has head - same for finalized etc, they should always go into the same frame as the block that gets selected so that when we write the base to disk, we don't accidentally write a head hash that is not yet on disk

this "lastestFinalizedNumber" is suffcient to implement the correct "base update" logic after importBlock, ie block arrives, we check against latest known finalized number, and if the finalized number is far in the future, we can update the base (with some lag - ie the base "target" becomes roundDown32(min(max(head-distance, lastFinalizedNumber)) where head is the latest block we imported, lastestFinalizedNumber is the current finality point according to the CL (ie it can be both higher and lower than head depending on whether we're in sync or not)
the syncer has the right idea here, ie it does something similar - it's just implemented in the wrong place, ie

nimbus-eth1/execution_chain/core/chain/forked_chain/chain_header_cache.nim

Line 83 in 9244cc9

finNumber: BlockNumber # number of final block

[arnetheduck]

Regarding handling of block bodies/headers in non-final branches, after the above changes have been done, the steps are:

• introduce a list of heads - these are the head blocks of each known chain branch, one of which is the canonical head - this is analogous to to ChainDagRef.heads in eth2 and can be kept updated the same way
• store this list of heads in the database and load it on startup, together with all other "candidate chains" (this step does not exist in eth2, but should eventually be added)
• store blocks that are parents of heads in the database - right now, we only store blocks earlier than base, but if we store the heads we can also store head ancestors
• when pruning the list of heads as finality happens, also prune the blocks in the database

Likely, we will keep the dag outline -(hash, parent,number) tuples - in memory for all non-final blocks - the rest can be persisted eagerly

[jangko]

Regarding handling of block bodies/headers in non-final branches, after the above changes have been done, the steps are:

We have separate issue open for this #2888