Transaction gossip inconsistency during high traffic

[deepthiskumar]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

During high traffic such as staking payouts, nodes drop transactions requiring senders to retry multiple times

Steps to Reproduce

Create simple value transfer txns once in 3-5 seconds in total # of txn to be ~150-200

Expected Result

Transactions should be included in blocks

Actual Result

Some transactions get dropped

Daemon version

How frequently do you see this issue?

Frequently

What is the impact of this issue on your ability to run a node?

High

Status

NA

Additional information

Related tickets
Transactions removed from mempool
Investigate: Inconsistent Transaction Pool Sizes on gossipqa
payment transaction dropped after inclusion in orphan
Issues with the mempool empty. My mempool is different from the others.
#9868

Potential solutions:

1. Pull based transaction pool sync: Current rebroadcast logic is not robust; nodes that initiate transactions are the ones rebroadcasting. But due to the delay in propagating the transactions can still arrive out of order and therefore be rejected. Also, if a node restarts outside the rebroadcast window which is 50mins, then they don't receive any of the old transactions since mempool is not persisted across restarts. Mempool can be persisted but there is still the issue of transactions dropping when received out of order.
   To get around this, one solution is for nodes to request transactions from peers after catchup and/or periodically when synced.
2. Adding bitswap should alleviate the issue further as transactions can be forwarded with minimum delay
3. Another potential solution is described in The lost transactions issue. #13313 where node allows for nonce gaps in the mempool.