fix: update miner mempool iterator query to consider both nonces and fee rates

[rafaelcr]

Problem

The current query used by Stacks miners to iterate over pending mempool transactions sorts them only by fee_rate in descending order. This approach creates the following problems:

1. Since it does not consider transaction nonces, non-mineable transactions with high fees but old (or very future) nonces go to the top of the list and get visited every time, thus wasting block construction time budget.
2. Transactions with null fee rates are sent to the bottom of the list even if their nonces should be evaluated next, forcing the algorithm to query them separately with some probability of inclusion.

Solution

This PR adds a new mempool walk strategy (NextNonceWithHighestFeeRate) which prioritizes transactions that can be confirmed as fast as possible (next expected nonces for both origin and sponsor) with the highest fee rates as possible. This means that even if it doesn't select the transactions with the highest global fees first, it will select those that can be mined immediately therefore optimizing block building time and allowing the miner to fit in more transactions in total within the allotted time. This strategy also mixes in transactions with null fee rates in results by simulating a fee rate for the purposes of ordering only.

For easier testing, a mempool walk strategy enum value was added to the MinerConfig, which includes:

• GlobalFeeRate: Current global fee rate sorting behavior. Enabled by default.
• NextNonceWithHighestFeeRate: The new strategy added by this PR

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jcnelson]

Thanks for taking this on @rafaelcr!

This is something that I think is going to need somewhat extensive real-world testing before we make it the default behavior. Do you think you could add this new mempool walk logic as an opt-in alternative, which the node operator can opt to use? This would give miners a chance to compare/contrast the new behavior with the old behavior in an easy-to-rollback manner. A subsequent PR could make the new behavior the default.

What do you think?

[rafaelcr]

Great idea @jcnelson and thanks for the feedback! I'll get on it.

[rafaelcr]

your new commits look great @obycode ! thanks for pushing this to the finish line 🙌

[obycode]

This new mempool iteration algorithm sacrifices optimal fee-based transaction selection in order to guarantee high performance. Since transaction ordering is not consensus critical (other than nonce-ordering per account), this is allowed. There are a couple of downsides I can think of.

First is the potential for lower total fees for miners. In practice, this is not likely to be the case, because when the mempool is not very full, the order will be less optimal, but all ready transactions are likely to be included in the block anyway. When the mempool is more full, the current algorithm would stall or only generate tiny blocks, so even though the order is not perfect, miners will end up with more total fees, thanks to the selection speedup.

Next is disturbance to the fee market. With this algorithm, chained nonce transactions (where one account has multiple outstanding transactions in the mempool) can end up prioritized behind lower fee-rate transactions. The query used selects all "ready" transactions based on their nonces. If account A has transaction A1 with nonce 0 and A2 with nonce 1, both paying fees of 10 STX, and account B has transaction B0 with nonce 0 paying 0.001 STX, the selection algorithm will choose A0, then B0, then B1. This is due to the fact that on the first query, A1 is not "ready", so it is not included in the results and thus not considered. After A0 has been mined, then A1 will be "ready".

[codecov]

Codecov Report

Attention: Patch coverage is 63.77102% with 905 lines in your changes missing coverage. Please review.

Project coverage is 83.61%. Comparing base (9937c41) to head (b2503f2).
Report is 20 commits behind head on develop.

Files with missing lines	Patch %	Lines
...net/stacks-node/src/tests/nakamoto_integrations.rs	1.25%	553 Missing ⚠️
testnet/stacks-node/src/tests/signer/v0.rs	51.40%	224 Missing ⚠️
stackslib/src/core/util.rs	87.36%	59 Missing ⚠️
stackslib/src/core/mempool.rs	86.28%	31 Missing ⚠️
stacks-common/src/util/lru_cache.rs	90.33%	20 Missing ⚠️
stackslib/src/core/nonce_cache.rs	93.71%	13 Missing ⚠️
stackslib/src/config/mod.rs	50.00%	3 Missing ⚠️
stackslib/src/chainstate/stacks/miner.rs	94.44%	1 Missing ⚠️
.../src/chainstate/stacks/tests/block_construction.rs	99.55%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #5541       +/-   ##
============================================
+ Coverage    68.14%   83.61%   +15.46%     
============================================
  Files          521      524        +3     
  Lines       382814   384636     +1822     
  Branches       323      323               
============================================
+ Hits        260884   321604    +60720     
+ Misses      121922    63024    -58898     
  Partials         8        8               

Files with missing lines	Coverage Δ
stacks-common/src/types/sqlite.rs	100.00% <100.00%> (ø)
stacks-common/src/util/mod.rs	88.23% <ø> (+1.47%)	⬆️
stackslib/src/chainstate/stacks/tests/mod.rs	81.32% <100.00%> (+2.88%)	⬆️
stackslib/src/core/mod.rs	61.78% <ø> (+3.29%)	⬆️
stackslib/src/core/tests/mod.rs	91.49% <100.00%> (+41.78%)	⬆️
stackslib/src/net/mod.rs	78.61% <ø> (+1.36%)	⬆️
stackslib/src/net/tests/download/nakamoto.rs	96.08% <ø> (+56.95%)	⬆️
stackslib/src/net/tests/inv/nakamoto.rs	96.66% <ø> (+44.70%)	⬆️
stackslib/src/net/tests/mempool/mod.rs	100.00% <ø> (+30.62%)	⬆️
testnet/stacks-node/src/nakamoto_node/miner.rs	83.95% <100.00%> (-0.79%)	⬇️
... and 19 more

... and 294 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c715b5...b2503f2. Read the comment docs.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[obycode]

The remaining issue is due to the fact that the nonce cache is not getting reset after a block is rejected. I will fix this next.

[obycode]

I think the only remaining task here is to ensure that we don't get stuck if there are 11,650 problematic and ready transactions in the mempool.

[obycode]

There is room for further improvement by not resetting the nonce DB after a block is successfully mined. I think I will leave that for a future PR though, because it is a bit troublesome.