feat: sequence aware mempool prototype #2414

cmwaters · 2025-09-05T13:52:11Z

A priority mempool based on gas price has always been incompatible with the replay protection mechanism that relies on monotonically increasing sequences. This becomes increasingly apparent when the same signer is submitting multiple blobs per block.

I have written this ADR proposing a simple-ish solution which bundles pending transactions per signer. I also have written a working prototype (some of it vibe-coded) which solves the issue with the tx client test here: celestiaorg/celestia-app#5655

rach-id · 2025-09-08T13:44:04Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+
+- Signer-bundled sets: A `txSet` holds transactions from a single signer. The set tracks:
+    - `txs`: maintained in ascending sequence order; ties broken by earlier arrival timestamp.
+    - `aggregatedPriority`: gas-weighted average of member transaction priorities, floored to `int64`.


since the priority is the average, couldn't someone post very cheap transactions at the beginning, and have the 100th transaction with crazy high fee forcing you to include the first ones, but when you arrive to the last, the block space is full 😆

What do you think of switching to median?

My initial intuition was to use "next tx gas" as txSet priority. So we can always extract the most profitable transactions from mempool and maintain ordering inside sets.

But Increasing fees under congestion section clarifies this for me.

couldn't someone post very cheap transactions at the beginning, and have the 100th transaction with crazy high fee forcing you to include the first ones, but when you arrive to the last, the block space is full 😆

The user would presumably still pay for the 100th transaction because it would just be included in the following block unless you can find a way to invalidate it.

A counter proposal to your median idea (which I think could also be effective):

Evaluate the priority based on the first n bytes worth of transactions (where n is less than the size of a block) rather than the aggregated priority of all the transactions

rach-id · 2025-09-08T13:52:05Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

can we run some benchmarks while we're at it to see if this is introducing any delays in processing transactions?

Do we have existing benchmarks that would be good to understand any changes in the performance (or do we need to write new ones)

I don't think we have anything on main. I have personal ones I use for my benchmarks, but they're unmergeable 😄

rach-id · 2025-09-08T13:53:38Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+- Global ordering of sets: `orderedTxSets` is kept sorted by `(aggregatedPriority desc, firstTimestamp asc)`. This preserves FIFO between sets of equal aggregated priority.
+- Within-set ordering: Transactions are iterated in ascending sequence, guaranteeing that a later sequence from the same signer is never submitted before an earlier one.
+- Eviction policy when full: Compute the new set aggregated priority if the incoming transaction were added (`aggregatedPriorityAfterAdd`). Identify victim sets with aggregated priority strictly below the incoming set. Evict transactions from victim sets starting from the end (highest sequence numbers first) until enough bytes are freed. If insufficient bytes can be freed, reject the incoming transaction.
+- TTL purge: Expiration by height/time removes transactions from sets and reorders or removes empty sets accordingly.


when purging after TTL, we could delete the earliest nonce transaction while the ones after it are not removed, right?

No we'd remove an entire set all together - the reasoning is that all subsequent transactions would have invalid nonces and be rejected by recheck tx anyway

rach-id · 2025-09-08T13:53:55Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+        - transaction index, byte accounting, reservation tracking
+
+- Aggregation:
+    - On insert: update aggregated prioritt `totalGasWanted += gasWanted`, `weightedPrioritySum += priority * gasWanted`, then `aggregatedPriority = weightedPrioritySum / totalGasWanted` (integer division). Aggregated priority is relative to the amount of gasWanted per transaction. Transactions with more gasWanted will have larger weighting. Alternatively we could have used transaction size.


Suggested change

- On insert: update aggregated prioritt `totalGasWanted += gasWanted`, `weightedPrioritySum += priority * gasWanted`, then `aggregatedPriority = weightedPrioritySum / totalGasWanted` (integer division). Aggregated priority is relative to the amount of gasWanted per transaction. Transactions with more gasWanted will have larger weighting. Alternatively we could have used transaction size.

- On insert: update aggregated priority `totalGasWanted += gasWanted`, `weightedPrioritySum += priority * gasWanted`, then `aggregatedPriority = weightedPrioritySum / totalGasWanted` (integer division). Aggregated priority is relative to the amount of gasWanted per transaction. Transactions with more gasWanted will have larger weighting. Alternatively we could have used transaction size.

rach-id · 2025-09-08T13:55:25Z

mempool/cat/pool.go

@@ -334,10 +334,11 @@ func (txmp *TxPool) TryAddNewTx(tx *types.CachedTx, key types.TxKey, txInfo memp
 		txmp.metrics.FailedTxs.Add(1)
 		return rsp, nil
 	}
+	// removed noisy print used during debugging


Suggested change

// removed noisy print used during debugging

rach-id · 2025-09-08T13:55:33Z

mempool/cat/pool.go

@@ -461,6 +462,7 @@ func (txmp *TxPool) Update(
 	newPreFn mempool.PreCheckFunc,
 	newPostFn mempool.PostCheckFunc,
 ) error {
+	// removed noisy print used during debugging


Suggested change

// removed noisy print used during debugging

rach-id · 2025-09-08T13:56:38Z

mempool/cat/pool.go

+	outer:
+		for _, set := range victimSets {
+			// iterate in reverse order removing the higher sequence numbers first
+			for i := len(set.txs) - 1; i >= 0; i-- {
+				tx := set.txs[i]
+				txmp.evictTx(tx)
+				availableBytes += tx.size()
+				if availableBytes >= wtx.size() {
+					break outer
+				}


can this be reorganized to avoid having a tag? it becomes harder to understand the logic

tzdybal

I would emphasize two facts in context section:

Eviction never removes lower sequence numbers in a set before removing higher ones, preserving the ability to submit remaining sequences correctly.
Rather than replace by fee, users can improve the likeness of their transaction landing by increasing the gas price of later transactions, thus raising the aggregated priority. Given sequence awareness, a replace by fee protocol may easily be supported in the future.

tzdybal · 2025-09-09T06:04:05Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+
+- Signer-bundled sets: A `txSet` holds transactions from a single signer. The set tracks:
+    - `txs`: maintained in ascending sequence order; ties broken by earlier arrival timestamp.
+    - `aggregatedPriority`: gas-weighted average of member transaction priorities, floored to `int64`.


My initial intuition was to use "next tx gas" as txSet priority. So we can always extract the most profitable transactions from mempool and maintain ordering inside sets.

But Increasing fees under congestion section clarifies this for me.

tzdybal · 2025-09-09T06:11:58Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+- Within a signer, transactions are never emitted out of ascending sequence order.
+- Across signers, ordering is by set weighted average priority (desc), with earlier `firstTimestamp` breaking ties (FIFO between equal-priority sets).
+- Weighted average priority updates are consistent with gas-weighted means as transactions are added/removed.
+- Eviction never removes lower sequence numbers in a set before removing higher ones, preserving the ability to submit remaining sequences correctly.


nit: This is very important - I would mention this even in context section.

rootulp

One idea to avoid the attack Rachid raised is instead of calculating the average weighted priority, preserve the logic to capture tx sets but only inspect the lowest sequence in each tx set when evaluating which txs to include in a block. Then choose txs to include based on priority.

In other words, I'm proposing a less sophisticated mempool that does not calculate average weighted priority but does respect the signer's need to get lower sequence txs included before higher sequence txs.

rootulp · 2025-09-09T14:34:27Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+
+## Decision
+
+Group transactions by signer into signer-bundled sets ("tx sets") and order based on average weighted priority per set. Within tx set order transactions in ascending sequence order. Block building and recheckTx iterates through transactions in those order. This will preserve sequence ordering per signer while maximising tx fees per block.


[question] Is it possible for an attacker to submit two txs:

tx 1 with sequence 1 and a low priority

tx 2 with sequence 2 and a high priority

such that the second tx increases the average weighted priority of the tx set such that tx 1 gets included on-chain, however once tx 1 gets included on-chain it invalidates tx 2 so tx 2 can no longer get on chain. The end result is that the signer got tx 1 included on-chain ahead of competing txs that would have paid more in fees?

Update: oops disregard, I see Rachid raised the same concern.

Responded to it in rachid's comment

adlerjohn · 2025-09-09T15:17:52Z

docs/celestia-architecture/adr-011-sequence-aware-cat.md

+
+## Decision
+
+Group transactions by signer into signer-bundled sets ("tx sets") and order based on average weighted priority per set. Within tx set order transactions in ascending sequence order. Block building and recheckTx iterates through transactions in those order. This will preserve sequence ordering per signer while maximising tx fees per block.


Rather than an attack, is this an issue in the honest but congested case?

Yes we need this mechanism in the case of congestion or more broadly whenever we expect priorities to be different

ninabarbakadze

Callum will hand this over to me and I'll focus on cleaning it up

…t-mempool

cmwaters · 2025-09-15T13:59:02Z

One idea to avoid the attack Rachid raised is instead of calculating the average weighted priority, preserve the logic to capture tx sets but only inspect the lowest sequence in each tx set when evaluating which txs to include in a block. Then choose txs to include based on priority.

Can you perhaps pseudocode the design? So could only the first transaction with the lowest sequence make it into a block or would you be able to do multiple?

cmwaters added 2 commits September 5, 2025 15:41

feat: sequence aware cat mempool prototype

4a568a0

Merge branch 'v0.39.x-celestia' into cal/2d-cat-mempool

e79013d

evan-forbes mentioned this pull request Sep 5, 2025

[Feature Request]: allow user to set account sequence celestiaorg/celestia-app#5672

Open

Merge branch 'v0.39.x-celestia' into cal/2d-cat-mempool

a6ea411

cmwaters marked this pull request as ready for review September 8, 2025 11:10

cmwaters requested review from rach-id, evan-forbes, tzdybal and ninabarbakadze as code owners September 8, 2025 11:10

rach-id reviewed Sep 8, 2025

View reviewed changes

rootulp assigned cmwaters Sep 8, 2025

rootulp self-requested a review September 8, 2025 15:02

tzdybal reviewed Sep 9, 2025

View reviewed changes

rootulp reviewed Sep 9, 2025

View reviewed changes

adlerjohn reviewed Sep 9, 2025

View reviewed changes

ninabarbakadze reviewed Sep 11, 2025

View reviewed changes

ninabarbakadze marked this pull request as draft September 11, 2025 14:23

ninabarbakadze assigned ninabarbakadze and unassigned cmwaters Sep 11, 2025

ninabarbakadze added 3 commits September 12, 2025 13:28

Merge remote-tracking branch 'origin/v0.39.x-celestia' into cal/2d-ca…

804434d

…t-mempool

refactor: initial

89831e6

refactor: the store for better readbility

0100c3d

ninabarbakadze and others added 2 commits September 15, 2025 20:36

test: add comprehensive tests and what to add

96d87ef

make the block building code more sophisticated

e684a13

	- On insert: update aggregated prioritt `totalGasWanted += gasWanted`, `weightedPrioritySum += priority * gasWanted`, then `aggregatedPriority = weightedPrioritySum / totalGasWanted` (integer division). Aggregated priority is relative to the amount of gasWanted per transaction. Transactions with more gasWanted will have larger weighting. Alternatively we could have used transaction size.
	- On insert: update aggregated priority `totalGasWanted += gasWanted`, `weightedPrioritySum += priority * gasWanted`, then `aggregatedPriority = weightedPrioritySum / totalGasWanted` (integer division). Aggregated priority is relative to the amount of gasWanted per transaction. Transactions with more gasWanted will have larger weighting. Alternatively we could have used transaction size.


		## Decision

		Group transactions by signer into signer-bundled sets ("tx sets") and order based on average weighted priority per set. Within tx set order transactions in ascending sequence order. Block building and recheckTx iterates through transactions in those order. This will preserve sequence ordering per signer while maximising tx fees per block.

feat: sequence aware mempool prototype #2414

Are you sure you want to change the base?

feat: sequence aware mempool prototype #2414

Conversation

cmwaters commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tzdybal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rootulp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ninabarbakadze left a comment

Choose a reason for hiding this comment

Uh oh!

cmwaters commented Sep 15, 2025

Uh oh!

Uh oh!

cmwaters commented Sep 5, 2025 •

edited

Loading