Da sequencer bench test plan

[musitdev]

Da sequencer bench test plan

Description of DA-Sequencer Evolution

The new DA-Sequencer includes the following changes:

• Block production and streaming
  A new process, movement-da-sequencer, now fetches batches from the full node, produces and saves blocks at regular intervals, and then streams them to all connected full nodes.
• Refactored transaction pipeline
  The transaction pipeline inside the full node has been completely refactored.
• Removal of Celestia components
  All previous Celestia processes have been removed.

Scope: These changes only affect the transaction pipeline and block production. The core Aptos framework and block execution remain unchanged, so our testing will focus primarily on transaction processing.

---

Types of Tests

We will perform several types of tests:

1. Load Test
   Gradually increase the stream of transactions (TX) to determine the maximum throughput the node can sustain and identify the failure mode when it breaks.
2. Soak Test
   Send a steady, high-volume TX stream to a single node over an extended period (e.g., 5 hours at 300 TPS) to uncover memory leaks or performance degradation.
3. Follower-Node Test
   Verify that follower nodes—now able to submit TX—keep up with the DA-Sequencer and maintain correct state.
4. Start/Stop/Update Tests
   Ensure that restarting or upgrading the DA-Sequencer (and full nodes) does not disrupt node synchronization or cause idle connections.

Note: For this initial session, all tests will be run manually—no test automation is in place yet.

---

Test Topology

• Instances: 3 full nodes + 1 DA-Sequencer
• Hardware:
  • 1 × c5.4xlarge on AWS for the DA-Sequencer + one full node
  • 2 × c5.4xlarge for the remaining full nodes
• Software: Identical versions on all instances
• All full nodes connect to the same DA-Sequencer endpoint.

---

Load Test

We will use three load generators:

• Alice & Bob CLI
  Parallel transfer TX with configurable concurrency. Increase until the node fails.
• Go-based CLI
  Similar to Alice & Bob, but uses our Go test harness.
• Howzit
  Executes more complex smart-contract workflows.

Strategy:

1. Run Alice & Bob and the Go CLI side by side, gradually increasing concurrency.
2. Stop when the node breaks; record the maximum sustainable TPS (target ≥ 500 TPS).
3. Use a single full node instance for these tests.

---

Soak Test

• Duration: 5 hours
• Load: 300 TPS (constant)
• Instance: Single full node + DA-Sequencer

---

Follower-Node Test

• Run both load and soak tests against 3 full nodes, each attached to the same da-sequencer.
• Load Session: Ramp up to failure (target ≥ 1000 TPS).
• Soak Session: 5 hours at 600 TPS.

---

Start/Stop/Update Tests

DA-Sequencer Restart

1. Start network with 3 full nodes + 1 DA-Sequencer.
2. Send TX only while all nodes are up.
3. Restart the DA-Sequencer 20 times: each shutdown takes 10 s longer than the last (first: 10 s; second: 20 s; etc.).
4. After each restart, wait for all full nodes to sync and process 10 TX, then verify they share the same version and block height.
5. Repeat with continuous TX load (soak mode).

Full-Node Restart

1. Keep the DA-Sequencer running.
2. Send TX continuously to all 3 nodes.
3. Stop one full node for 30 s, then restart it.
4. Verify it catches up to the others and matches their block height and version.

Rolling Update

1. Start with all components on version v1.
2. Stop and update the DA-Sequencer and one full node to version v2 (only TX execution logic changed).
3. Restart them, then verify all 3 full nodes converge to the same block height and version.

Fast Restart

1. Start network with 3 full nodes + 1 DA-Sequencer, streaming TX continuously.
2. Stop the DA-Sequencer for ~1 s, restart it, then stop again ~1 s later.
3. Repeat 5 times, then finally leave it running.
4. Verify all full nodes sync and have the same version/block height.

---

Test Results

Intermediary result.

Load Test one full node

Using one full node and the DA-Sequencer, with both Alice & Bob and the Go CLI, we sustained 250 TPS with no issues.

Next steps: Add more accounts, integrate the Go suite’s account files into Alice & Bob, and increase throughput further.

Soak Test one full node

Ran a 5-hour soak at an average 150 TPS with no issues detected.

Load Test result Summary

Load testing was performed using both the Alice & Bob CLI and the Go-based CLI tools.

• The Go-based CLI was able to send around 150 TPS per host at maximum.
• Using the addresses generated by the Go CLI together with the Alice & Bob CLI, I achieved up to 200 TPS per host.
• With 3 hosts, the maximum throughput reached 600 TPS.

The limiting factor was always the client; the full node host was never saturated. I did not test with more than 3 hosts, as 600 TPS at peak is sufficient to handle the load we expect in the coming months.

I did not test with more complex smart contracts. The scaling observed is related only to transaction scheduling and ordering, not transaction execution. The current version of Howzit test did not use the smart contracts and performed only transfers. Further development is required to use the test smart contract.

Soak Test result Summary

Load testing was performed using both the Alice & Bob CLI and the Go-based CLI tools.

• With 3 hosts, the test run 5 with a load between 200 and 400 TPS.
• The node never saturates.
• On all full node types (node that only execute block and node receive Tx and execute) a little memory leak was observed (see Memory and CPU data). The CPU usage was constant.
• On the da-sequencer no memory leak was observed, and the CPU consumption was constant.

Memory and CPU data

• RUN1:
  Node that run da-sequencer and full node that receive Tx.

 root@ip-172-31-38-32:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3648876      629732        3152    27858724    28006360
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1778196 449856 movement-full-n
 211067  211067 TS       - Ssl  9887360 2320752 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet

 root@ip-172-31-38-32:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3889284      328620        3148    27919428    27765916
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1777168 460592 movement-full-n
 211067  211067 TS       - Ssl  9933084 2549244 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet

 root@ip-172-31-38-32:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3944488      426064        3152    27766780    27710884
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1777168 471144 movement-full-n
 211067  211067 TS       - Ssl  9940288 2591092 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet

Node that only execute Tx:

root@ip-172-31-46-41:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3090464      599700        2980    28928820    29046868
Swap:              0           0           0
root@ip-172-31-46-41:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9785972 2019592 movement-full-n
root@ip-172-31-46-41:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3163508      257000        2980    29198320    28973824
Swap:              0           0           0
root@ip-172-31-46-41:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9786996 2078728 movement-full-n

root@ip-172-31-46-41:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3319780      356680        2976    28942356    28817552
Swap:              0           0           0
root@ip-172-31-46-41:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9818048 2241956 movement-full-n

• RUN2:
  Node that run da-sequencer and full node that receive Tx.

root@ip-172-31-38-32:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3937520      440264        3148    27759548    27717864
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1908240 500848 movement-full-n
 211067  211067 TS       - Ssl  9838376 2598672 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet
root@ip-172-31-38-32:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3973156      394404        3148    27769772    27682108
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1908240 499388 movement-full-n
 211067  211067 TS       - Ssl  9854676 2625296 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet
               total        used        free      shared  buff/cache   available
Mem:        32137332     4227672      419764        3148    27489896    27427832
Swap:              0           0           0
root@ip-172-31-38-32:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 210373  210373 TS       - Ssl  1908240 524120 movement-full-n
 211067  211067 TS       - Ssl  9915844 2853740 movement-full-n
 986927  986927 TS       - Ssl  1260008 28672 movement-faucet

Node that only execute Tx:

root@ip-172-31-46-41:~free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3380368      335336        2976    28903236    28756964
Swap:              0           0           0
root@ip-172-31-46-41:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9861592 2285260 movement-full-n
root@ip-172-31-46-41:~# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3446600      365976        2976    28806440    28690732
Swap:              0           0           0
root@ip-172-31-46-41:~# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9863896 2349400 movement-full-n
root@ip-172-31-46-41:/home/ubuntu# free
               total        used        free      shared  buff/cache   available
Mem:        32137332     3714656      348372        2976    28555648    28422676
Swap:              0           0           0
root@ip-172-31-46-41:/home/ubuntu# ps -eo pid,tid,class,rtprio,stat,vsz,rss,comm | grep movem
 133484  133484 TS       - Ssl  9990580 2645148 movement-full-n

DA-Sequencer Restart Test result

The da-sequencer was stopped and restarted several times, with different stop durations during both load and idle periods.

When a full node detects that the connection to the da-sequencer has been broken, the node stops and restarts.
If the restart time is short, local full nodes detect the restart and reconnect to the da-sequencer.
However, remote full nodes do not detect the restart and continue receiving blocks as if nothing happened.

No full node encountered a state issue. When a node restarts, it resynchronizes with the da-sequencer.

Full-Node Restart Test result

Restarts of both local and remote full nodes were performed on nodes that were either only executing transactions or receiving and executing transactions. The nodes restarted at the correct height and resynchronized up to the current da-sequencer height. No node encountered a sync state issue. All nodes resynchronized correctly and re-executed the blocks in the correct order.

Rolling Update Test result

Tests were conducted with several scenarios. The initial test scenario executed correctly, other scenario was, only one full node was updated, only the da-sequencer was updated, and all full nodes were running. All nodes resynchronized correctly.

The updates concerned only the full node or da-sequencer logic. The Aptos core component was never changed.

Fast Restart Test result

This test was done during the start/stop test. Fast restart was only detected by local fullnode. All node resync and keep in sync.

[0xmovses]

I am assuming you want all these tests to pass before merging in #1117 ?

[musitdev]

I don't think so. For now, I just want the whole architecture to be working (several nodes, da-sequencer, backup restore, and some load).
I've installed a network with two full nodes, the da-sequencer, and tested basic backup, restore, and sync after restart.
I have to do some cleaning and I have a little issue with the start DA height. After the PR can be reviewed.

[musitdev]

Done