Run bench on demo

[ffakenz]

This PR enhances the demo tutorial by enabling hydra-cluster benchmarks to run on an active Hydra cluster.

usage

nix run .#legacyPackages.x86_64-linux.hydra-cluster.components.benchmarks.bench-e2e -- \
          demo \
          --output-directory=$(pwd)/benchmarks \
          --scaling-factor=100 \
          --timeout=1000s \
          --testnet-magic 42 \
          --node-socket=${NETWORK_DIR}/node.socket \
          --hydra-client=localhost:4001 \
          --hydra-client=localhost:4002 \
          --hydra-client=localhost:4003

prerequisites

• A Cardano node must be running on specified node-socket.
• Hydra nodes must be operational on provided hydra-client hosts.
• There’s no need to pre-seed the keys, as the bench-demo script will automatically fund them using the faucet.
• Note that the reference scripts should already be published, and the Hydra nodes must be running with those scripts.

Todo

• Fix the FIXME about > 33
• Remove duplicate seeding
• Make sure the entire CI process doesn't fail when the pumba causes the network to fail
• Make it so that if it fails the head is closed.
• Quick little matrix to run a few different scenarios
• Make the bench-e2e fail if it didn't submit all the txns ( ideally would also be able to see visually in the job list; but Github is missing a feature see also Please support something like "allow-failure" for a given job actions/runner#2347 )
• Get docker info via docker inspect instead of parsing yaml (!)
• Make sure results.csv is written to the outputDirectory not the tmp directory
• Upload the results as part of the artifacts
• Write the summary out even when it failed

---

• CHANGELOG updated or not needed
• Documentation updated or not needed
• Haddocks updated or not needed
• No new TODOs introduced or explained herafter

[github-actions]

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-08-29 13:04:41.231504601 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	2fac819a1f4f14e29639d1414220d2a18b6abd6b8e444d88d0dda8ff	3799
νCommit	2043a9f1a685bcf491413a5f139ee42e335157c8c6bc8d9e4018669d	1743
νHead	bd9fad235c871fb7f837c767593018a84be3083ff80f9dab5f1c55f9	10194
μHead	c8038945816586c4d38926ee63bba67821eb863794220ebbd0bf79ee*	4607

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5188	5.81	2.30	0.44
2	5389	7.18	2.84	0.47
3	5588	8.46	3.34	0.49
5	5993	11.32	4.48	0.54
10	6998	18.20	7.20	0.66
56	16250	81.53	32.25	1.76

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	559	10.52	4.15	0.29
2	747	13.86	5.65	0.34
3	931	17.33	7.20	0.38
5	1304	24.65	10.44	0.48
10	2247	45.22	19.36	0.75
20	4121	95.99	40.76	1.40

CollectCom transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	549	22.14	8.66	0.42
2	113	663	32.96	13.05	0.54
3	169	769	44.09	17.69	0.67
4	227	879	60.01	24.20	0.85
5	283	994	78.11	31.66	1.05
6	336	1100	91.73	37.56	1.21

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	615	17.71	7.79	0.38
2	847	20.18	9.47	0.42
3	945	21.33	10.65	0.44
5	1153	22.61	12.56	0.47
10	1925	31.25	19.56	0.63
50	8087	98.91	74.98	1.85

Close transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	659	21.02	9.43	0.42
2	735	22.07	10.47	0.44
3	862	23.46	11.84	0.46
5	1197	26.78	15.00	0.53
10	2043	35.48	23.25	0.69
50	8051	99.98	84.29	1.92

Contest transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	670	27.16	11.67	0.48
2	763	28.48	12.83	0.51
3	1022	31.06	15.03	0.55
5	1265	34.49	17.99	0.61
10	2091	44.83	26.75	0.80
38	5990	95.82	70.29	1.69

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5055	17.36	7.55	0.57
2	5220	29.19	12.87	0.71
3	5320	41.54	18.32	0.85
4	5543	59.66	26.52	1.07
5	5672	77.35	34.44	1.27
6	5813	98.64	44.02	1.52

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	5022	7.75	3.28	0.46
5	1	57	5057	9.08	4.08	0.48
5	5	285	5192	13.69	6.96	0.54
5	10	570	5362	18.87	10.31	0.61
5	20	1139	5702	30.38	17.51	0.77
5	30	1706	6041	42.10	24.80	0.94
5	40	2279	6383	53.03	31.76	1.09
5	50	2847	6721	64.37	38.89	1.25
5	81	4615	7776	99.53	61.01	1.74

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-08-29 13:06:50.382260433 UTC

Baseline Scenario

Number of nodes	1
Number of txs	3000
Avg. Confirmation Time (ms)	4.194986589
P99	8.141198899999969ms
P95	5.328280549999999ms
P50	3.9952555ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	9000
Avg. Confirmation Time (ms)	23.726739321
P99	104.91504140000006ms
P95	32.825234349999995ms
P50	21.1135175ms
Number of Invalid txs	0

[github-actions]

Test Results

469 tests  ±0   462 ✅ ±0   18m 21s ⏱️ +58s
150 suites ±0     7 💤 ±0 
  5 files   ±0     0 ❌ ±0 

Results for commit c5d1208. ± Comparison against base commit a51e04c.

♻️ This comment has been updated with latest results.

[ch1bo]

I like that you have put a pull request description explaining how to use this.

However, it would be better (maybe even required) that you leave instructions somewhere on the repository that this mode exists and how to use it! Potential places:

• getting started documentation page (maybe not as this is a first touch point for people
• hydra-cluster/README.md which also explains the other running modes of the benchmarks: https://github.com/input-output-hk/hydra/blob/1b0111f0bd23349e1381a7952b38d22ff1386b24/hydra-cluster/README.md#L114 (probably best to explain it here)

Furthermore, I'm not too happy about where you "cut" and duplicated the existing dataset generation logic. I think we can re-use much more code if we not distinguish too and just fetch the faucet utxo before generating the UTxO. In the clean devnet case the dataset should still be deterministic (that's why we did do it with genesis utxo originally).

I see quite some unnecessary complexity also coming from the fact that we have Dataset unchanged, while in demo mode we don't actually need the "fuel keys" in the ClientKeys (actually we also don't need the hydra keys, see other comments). Our assumption is (and can be) that the nodes we connect to have access to the right keys and have some fuel to spend.