Packet loss fault injection test

[ffakenz]

Why

The goal is to take a first, simple step towards setting up a hydra cluster that simulates high transaction loads under network failures.

By doing so, we aim to identify and fix problems in our networking stack, ultimately improving the robustness of the hydra cluster under adverse network conditions.

Hopefully, this setup will help us consistently reproduce issues that cause our hydra-node to get stuck, as detailed in #1436.

What

The task involves setting up a local hydra cluster running on devnet and using Docker containers to run the necessary components under a custom Docker network.

Then, a simulation script, similar to the existing bench program, will be executed.

This script will send a high volume of transactions to the nodes while network failures are generated using Pumba.

How

• run-docker.sh will be used to setup the hydra cluster using Docker.

  this results in a cluster of 3 hydra nodes (Alice, Bob, and Carol) connected to the same cardano node running on devnet, with the hydra scripts published and the parties funded by the faucet.

• refactor the bench executable script to drive the simulation.

  • it should run under devnet.
  • it should reuse alice, bob and carol keys.
  • it should connect to already running cardano and hydra node running containers.

• use Pumba, on demand, to inject network failures into hydra nodes.

  • only use netem loss for now.
  • failures should be interleaved during transaction processing (once the head is open).
  • only one failure should occur on a hydra node at any given time.

• have step in CI that tests how the network reliability handles package loss failures.