Automated fault tolerance test

[sonallux]

Hey everyone,

to support and track the progress of my thesis about fault tolerance in microservice-based architectures I have written some automated fault tolerance tests for the ODS. They are running on GitHub actions, so this status badge will always reflect the latest status of my fault tolerance tests:

Current test cases

Restart test

Tests that every service can be restarted, without causing any problems. After starting the ODS it performs some requests to add some state. Then one service at a time is stopped and restarted. After the service is running again checks are performed to validate that no data is lost and the ODS is still completely functional. Because the services are currently not replicated, the test ensures that no traffic is performed between stopping and the successful restart of service. Persistent storage is also not erased and therefore no state inconsistencies can arise.

Following bugs have been detected by this test:

• closing the database connection will crash the microservice Make PostgresRepository fault tolerant node-dry-pg#2
• Already consumed events are redelivered after a service restart
• Connections to RabbitMQ are not restarted if they brake

Scheduler initialization test

This test is for issue #284

[lunedis]

Really nice work!

If GitHub is running their GitHub Action runners inside e.g. kubernetes, and they use containers to create the jobs, and you use containers inside such a job to run services, how many levels deep are we and when do we reach the limbo? 😄