Testing: chaos/fault-injection for RPC, DB, and Redis outages

[Depo-dev]

Summary

Add chaos / fault-injection tests for RPC, DB, and Redis outages to verify graceful degradation and recovery.

Background & current behaviour

Several resilience features are planned/present (retries, circuit breaker #197, RPC failover #213, fail-open rate limiting, outbox #200). None are verified under real induced faults; we only assume they degrade gracefully.

Why this matters for MVP

Resilience code that isn't tested under actual faults is unproven. Chaos tests turn "should degrade gracefully" into "verified to".

Proposed implementation

1. Build scenarios that kill/slow each dependency: RPC down/slow, Postgres down, Redis down — using compose (pause/stop containers) or a fault-injection proxy (toxiproxy).
2. Assert expected behaviour during the fault (breaker opens, API returns 503 not hang, rate limiter fails open, cursor stays safe) and after restore (ingest resumes, no data loss/duplication, gaps closed).
3. Make them runnable locally via compose and documented.

Acceptance criteria

• Scenarios for RPC down/slow, DB down, Redis down with expected behaviour asserted.
• Recovery asserted (cursor safe, no loss/dup, gaps closed) after restore.
• Runnable locally via compose; documented.

Affected files / components

• docker/ (chaos harness / toxiproxy)
• integration tests in crates/ and services/api/

Testing & verification

Run each scenario and assert the documented degradation + recovery; wire an opt-in CI job.

Related / dependencies

Exercises #197/#200/#213/#196 (reorg) and fail-open rate limiting.

Out of scope

Production chaos engineering (game days).

---

Target branch

All work for this issue MUST target the dev branch. Cut your feature branch from dev and open the pull request against dev, never against main. Any PR opened against main for this issue will be closed and asked to re-target dev.

[Andreschuks101]

@Andreschuks101 has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

I would love to resolve this issue, please assign it to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Andreschuks101 to this issue.

[josephchimebuka]

@josephchimebuka has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

Hi GM, I have reviewed this issue and taken the time to understand the requirements, current implementation, and the expected outcome. Based on my analysis, I believe this is something I can successfully handle and contribute to... Kindly assign

ℹ️ Repo Maintainers: To accept this application, review their application or assign @josephchimebuka to this issue.

[DioChuks]

@DioChuks has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

Hello, I'll add chaos / fault-injection tests for RPC, DB, and Redis outages to verify graceful degradation and recovery. I'll run both the unit test, & suite test to verify things concerning a graceful recovery.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @DioChuks to this issue.

[drips-wave]

Congratulations, @DioChuks! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on July 30, 2026.

🧑‍💻 @DioChuks: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊