Add chaos-testing scenarios: RPC outage, timeout, malformed response

[Gbangbolaoluwagbemiga]

Overview

Add automated chaos-testing scenarios that inject RPC outages, slow responses, and malformed/truncated responses into the analysis pipeline to verify the failover logic from #1146 and error handling degrade gracefully rather than crashing or hanging.

Background / Context

#1146 adds RPC failover logic; this issue proves it actually works under realistic fault injection, not just the happy path, matching the resilience testing expected of production infrastructure.

Goal

A fault-injection test harness (e.g. a mock RPC server with configurable failure modes) exercised in CI, asserting the system degrades gracefully (clear error, no hang, no crash) under each injected fault.

Scope of Work

1. Build a mock RPC server supporting configurable outage/timeout/malformed-response modes.
2. Write test scenarios covering: full outage, partial outage (some providers down), slow response near timeout, malformed JSON/XDR response.
3. Assert graceful degradation for each scenario.

Out of Scope

• Testing infrastructure-level chaos (e.g. killing containers) — application-level fault injection only.

Acceptance Criteria

• Chaos test suite covers at least 4 distinct fault scenarios.
• All scenarios result in graceful degradation, not crashes or hangs.

Files & Pointers

• tooling/sanctifier-core/tests
• tests/

Dependencies / Related

Depends on #1146.

Difficulty & Estimated Effort

Medium — ~3-4 days.

[YngPrince11]

@YngPrince11 has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

I would like to work on this issue. Please assign it to me,Thanks

ℹ️ Repo Maintainers: To accept this application, review their application or assign @YngPrince11 to this issue.

[dahmeezy]

@dahmeezy has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

I would love to work on this issue, please assign it to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @dahmeezy to this issue.

[bakarezainab]

@bakarezainab has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

Hi Maintainer, I am a web3 fullstack developer. I’ve reviewed this issue and I understand the requirements. I’d like to implement this and submit a clean PR. Kindly assign this issue to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @bakarezainab to this issue.

[drips-wave]

Congratulations, @bakarezainab! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on July 30, 2026.

🧑‍💻 @bakarezainab: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊