Devnet soak: observability stack (Grafana / Datadog)

[kh0ra]

Why

The 24 h devnet soak (#67) needs real observability so we can answer:

• Did any tx fail with an unexpected error code?
• Did CU baseline drift slot-over-slot?
• Did the soak harness itself stay alive?

Sub-tasks

• Datadog or Grafana Cloud free tier setup
• Scrape Solana RPC for our PROGRAM_ID metrics
• Alert on: CU drift > ±10 %, parsed-error rate > 0 (we don't expect any), soak harness down
• Dashboards committed to docs/grafana/

Note

Without this we'll have a 24 h log of "everything was fine probably" — that doesn't pass audit-grade review.

[kh0ra]

Observability stack shipped

mosaic-soak now emits Prometheus metrics (*.prom) alongside its markdown report, and docs/devnet-soak/observability.md specifies the full Grafana/Datadog dashboard + alerts.

Metrics (text exposition format, Grafana textfile-collector / pushgateway / Datadog openmetrics):

• mosaic_soak_unexpected_failure_total - the gate (alert P0 when > 0)
• outcome counters, mosaic_soak_duration_seconds, mosaic_soak_cu_drift_alerts_total
• mosaic_soak_cu{dispatch,stat} gauges - per-dispatch min/median/p95/max + pinned baseline, so the dashboard plots measured-vs-baseline and catches drift the static bench baseline can't

Dashboard: 5 panels (soundness gate single-stat, outcome mix, CU-vs-baseline per dispatch, drift counter, throughput).

Alerts: P0 soundness failure (wired to docs/rollback-playbook.md), P1 CU approaching the 1.4M cap, P2 drift / stall.

23 mosaic-soak tests green (added the render_prometheus test). The binary writes <stem>.prom next to <stem>.md on every run.

The code is done; standing up the actual Grafana/Datadog instance + wiring the scraper is operational, and the soak produces real series once it runs against a deployed program (#68 deploy).