feat(tooling): add event-monitor arrival-time dashboard (lambdaclass#538)

## Summary

Adds `tooling/event-monitor/`, a standalone live arrival-time dashboard
for
lean-consensus (ethlambda) nodes. It dials the `GET /lean/v0/events` SSE
stream
of several nodes, timestamps each event on a single collector clock, and
serves
a browser dashboard visualizing:

- a **rolling beeswarm** of each event's arrival offset *within the
slot*, one lane per node, and
- a **propagation-delta beeswarm**: for a given block / aggregate, how
long after the *first* node each other node saw it.

The rolling window is adjustable live from the header, and a fresh page
load
backfills recent history from the collector so it is never blank.

## Design

- **Standalone Cargo workspace** under `tooling/event-monitor/` (empty
`[workspace]` table); it is not a member of the parent `ethlambda`
workspace and does not affect the main build.
- **Zero dependency on any ethlambda crate** — it only speaks the
documented SSE/HTTP wire shape. `CONTRACT.md` is the frozen interface
between the Rust/axum backend (`src/`) and the vanilla-JS frontend
(`web/`, no build step).
- A single collector clock makes propagation deltas skew-free across
nodes.
- `?demo=1` runs the frontend fully offline with synthetic data.

This PR contains the tooling plus the CI wiring needed to actually check
it. The
RPC / event-bus changes that make a node emit these events shipped
separately in
the chain-events series (below).

## Required ethlambda API functionality

All dependencies are **satisfied on `main`** — the default dashboard
view works
against a current node with no extra PRs.

| Capability | Endpoint / topic(s) | Status |
|---|---|---|
| SSE events stream | `GET /lean/v0/events` | ✅ lambdaclass#517 |
| Topic filtering | `?topics=<csv>` | ✅ lambdaclass#518 |
| Slot-geometry bootstrap | `GET /lean/v0/genesis`, `GET
/lean/v0/config/spec` | ✅ |
| Base topics | `block`, `head`, `justified_checkpoint`,
`finalized_checkpoint` | ✅ lambdaclass#516 |
| Attestation / aggregate topics | `attestation`, `aggregate` | ✅ lambdaclass#534 |
| Block-gossip topic | `block_gossip` | ✅ lambdaclass#535 |

The collector accepts `block`, `block_gossip`, `head`,
`justified_checkpoint`,
`finalized_checkpoint`, `attestation` and `aggregate`; any other topic
is logged
and skipped. `safe_target` / `chain_reorg` (lambdaclass#533, closed) were removed
in
4eeafff — both panels' topic filters discarded them, and `chain_reorg`
is a
point-in-time event rather than a per-node arrival race, so it does not
belong on
a beeswarm.

Both panels default to `block` / `attestation` / `aggregate`.

## Review fixes

Three defects found in review, each with a regression test:

- **Reconnect backoff never reset after a healthy session.** The reset
was keyed on a clean end-of-stream, but a node restart surfaces as a
stream *error*, so every restart ratcheted the delay one step
permanently; after ~7 of them a healthy node reported `down` with 10s
reconnects. Now keyed on session duration (survived ≥1 heartbeat). This
also closes the inverse case: a peer that accepts the request and
instantly closes the stream used to be retried at `INITIAL_BACKOFF`
forever.
- **One bogus slot blanked the dashboard.** Both the history ring and
the rolling window key retention off the highest slot seen, and that
watermark only moves up, so a single event from a node on a different
genesis aged out every real event until a restart. Events more than
`MAX_FUTURE_SLOTS` ahead of the collector's own slot are now dropped,
warning once per node per connection. Only the future side is bounded;
old slots are legitimate (`finalized_checkpoint` trails head) and cannot
move the watermark.
- **Stale status clobbered fresh status.** The frontend applies `status`
immediately but buffers `chain` during backfill, so the `/api/history`
snapshot could overwrite a newer live status. Live status now wins.

Also from that review:

- **Slot geometry is re-resolved every 60s** and republished when it
changes, dropping the retained history and slot watermark, so a
regenerated genesis no longer silently corrupts every subsequent
`offset_ms`. Collectors re-read per frame and `/api/meta` per request;
an already-open tab needs a reload to pick up new geometry.
- **The arrival axis now spans two slots at two scales**: the first at
full resolution (90.9% of the width at 5 intervals), the next compressed
into a tinted band half a first-slot interval wide, saturating beyond
that. A block spilling past its slot boundary is the failure mode worth
seeing, and it used to be indistinguishable from one landing exactly on
the boundary.
- **`offset_ms` includes the collector↔node round trip.** One clock is
what makes propagation deltas skew-free, but nodes reached over
different links carry a systematic offset that reads as lag. Now
documented in `README.md` and `CONTRACT.md §2`.
- History events are `Arc`-shared, so `publish_chain` no longer
deep-copies per event and `/api/history` no longer clones up to
`HISTORY_MAX_EVENTS` events while holding the mutex.
- `topics = []` / `nodes = []` are rejected at load rather than becoming
an opaque retry loop against a 400.

Known and deliberate, left as follow-ups: the collector cannot see
upstream
`: error - dropped N messages` comments (`eventsource-stream` swallows
them), so
a lagging subscription silently under-samples; both canvases still
repaint
unconditionally at 60fps; and `bootstrap` has no retry, so the monitor
exits if
started before its nodes are listening.

## CI

`tooling/event-monitor` declares its own `[workspace]`, so `cargo fmt
--all`,
`cargo check --workspace`, `cargo clippy --workspace`, `make lint` and
`make test`
all stop at the root workspace members and never reached it — it landed
with
nothing verifying it. The `Lint` job now also runs `cargo fmt --check`,
`cargo clippy --all-targets -- -D warnings` and `cargo test` under
`working-directory: tooling/event-monitor`, and `rust-cache` lists it as
a second
workspace so its target dir is cached and its `Cargo.lock` feeds the
cache key.

## Testing

- `cargo test` in `tooling/event-monitor/`: **41 passed** (39 lib + 2
integration), 0 failed.
- `cargo fmt --check` and `cargo clippy --all-targets -- -D warnings`:
clean.
- `?demo=1` offline synthetic mode renders both panels with no live
nodes.
- Verified end-to-end against a live local devnet (blocks land ~0.1–0.2
s into the slot, attestations ~0.8 s = interval 1).
- Two-slot axis geometry checked numerically: the overflow band is
exactly 0.5 × one first-slot interval, the mapping is monotonic, and the
endpoints land on the margins.
- Geometry refresh checked against a stub node: rewriting its
`genesis_time` mid-run is picked up within one refresh interval (`WARN
slot geometry changed`), `/api/meta` then serves the new epoch, and the
history ring holds the restarted chain's low slots with in-slot offsets
— which only works because the watermark is reset, otherwise they would
prune as older than the previous high-water mark.

## Run

```bash
cd tooling/event-monitor
cp config.example.toml config.toml
$EDITOR config.toml                 # list your nodes' RPC URLs + topics
cargo run --release -- --config config.toml
# open the `listen` address (default http://127.0.0.1:8080)
```

Author: MegaRedHand

.github/workflows/ci.yml

@@ -30,7 +30,14 @@ jobs:
           components: rustfmt, clippy
 
       - name: Setup cache
+        # Tools under tooling/ are separate Cargo workspaces with their own
+        # target dir and Cargo.lock, so they need listing explicitly or their
+        # builds are neither cached nor reflected in the cache key.
         uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: |
+            .
+            tooling/event-monitor
 
       - name: Check formatting
         run: cargo fmt --all -- --check
@@ -41,6 +48,24 @@ jobs:
       - name: Clippy
         run: cargo clippy --workspace --all-targets -- -D warnings
 
+      # tooling/event-monitor declares its own [workspace] table, so every step
+      # above stops at the root workspace members and never reaches it. Its
+      # tests run in this job rather than in `test` because clippy has already
+      # compiled the test targets, and because they need none of that job's
+      # leanSpec fixtures.
+      # `--locked` so the committed Cargo.lock is actually enforced: without it
+      # cargo silently resolves and rewrites the lockfile in CI, and a stale or
+      # missing entry never fails the build.
+      - name: Lint tooling
+        working-directory: tooling/event-monitor
+        run: |
+          cargo fmt --all -- --check
+          cargo clippy --locked --all-targets -- -D warnings
+
+      - name: Test tooling
+        working-directory: tooling/event-monitor
+        run: cargo test --locked
+
   test:
     name: Test
     runs-on: ubuntu-latest

tooling/event-monitor/.gitignore

@@ -0,0 +1,2 @@
+/target
+/config.toml