chore(skills): graph gossip arrival timing in the client dashboard

#566 added `lean_gossip_*_arrival_delay_seconds` and `lean_gossip_*_arrival_total`,
which answer "are votes arriving late, or not arriving at all?" from inside each
node. Nothing graphed them, so answering it still meant an out-of-band
event-monitor run against a collector clock.

Adds a "Gossip Arrival Timing" row to `client-dashboard.json`: a 3x4 grid, one
column per message kind (block / attestation / aggregate), one row per view.

- delay p99/p50 per node, with a dashed line at one interval (0.8s): above it the
  typical message misses the interval it was due in
- delay distribution heatmap, which exposes the bimodal profile that percentiles
  average away
- arrival position stacked by `position`, the only view that recovers the sign the
  absolute-value histogram discards (rising `before` is clock skew, rising `after`
  is propagation or CPU). The aggregate panel omits `before`, unreachable by
  construction since aggregates anchor to the latest aggregation-interval boundary
  rather than their own data slot
- on-time fraction per node, so one late node separates from a fleet-wide drop

All 15 queries were run against the central Prometheus before committing.

Also adds a receiver-side timing block to the node-health checklist, whose item 5
covered only the node's own duties, and corrects where dashboards get deployed:
the JSONs live in the host dir bind-mounted at `/var/lib/grafana/dashboards`, not
in `<GRAFANA_PROV_DIR>/dashboards`, which holds only the provider yaml. A JSON
dropped in the provisioning tree is silently ignored, which reads as a working
copy that never appears. Recorded as `GRAFANA_DASHBOARDS_DIR`.

Author: MegaRedHand

.claude/skills/multi-server-devnet/SKILL.md

@@ -207,16 +207,33 @@ Relabel gotcha: that file is a single-file bind-mount — edit by full overwrite
 `--web.enable-remote-write-receiver`; Grafana groups by **job name**, so a
 converted node must be relabelled (both `job_name:` and `client_type:`).
 `scripts/finality-dashboard.json` is a ready Grafana dashboard (head / justified
-/ finalized per devnet, one series per `network`) — drop it in the provisioning
-dashboards dir; it auto-loads. `scripts/client-dashboard.json` is the main
-per-node dashboard for the same dir (see the inventory below). Every dashboard
-picks its datasource through a **template variable** (`ds_prom`/`ds_loki`;
+/ finalized per devnet, one series per `network`) — copy it to the central
+Grafana's **dashboards dir** and it auto-loads. `scripts/client-dashboard.json` is
+the main per-node dashboard for the same dir (see the inventory below). Every
+dashboard picks its datasource through a **template variable** (`ds_prom`/`ds_loki`;
 `datasource` on the client dashboard) rather than a pinned uid, so they drop into
-any Grafana unedited — no deployment-specific datasource ids in the JSON. Gotcha: that dir
-is bind-mounted **read-only** into the container, so Grafana UI edits are
-reverted on the next provisioner sweep (~30s) despite `allowUiUpdates: true`.
-Edit the JSON and re-copy; treat the repo copy as the source of truth and keep it
-in sync, since nothing pulls server-side edits back.
+any Grafana unedited — no deployment-specific datasource ids in the JSON.
+
+**That dir is NOT `<GRAFANA_PROV_DIR>/dashboards`** — the provisioning tree holds
+only the provider yaml (`dashboards.yml`, which declares
+`options.path: /var/lib/grafana/dashboards`); the JSONs live in whatever host dir
+is bind-mounted there, a sibling of the provisioning tree in this deployment
+(`GRAFANA_DASHBOARDS_DIR` in `devnet.env`). Confirm before copying, since dropping
+a dashboard in the provisioning dir loads nothing and looks like a silent no-op:
+```bash
+ssh "$METRICS_HOST" 'sudo docker inspect '"$GRAFANA_CONTAINER"' \
+  --format "{{range .Mounts}}{{.Source}} -> {{.Destination}} (ro={{not .RW}}){{println}}{{end}}"'
+# ship it (back up first; the provisioner sweeps every ~30s)
+ssh "$METRICS_HOST" "cp -p $GRAFANA_DASHBOARDS_DIR/client-dashboard.json{,.bak-\$(date +%Y%m%d-%H%M%S)}"
+cat scripts/client-dashboard.json | ssh "$METRICS_HOST" "cat > $GRAFANA_DASHBOARDS_DIR/client-dashboard.json"
+```
+Verify it landed with
+`curl -s $GRAFANA_BASE_URL/api/dashboards/uid/<uid> | jq '.dashboard.version'` (the
+version bumps on each sweep that reads a changed file). Gotcha: that dir is
+bind-mounted **read-only** into the container, so Grafana UI edits are reverted on
+the next provisioner sweep (~30s) despite `allowUiUpdates: true`. Edit the JSON and
+re-copy; treat the repo copy as the source of truth and keep it in sync, since
+nothing pulls server-side edits back.
 
 **Logs (Loki + promtail).** Mirrors the metrics path: a per-host **promtail**
 ships node-container stdout/stderr to the **central Loki** (7d retention;
@@ -368,12 +385,13 @@ Swap (persistent): `fallocate -l 16G /swapfile && chmod 600 && mkswap && swapon`
 | `deploy-finality-alert.sh` | operator | `[WEBHOOK_FILE]`; `METRICS_HOST`, `GRAFANA_*`, `PROM_DS_UID`; `DRY_RUN` | Render + ship the "lost finality" Slack alert to the central Grafana |
 | `devnet-env.sh` / `devnet.env.example` | operator | `$DEVNET_ENV`, `./devnet.env`, script dir | Load this deployment's hosts/urls/Grafana ids as defaults; exported vars win. Copy the example to `devnet.env` (gitignored) once |
 
-**Grafana dashboards** (drop into the provisioned dashboards dir; they auto-load in
-~30s and pick their datasource through a template variable, so no editing):
+**Grafana dashboards** (copy into the central Grafana's dashboards dir —
+`GRAFANA_DASHBOARDS_DIR`, *not* the provisioning tree; they auto-load in ~30s and
+pick their datasource through a template variable, so no editing):
 
 | File | uid | Content |
 |---|---|---|
-| `client-dashboard.json` | `lean-ethereum-clients-dashboard` | The main per-node dashboard: Overview (start time, validators, committees, head/justified/finalized, slot + finality-delay graphs) plus 13 collapsed sections (config, sync, peers, req/resp + mesh, gossip, PQ signatures, aggregation coverage, block production, proposal internals, fork choice, attestations, state transition, storage + tick health). Filters `network`/`job`/`instance` — `instance` is the **host**, not the node |
+| `client-dashboard.json` | `lean-ethereum-clients-dashboard` | The main per-node dashboard: Overview (start time, validators, committees, head/justified/finalized, slot + finality-delay graphs) plus 14 collapsed sections (config, sync, peers, req/resp + mesh, gossip, gossip arrival timing, PQ signatures, aggregation coverage, block production, proposal internals, fork choice, attestations, state transition, storage + tick health). Filters `network`/`job`/`instance` — `instance` is the **host**, not the node |
 | `finality-dashboard.json` | `devnet-finality-overview` | head / justified / finalized per devnet, one series per `network` |
 | `resources-dashboard.json` | `devnet-resources` | Per-node CPU cores + memory working set (+ limit + %-of-limit for OOM watch) from cAdvisor |
 | `logs-dashboard.json` | `devnet-logs` | Loki logs panel + log-volume-by-level, filtered `network`/`node`/`stream`/`search` |

.claude/skills/multi-server-devnet/references/node-health.md

@@ -236,6 +236,24 @@ histogram_quantile(0.9, sum by (le) (rate(lean_committee_signatures_aggregation_
 histogram_quantile(0.9, sum by (le) (rate(lean_pq_sig_aggregated_signatures_building_time_seconds_bucket{network="$NET"}[10m])))
 ```
 
+**Receiver side** — the same question asked of what *arrives* rather than what
+this node produces. `lean_gossip_*_arrival_total` classifies each gossip message
+by whether it landed inside the interval it was due in, so an on-time fraction
+well under 1 says votes are arriving late rather than not at all:
+```promql
+# share of arrivals inside their due interval, per node (block|attestation|aggregation)
+sum by (job) (rate(lean_gossip_attestation_arrival_total{network="$NET", position="inside"}[10m]))
+/ sum by (job) (rate(lean_gossip_attestation_arrival_total{network="$NET"}[10m]))
+# how late, in seconds (absolute distance from the due interval's start)
+histogram_quantile(0.9, sum by (le, job) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network="$NET"}[10m])))
+```
+One node low while the fleet is fine is that node's own clock or CPU; everyone
+dropping together is chain-wide production or propagation. Graphed in the client
+dashboard's **Gossip Arrival Timing** row. Aggregates are anchored to the latest
+aggregation-interval boundary, not their own data slot, so their tail reads as
+aggregation cost (cross-check the two aggregation-time histograms above), and
+`position="before"` never appears for them.
+
 **Log:** presence/absence timing — e.g. `"Finished building block"` arriving
 after the next slot boundary. For exact slot-relative offsets and spill
 detection, use the **`devnet-profiling`** skill (this item only says in/out of

.claude/skills/multi-server-devnet/references/operations.md

@@ -205,7 +205,13 @@ The **central** stack is a separate one-time deployment: a prometheus started wi
 `:9099`), a Loki with `allow_structured_metadata: true` + schema v13/tsdb and
 `retention_enabled: true` (it is false by default and `retention_period` alone is
 silently ignored), and a Grafana with provisioning dirs for datasources,
-dashboards, and alerting.
+dashboards, and alerting. Two distinct dirs matter here: the **provisioning** tree
+(`GRAFANA_PROV_DIR`, mounted at `/etc/grafana/provisioning`) holds the datasource,
+dashboard-provider and alerting yamls; the **dashboard JSONs** live in a separate
+host dir (`GRAFANA_DASHBOARDS_DIR`) bind-mounted read-only at the provider's
+`options.path` (`/var/lib/grafana/dashboards`). `deploy-finality-alert.sh` writes
+to the first; dashboards go to the second. A JSON dropped in
+`GRAFANA_PROV_DIR/dashboards` is silently ignored.
 
 **node_exporter is a systemd service, not a container.** It's installed from
 `github.com/lambdaclass/monitoring-stack` (ansible: `make inventory TARGET=...`