dashboard: mark service latency sources

[carlos4s]

Summary

• add latency_source to /api/services so rows distinguish cached health latency, live endpoint latency, and systemd-only status
• add short live probes for known HTTP endpoints when no cached health row exists
• render systemd-only dashboard rows as n/a instead of ambiguous --

Fixes #86

Testing

• cargo fmt --check
• cargo test -p genie-api
• cargo clippy -p genie-api --all-targets --locked -- -D warnings
• Jetson aarch64 release build and deploy of genie-api

Real Behavior Proof

• Unit tests cover cached health latency, live endpoint latency, and systemd-only not_applicable latency.
• Dashboard rendering now maps latency_source=not_applicable to n/a and marks live probes as ms live.
• Jetson /api/services reports explicit latency sources for all service rows.

Jetson Validation

Sanitized operator-run Jetson result:

core          healthy=True   latency=87    latency_source=health          source=health+systemd   error=None
llm           healthy=True   latency=6     latency_source=health          source=health+systemd   error=None
api           healthy=True   latency=0     latency_source=live            source=live+systemd     error=None
health        healthy=True   latency=None  latency_source=not_applicable  source=systemd          error=None
governor      healthy=True   latency=None  latency_source=not_applicable  source=systemd          error=None
mqtt          healthy=False  latency=None  latency_source=not_applicable  source=systemd          error=auto-restart
audio         healthy=True   latency=None  latency_source=not_applicable  source=systemd          error=None
whisper       healthy=True   latency=None  latency_source=not_applicable  source=systemd          error=None
wakeword      healthy=False  latency=None  latency_source=not_applicable  source=systemd          error=failed
homeassistant healthy=True   latency=None  latency_source=not_applicable  source=systemd          error=None
OK

[ai-hpc]

Right shape for #86's gap. PR #84 expanded /api/services to include systemd-only units but left their latency field as a misleading --. This PR adds a third explicit state, so the wire format now distinguishes:

• latency_source = "health" — cached row from genie-health's SQLite (the original path).
• latency_source = "live" — fresh probe issued from this request, used when a service exposes HTTP but doesn't have a cached row yet (typically the dashboard's own genie-api).
• latency_source = "not_applicable" — systemd-only services with no HTTP endpoint (governor, mqtt, audio, whisper, wakeword by default).

Dashboard JS maps the three states to Nms / Nms live / n/a accordingly. The n/a is the user-facing payoff — operators can finally tell "this service is healthy but has no HTTP probe by design" apart from "this service should have a probe but isn't responding".

Implementation pieces worth calling out:

• latency_url: Option<String> on each ServiceTarget — present only for services that actually expose HTTP. core / llm / homeassistant / nextcloud / jellyfin read from config.services.<name>.url (so the LLM probe automatically follows [services.llm].backend between genie_llm and genie_ai_runtime URLs). api hardcodes http://127.0.0.1:3080/api/status — self-probe, a tiny circularity but always-reachable for a healthy genie-api. The rest are None, which is what cleanly drives the not_applicable branch in merge_service_rows.
• collect_live_latency_rows — gated on if health.contains_key(&target.service) { continue; } so we only probe when no cached health row exists. Prevents redundant fork-of-systemctl-equivalents on the hot path.
• probe_http_latency — raw TCP + minimal HTTP/1.1 GET with two 750 ms timeouts (connect, read), only reads first 256 bytes (enough for the status line), parses status code, returns healthy on 2xx-3xx. Intentionally minimalist — no reqwest/hyper dependency, no TLS path, no chunked-response handling. Right call for a local-only probe.

Worth flagging, not blocking:

• Sequential probes in the request handler. Each /api/services request walks the target list and awaits each probe_http_latency serially. Worst-case (all 5 probable targets time out at 1.5 s = 750 ms connect + 750 ms read): ~7.5 s for one /api/services response. Realistic case under healthy operation: ~50 ms × 5 = ~250 ms, fine for a 5 s dashboard poll. If you ever notice the dashboard stalling under failure mode, parallelizing with futures::future::join_all over the probe set would be a one-screen change. Not blocking — failure-mode dashboard lag is a poor symptom but not a critical one.
• genie-api doesn't have PR #87's LocalSet::spawn_local refactor. That fix only landed on genie-core::server. So /api/services blocking inside genie-api is currently a single-threaded queue with everything else on :3080. Same flagging as above — if probe timeouts cascade and block /api/status, that's the lever to pull.
• Self-probe of api via 127.0.0.1:3080/api/status is essentially "if I can serve this request, can I serve this request?" — guaranteed to succeed in practice, since the response is generated by the same daemon answering the probe. Useful as an end-to-end response-time signal in the rollup, but the operator should know it's measured from inside the box.

Tests are exactly the right shape:

• service_rows_merge_health_and_systemd_status updated to assert both latency_source="health" (for core, via cached HealthRow) and latency_source="live" (for api, via injected LiveLatencyRow { healthy: false, response_ms: 17, error: Some("HTTP 503") }). The 5xx-with-non-zero-response-ms case is the trickiest combination — "probe got a response, just not a healthy one" — and the test pins it.
• New systemd_only_service_rows_mark_latency_not_applicable covers the latency_url: None path for wakeword, asserts latency_source="not_applicable" + source="systemd" + response_ms = None.

Operator's Jetson run in the PR body is the strongest signal — all 10 service rows now have explicit latency_source values that match the design (core/llm from health, api from live, the rest from not_applicable).

All 6 CI checks green on 0e6d55a (fmt, clippy, test, aarch64 cross-compile, --no-default-features, PR body checklist). Going in.

[ai-hpc]

Merged at 3f461f5af4170486d36ce0c58bdb86fee91e2a16.
Thanks to @carlos4s!