The two mechanisms that strand a machine -- state-handler enqueue failure and accumulating retry_count -- are logged or persisted but never metered, and firmware/validation outcomes are uncountable. This makes "stuck" alertable.
What this involves
- State-handler wake-up/enqueue-failure counter (
machine_scout.rs, host_reprovisioning.rs, dpu.rs) -- if the enqueue fails, the machine silently stops being driven; that's the stuck-machine mechanism and it's warn!-only today.
- Reprovision retry-depth gauge/histogram --
retry_count is persisted today and exported nowhere, so "machines churning on retries" has no signal.
- Firmware update started/completed/failed counters by cause -- including the "wrong version after attempted update" case that's warn!-only today.
- Machine-validation and measured-boot outcome counters (by alert cause, so stale-timeout storms and genuine test failures are distinguishable) + collector iteration duration.
- A periodic-loop heartbeat (iterations, last-success-age, error counter) for every background manager -- a wedged loop only shows as stale gauges today. And make the health discovery loop retry instead of terminating on first failure.
- Add the state-controller framework's missing
_object_tasks_errored_total -- the per-iteration log line has errored_tasks, the metric family doesn't.
Depends on the carbide-instrument framework. Fans out as a few PRs at pickup.
Part of #3169.
The two mechanisms that strand a machine -- state-handler enqueue failure and accumulating retry_count -- are logged or persisted but never metered, and firmware/validation outcomes are uncountable. This makes "stuck" alertable.
What this involves
machine_scout.rs,host_reprovisioning.rs,dpu.rs) -- if the enqueue fails, the machine silently stops being driven; that's the stuck-machine mechanism and it's warn!-only today.retry_countis persisted today and exported nowhere, so "machines churning on retries" has no signal._object_tasks_errored_total-- the per-iteration log line haserrored_tasks, the metric family doesn't.Depends on the
carbide-instrumentframework. Fans out as a few PRs at pickup.Part of #3169.