Skip to content

Make stuck machines and provisioning outcomes alertable #3175

Description

@chet

The two mechanisms that strand a machine -- state-handler enqueue failure and accumulating retry_count -- are logged or persisted but never metered, and firmware/validation outcomes are uncountable. This makes "stuck" alertable.

What this involves

  • State-handler wake-up/enqueue-failure counter (machine_scout.rs, host_reprovisioning.rs, dpu.rs) -- if the enqueue fails, the machine silently stops being driven; that's the stuck-machine mechanism and it's warn!-only today.
  • Reprovision retry-depth gauge/histogram -- retry_count is persisted today and exported nowhere, so "machines churning on retries" has no signal.
  • Firmware update started/completed/failed counters by cause -- including the "wrong version after attempted update" case that's warn!-only today.
  • Machine-validation and measured-boot outcome counters (by alert cause, so stale-timeout storms and genuine test failures are distinguishable) + collector iteration duration.
  • A periodic-loop heartbeat (iterations, last-success-age, error counter) for every background manager -- a wedged loop only shows as stale gauges today. And make the health discovery loop retry instead of terminating on first failure.
  • Add the state-controller framework's missing _object_tasks_errored_total -- the per-iteration log line has errored_tasks, the metric family doesn't.

Depends on the carbide-instrument framework. Fans out as a few PRs at pickup.

Part of #3169.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Enhancement.

Projects

Status
In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions