Skip to content

Task remains wedged after host verification failure: successor Attempt inherits exhausted timeout-recovery state and settles as lifecycle-progression #3154

Open

Description

@metacoma

Problem

In GSD 1.23.0 an execute-task can remain permanently wedged after a host verification failure.

Observed on Task M007/S08/T03 (unit execute-task):

  1. The task-plan Verify command ran the full BDD suite and failed with exit 101 because required broker dependencies were absent.
  2. GSD recorded the failure and routed recovery.
  3. The broker dependencies were subsequently restored, and the full BDD suite passed outside GSD.
  4. The Task still could not complete:
    • gsd_task_complete returned: Canonical Task completion has no running Attempt to close.
    • The latest Attempt was settled as failed with failureClass=lifecycle-progression.
    • Recovery repeatedly entered terminal task-recovery-abort.
    • The recovery blocker said: idle recovery exhausted 2 attempts. Status: DB task status is not closed.

Reproduction sequence

  1. An execute-task Attempt starts.
  2. Initial host verification fails.
  3. Recovery is created; the Task is retried.
  4. A successor Attempt is claimed after an earlier run exhausted idle recovery.
  5. On its first idle timeout, the successor is immediately treated as recovery-exhausted.
  6. The unit is advanced without an executor result, so the Attempt settles failed / lifecycle-progression.
  7. Recovery routes a terminal abort.
  8. No running Attempt remains; gsd_task_complete correctly refuses completion.
  9. Re-entering /gsd auto repeats the same abort path.

Root cause

Timeout-recovery state was stored per unit, not per Attempt.

  • Dispatch-time writeUnitRuntimeRecord() passed timeoutAt: null and a recovery count from resolveDispatchRecoveryAttempts(), but the merge used nullish coalescing:
    • null for timeoutAt did not clear the previous run's value.
    • undefined for recoveryAttempts fell back to the previous run's exhausted count.
  • recoverTimedOutUnit() then used runtime.recoveryAttempts as the exhaustion counter instead of the budget on the current dispatch row.

Evidence from the affected database:

  • A successor Attempt started at 22:43.
  • Its runtime record still had timeout_at around 22:01 and recovery_attempts: 7.
  • Its first idle recovery was immediately considered exhausted and ended the unit without a result.

This turns a stale runtime counter into a terminal lifecycle-progression failure, an agent-owned abort, and a Task with no running Attempt.

Expected behavior

  • Every newly dispatched Attempt must start with a fresh per-dispatch timeout-recovery budget.
  • Recovery resume must claim exactly one authorized successor Attempt.
  • Fresh host verification must run, produce a passing Technical Verdict, and publish the completion.
  • If recovery is genuinely exhausted, completion must stay fail-closed; the running-Attempt requirement must not be removed.

Additional note

The task-plan Verify command also contains dependency-gated @export scenarios. That is a separate verify-contract/environment issue. The lifecycle defect above is what makes the Task unwedgeable even after the environment is restored.

Fix

A fix is prepared in metacoma/gsd-pi on branch fix/task-recovery-lifecycle-deadlock (commit bc1abf6d):

  • Reset run-scoped runtime timeout/recovery state when a new Attempt starts.
  • Derive recovery exhaustion from the dispatch-row budget when a dispatch exists.
  • Keep the runtime record as fallback only for units without a dispatch row.

Regression coverage: before the fix 0/3 regression tests pass; after the fix 3/3 pass.

Activity

  1. added
    needs-triageUnlabeled or unassigned — needs maintainer review
    bugSomething isn't working
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageUnlabeled or unassigned — needs maintainer review
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    High PrioritybugSomething isn't workingready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions