Skip to content

fix: distinguish terminal fixed lease replays - #1925

Merged
steipete merged 2 commits into
openclaw:mainfrom
Melbourneandrew:fix/fixed-terminal-replay
Sep 12, 2026
Merged

fix: distinguish terminal fixed lease replays#1925
steipete merged 2 commits into
openclaw:mainfrom
Melbourneandrew:fix/fixed-terminal-replay

Conversation

@Melbourneandrew

@Melbourneandrew Melbourneandrew commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Distinguish a matching fixed-ID replay whose stored lease has already ended from an actual change to its create intent. Keep HTTP 409, the existing owner/organization/intent/attempt checks, and the rule that a terminal replay never re-enters provisioning. The CLI already propagates the definitive response; no CLI production compatibility path is added.

Integrated current main, retained Andrew Melbourne's original correction and regression coverage, clarified the command/coordinator/identifier documentation, and added the maintainer changelog entry thanking @Melbourneandrew.

Verification

  • go test -race ./internal/cli -run 'TestCoordinatorFixedCreateAmbiguousError(RepeatsPutAndDoesNotAdoptConflictingGet|ReportsSameIntentTerminalResult)$' -count=20: 40 focused test invocations passed.
  • Worker suite: 3,148 tests passed, with 15 existing skips. Worker format, lint, typecheck, Worker/Node builds, docs links/site build, and CLI build passed.
  • Current-main integration checks passed again. The seven-file patch is byte-identical to the qualified patch; only the already-merged workspace-owner test repair was added from main. The rebuilt CLI and coordinator bundle are byte-identical to the passing integration-proof artifacts.
  • Managed Codex review of the current-main candidate: P0 scoped-clean. This is not an all-priority review certificate.

Actual coordinator / PostgreSQL / HTTP proof

The actual built CLI contacted the shipped Node coordinator through a socket recorder. The coordinator used its normal shared-token authentication, PostgreSQL runtime/storage, and real Hetzner provider/client. A task-process-only outbound routing shim sent the adapter's requests through native HTTP fetch to an owned loopback fixture. The fixture returned an ordinary provider HTTP 412; it did not replace the coordinator/provider classes, seed terminal state, or script coordinator responses.

Observed trace:

initial fixed-ID PUT       -> 500 after one provider POST /servers
identical CLI recovery PUT -> 409 fixed_lease_terminal
additional identical PUT  -> 409 fixed_lease_terminal
same-owner changed intent -> 409 lease_id_conflict
provider POST /servers count: 1 throughout
CLI exit: 1

The first three PUT request bodies had identical hashes. Actual PostgreSQL readback showed the same failed lease with no resource ID and no provisioning uncertainty/retryability. The provider fixture's SSH key was removed. All owned processes and HTTP handlers finished; the task PostgreSQL container/volume and isolated HOME/keys/repository were removed, with zero cleanup errors. Source, index, runtime, and artifact bindings remained unchanged.

This is local integration proof over real sockets and PostgreSQL, not hosted Hetzner/cloud proof. Only synthetic fixture credentials were used. An initial database-readiness timeout and an incomplete startup diagnostic are retained as failed/incomplete evidence. The successful attempt explicitly waited for PostgreSQL's final TCP listener before the unchanged host SQL and replay checks; production behavior and replay assertions were not weakened.

Configuration / rollout

No secrets or configuration changes are required. Callers still receive HTTP 409 for a terminal replay, but its diagnostic code is now fixed_lease_terminal instead of the misleading lease_id_conflict. Existing request-drift checks and provider-side-effect boundaries remain unchanged.

Pre-merge CI passed all 12 jobs, plus the five Connector checks, Docs UI, and Release Check: https://github.com/openclaw/crabbox/actions/runs/34686358617. Merged as 6ab890a. Fresh actual-merge builds and the complete local PostgreSQL/HTTP proof passed; post-merge CI passed all 12 jobs at https://github.com/openclaw/crabbox/actions/runs/34687818856, with the five Connector and four CodeQL analysis jobs also passing.

@clawsweeper

clawsweeper Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 6, 2026
@clawsweeper

clawsweeper Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 12, 2026, 5:43 AM ET / 09:43 UTC (Revision 2).

ClawSweeper review

What this changes

The coordinator distinguishes matching requests for ended leases from conflicting create requests, with regression coverage and updated documentation.

Merge readiness

Ready for maintainer review

The correction remains necessary: current main and v0.57.0 still misclassify matching terminal replays. The new production-path transcript satisfies the previous proof request, and no blocking patch defects were found.

Priority: P2
Reviewed head: 42c7dbb89d79053e45ac17fa16d0a5bfa7891463

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused correction with relevant regression coverage and sufficient production-path proof resolves the earlier evidence gap.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (terminal): The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.
Evidence reviewed 8 items Introduced response split: The patch retains owner, organization, immutable intent, and checkpoint-attempt comparisons before returning fixed_lease_terminal. Both rejection branches remain HTTP 409 and return before provisioning or claim consumption.
Current main still needs the correction: The pinned main implementation combines identity mismatch and non-live state in the same lease_id_conflict branch.
Latest release retains the old response: Reading v0.57.0 through GitHub confirmed that matching terminal requests still enter the combined conflict branch.
Findings None None.
Security None None.

How this fits together

Crabbox’s coordinator receives fixed-ID lease requests from the CLI and compares them with stored lease identities before provisioning. Its replay response tells the CLI whether to reuse a live lease or stop with a definitive error.

flowchart TD
  A[CLI fixed-ID request] --> B[Coordinator replay check]
  C[Stored lease and attempt] --> B
  B --> D{Owner and intent match?}
  D -->|No| E[Conflict response]
  D -->|Yes| F{Lease live?}
  F -->|Yes| G[Return existing lease]
  F -->|No| H[Terminal response without provisioning]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test delta Production +10/-1; tests +116/-1 The small diagnostic correction has focused coverage for terminal replay and CLI recovery.

Technical review

Best possible solution:

Keep terminal replay diagnostics distinct while preserving HTTP 409, existing identity checks, and the prohibition on repeated provisioning.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection establishes that an identical request against a terminal stored lease reaches main’s misleading conflict response; this review did not execute a failing main run.

Is this the best way to solve the issue?

Yes, splitting the terminal response after existing identity checks is a narrow correction, and the unchanged CLI already handles the definitive HTTP status.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 751f6a420e98.

Labels

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This fixes a misleading error during fixed-ID recovery with a limited behavioral scope.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.
  • proof: sufficient: Contributor real behavior proof is sufficient. The supplied after-fix transcript exercises FleetCoordinator through the actual CLI, Node HTTP server, PostgreSQL storage, and real provider HTTP client with an injected 412; matching replays stop with fixed_lease_terminal while provider creation remains at one.

Evidence

What I checked:

  • Introduced response split: The patch retains owner, organization, immutable intent, and checkpoint-attempt comparisons before returning fixed_lease_terminal. Both rejection branches remain HTTP 409 and return before provisioning or claim consumption. (worker/src/fleet.ts:5067, 42c7dbb89d79)
  • Current main still needs the correction: The pinned main implementation combines identity mismatch and non-live state in the same lease_id_conflict branch. (worker/src/fleet.ts:5067, 751f6a420e98)
  • Latest release retains the old response: Reading v0.57.0 through GitHub confirmed that matching terminal requests still enter the combined conflict branch. (worker/src/fleet.ts:4994, baa6c9a783f5)
  • Existing CLI compatibility: Recovery repeats the identical create request and returns definitive HTTP errors. Its uncertainty classifier retries transport failures and server errors, so the unchanged HTTP 409 remains definitive without adding CLI production code. (internal/cli/provider_coordinator.go:688, 42c7dbb89d79)
  • After-fix production-path evidence: The complete supplied PR body, captured under sourceRevision c4b292196b153a20ad8d9fe85fd32dde32184d08cdcb117642261bdeaa1cb573, records the actual CLI, Node coordinator, PostgreSQL storage, and real Hetzner HTTP client against an owned HTTP fixture. A provider 412 yields an initial 500, then identical recovery and additional PUTs yield 409 fixed_lease_terminal; changed intent yields lease_id_conflict, with one provider POST throughout. PostgreSQL readback confirms the failed record. This resolves the prior review’s specific proof request; it establishes local integration behavior, not hosted cloud qualification. (42c7dbb89d79)
  • Production owner matches the proof: The Node server constructs FleetCoordinator, and its runtime uses PostgresCoordinatorStorage, matching the production owners described in the supplied transcript. (worker/node/server.ts:44, 42c7dbb89d79)

Likely related people:

  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)
  • reviewed 2026-09-06T19:50:17.318Z sha da4947e :: needs real behavior proof before merge. :: none

Integrate current main while retaining the original terminal-replay diagnostic split, regression coverage, and documentation. Add the maintainer changelog entry with contributor credit.

Verified the real CLI and Node coordinator over HTTP with PostgreSQL and the actual provider adapter against an owned local fixture: identical terminal replays remain distinct from intent drift and do not repeat provider creation. This is local integration proof, not hosted Hetzner proof.

Co-authored-by: Andrew Melbourne <melbourneandrew@gmail.com>
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Sep 12, 2026
@steipete
steipete merged commit 6ab890a into openclaw:main Sep 12, 2026
24 checks passed
@steipete

steipete commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Merged as 6ab890a, preserving Andrew Melbourne's contribution and adding the maintainer changelog entry.

The old replay check treated an already-ended lease as a different create intent. The fix separates those cases after the existing identity checks: matching terminal requests return fixed_lease_terminal; changed intent still returns lease_id_conflict. Both stay HTTP 409 and stop before another provider create.

Before merge, the repeated focused race tests, 3,148 passing Worker tests (15 existing skips), static/build/docs gates, managed P0-scoped Codex review, and all 12 CI jobs passed. The five Connector checks, Docs UI, and Release Check also passed. Fork workflow approval was handled by the maintainer; no test rerun was needed. CI: https://github.com/openclaw/crabbox/actions/runs/34686358617.

After merge, a fresh clean checkout, offline dependency install, CLI build, and Node build passed. The actual-merge CLI then exercised the shipped coordinator, PostgreSQL storage, and real provider client over owned local HTTP sockets. The trace again showed initial 500, identical recovery 409 fixed_lease_terminal, another exact replay with the same result, and changed-intent 409 lease_id_conflict; provider creation remained at one and CLI exit remained 1. Database readback and key removal passed. All owned processes, handlers, PostgreSQL container/volume, and test HOME/repository were cleaned up with zero errors. This is local integration proof, not hosted Hetzner/cloud proof.

The initial database-startup timeout and incomplete diagnostic remain preserved. Post-merge CI passed all 12 jobs at https://github.com/openclaw/crabbox/actions/runs/34687818856; all five Connector and four CodeQL analysis jobs also passed. Pages and the automatic coordinator deployment workflow succeeded. Those are workflow observations, not a serving-version or hosted-cloud proof claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants