Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- AWS: add administrator-only legacy cleanup recovery backed by authenticated original CloudTrail allocation evidence, preserving remaining key and access cleanup and recording an atomic scope-recovery audit without force-success. [PR 1975](https://github.com/openclaw/crabbox/pull/1975).
- AWS: complete cleanup after a verified empty instance response without skipping owned keys, bind new leases to the original account and Region, and retain unresolved historical cleanup when that authority is missing. [PR 1904](https://github.com/openclaw/crabbox/pull/1904). Thanks @vincentkoc.
- Add a credential-free local-container quickstart skill and publish both Crabbox skills through installer discovery, with validated catalog metadata and a responsive installation guide. [PR 1911](https://github.com/openclaw/crabbox/pull/1911). Thanks @zozo123.
- Distinguish identical fixed-ID coordinator replays of terminal leases from conflicting create intent, without repeating provider creation. [PR 1925](https://github.com/openclaw/crabbox/pull/1925). Thanks @Melbourneandrew.

## 0.57.0 - 2026-09-11

Expand Down
11 changes: 6 additions & 5 deletions docs/commands/warmup.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,12 @@ it and may append a short suffix if an active lease already uses that slug.
providers that explicitly support fixed identities. Direct AWS, Machine0, Daytona, Incus,
and local-container leases, managed coordinator leases, and explicitly capable
external providers accept it. Replaying the same normalized create intent
returns or joins the same lease, including after the creating process loses its
response. Reusing the ID with a different provider, slug request, SSH key,
machine or container shape, capabilities, lifetime, or other immutable create
input fails with `lease_id_conflict` before another provider create. Slugs
remain display aliases and are never used as the idempotency key.
returns or joins the same live lease, including after the creating process loses
its response. A managed coordinator reports `fixed_lease_terminal` when that
same intent has already ended. Reusing the ID with a different provider, slug
request, SSH key, machine or container shape, capabilities, lifetime, or other
immutable create input fails with `lease_id_conflict` before another provider
create. Slugs remain display aliases and are never used as the idempotency key.

Daytona binds the native organization before allocation and preserves that scope
across credential rotation. See [Daytona fixed operation IDs](../providers/daytona.md#fixed-operation-ids)
Expand Down
10 changes: 6 additions & 4 deletions docs/features/coordinator.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,10 +283,12 @@ receipt-bearing CLI.
The fixed-ID `PUT` route is fail-closed and does not replace legacy `POST`.
It atomically reserves a versioned normalized immutable request hash before
provider work. An identical owner-scoped replay returns an active lease or the
same provisioning record; request drift and terminal-ID reuse return
`lease_id_conflict` without invoking the provider. CLIs using `--lease-id`
poll a provisioning replay until it becomes active or terminal. Coordinators
that predate this route return not found before any create side effect.
same provisioning record; the same request against its terminal record returns
`fixed_lease_terminal`, while request drift and terminal-ID reuse by a different
request return `lease_id_conflict`. Neither terminal response invokes the
provider. CLIs using `--lease-id` poll a provisioning replay until it becomes
active or terminal. Coordinators that predate this route return not found before
any create side effect.
If the PUT response is ambiguous, the CLI repeats the full identical PUT until
the coordinator atomically confirms the same stored intent or returns a
conflict/definite error. Public GET is used only after that PUT confirmation,
Expand Down
6 changes: 4 additions & 2 deletions docs/features/identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,10 @@ to own replay, and caller cancellation never releases them.
Automation may instead supply the canonical ID with `warmup --lease-id`. For
direct AWS, direct Machine0, direct Daytona, direct local-container, and managed coordinator
leases, that ID is an immutable create identity: an identical semantic replay
returns the same lease, while intent drift returns `lease_id_conflict`. External
providers also accept requested IDs when their protocol explicitly advertises
returns the same live lease, while intent drift returns `lease_id_conflict`.
Managed coordinator replay of the same terminal intent returns
`fixed_lease_terminal`. External providers also accept requested IDs when their
protocol explicitly advertises
idempotent lease identity support. The coordinator durably stores a versioned
normalized request hash. Direct AWS durably stores the intent and current
resolved EC2 attempt in the normal lease claim before `RunInstances`, then uses
Expand Down
54 changes: 54 additions & 0 deletions internal/cli/provider_coordinator_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2259,6 +2259,60 @@ func TestCoordinatorFixedCreateAmbiguousErrorRepeatsPutAndDoesNotAdoptConflictin
}
}

func TestCoordinatorFixedCreateAmbiguousErrorReportsSameIntentTerminalResult(t *testing.T) {
t.Setenv("CRABBOX_OWNER", "test@example.com")
oldRecoveryTimeout := coordinatorCreateLeaseRecoveryTimeout
oldRecoveryInterval := coordinatorCreateLeaseRecoveryInterval
coordinatorCreateLeaseRecoveryTimeout = time.Second
coordinatorCreateLeaseRecoveryInterval = time.Millisecond
defer func() {
coordinatorCreateLeaseRecoveryTimeout = oldRecoveryTimeout
coordinatorCreateLeaseRecoveryInterval = oldRecoveryInterval
}()

puts := 0
gets := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodPut && r.URL.Path == "/v1/leases/cbx_abcdef123463":
puts++
if puts == 1 {
http.Error(w, `{"error":"provider_failure","message":"hetzner POST /servers: http 412"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusConflict)
_, _ = io.WriteString(w, `{"error":"fixed_lease_terminal","message":"lease id is bound to a terminal result for this create intent"}`)
case r.Method == http.MethodGet && r.URL.Path == "/v1/leases/cbx_abcdef123463":
gets++
http.NotFound(w, r)
default:
http.NotFound(w, r)
}
}))
defer server.Close()

cfg := baseConfig()
cfg.Provider = "hetzner"
cfg.TargetOS = targetLinux
cfg.Coordinator = server.URL
cfg.CoordToken = "user-token"
coord := mustNewCoordinatorClient(t, cfg)
backend := &coordinatorLeaseBackend{cfg: cfg, coord: coord, rt: Runtime{Stderr: &bytes.Buffer{}}}
_, err := backend.createCoordinatorLeaseWithProgressMode(
context.Background(), cfg, "ssh-ed25519 test", true,
"cbx_abcdef123463", "fixed-terminal", true,
)
if err == nil || !strings.Contains(err.Error(), "fixed_lease_terminal") {
t.Fatalf("err=%v, want same-intent terminal result", err)
}
if strings.Contains(err.Error(), "another create intent") {
t.Fatalf("err=%v, must not misclassify the same intent as conflicting", err)
}
if puts != 2 || gets != 0 {
t.Fatalf("puts=%d gets=%d, want one exact PUT recovery and no GET adoption", puts, gets)
}
}

func TestCoordinatorRecoveredProvisioningKeepsCreationLifetime(t *testing.T) {
t.Setenv("CRABBOX_OWNER", "test@example.com")
for _, fixed := range []bool{true, false} {
Expand Down
11 changes: 10 additions & 1 deletion worker/src/fleet.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5064,12 +5064,21 @@ export class FleetCoordinator {
(attempt &&
attempt.checkpointID === checkpointID &&
createAttemptMatchesLease(attempt, existing));
if (!sameOwner || !sameIntent || !sameAttempt || !leaseIsLive(existing)) {
if (!sameOwner || !sameIntent || !sameAttempt) {
return json(
{ error: "lease_id_conflict", message: "lease id is bound to another create intent" },
{ status: 409 },
);
}
if (!leaseIsLive(existing)) {
return json(
{
error: "fixed_lease_terminal",
message: "lease id is bound to a terminal result for this create intent",
},
{ status: 409 },
);
}
// Only the original attempt retains replay authority after claim consumption.
// Consume replacements before returning the lease, without releasing its provisioning fence.
if (
Expand Down
63 changes: 62 additions & 1 deletion worker/test/fleet.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22276,7 +22276,68 @@ describe("fleet lease identity and idle", () => {
request("PUT", "/v1/leases/cbx_abcdef123456", { headers, body }),
);
expect(terminalReplay.status).toBe(409);
await expect(terminalReplay.json()).resolves.toMatchObject({ error: "lease_id_conflict" });
await expect(terminalReplay.json()).resolves.toMatchObject({
error: "fixed_lease_terminal",
message: "lease id is bound to a terminal result for this create intent",
});
expect(creates).toBe(1);
});

it("reports a same-intent terminal replay after a definitive Hetzner 412", async () => {
const storage = new MemoryStorage();
let creates = 0;
const fleet = testFleet(storage, {
hetzner: fakeProvider(
() => {
creates += 1;
throw new HetznerProvisioningError(
"hetzner POST /servers: http 412: precondition_failed",
false,
false,
);
},
{ provider: "hetzner" },
),
});
const leaseID = "cbx_abcdef123477";
const body = {
leaseID,
slug: "fixed-412",
provider: "hetzner" as const,
serverType: "cx33",
sshPublicKey: "ssh-ed25519 fixed-412",
};
const headers = {
"x-crabbox-owner": "alice@example.com",
"x-crabbox-org": "example-org",
};

const first = await fleet.fetch(request("PUT", `/v1/leases/${leaseID}`, { headers, body }));
expect(first.status).toBe(500);
expect(storage.value<LeaseRecord>(`lease:${leaseID}`)).toMatchObject({
state: "failed",
serverID: 0,
cloudID: "",
provisioningResourceMayExist: false,
provisioningFailureRetryable: false,
failureError: "hetzner POST /servers: http 412: precondition_failed",
});

const replay = await fleet.fetch(request("PUT", `/v1/leases/${leaseID}`, { headers, body }));
expect(replay.status).toBe(409);
await expect(replay.json()).resolves.toEqual({
error: "fixed_lease_terminal",
message: "lease id is bound to a terminal result for this create intent",
});

const drift = await fleet.fetch(
request("PUT", `/v1/leases/${leaseID}`, {
headers,
body: { ...body, serverType: "cx43" },
}),
);
expect(drift.status).toBe(409);
await expect(drift.json()).resolves.toMatchObject({ error: "lease_id_conflict" });
expect(creates).toBe(1);
});

Expand Down
Loading