ASCII Box reused IP is blocked by provider-wide known_hosts state

[shunkakinoki]

Summary

ASCII Box SSH targets use the provider key directory's shared known_hosts file rather than UseLeaseKnownHosts. When ASCII Box assigns a previously used public IP to a new VM with a different host key, a new crabbox run remains in ssh-auth readiness until its 20-minute timeout.

Reproduction

1. Acquire and stop an ASCII Box lease.
2. Acquire a later lease that receives the same public IP with a different host key.
3. Run a command through Crabbox.

Observed with Crabbox revision da6484dc0ed73003ccc9850f40e4f170d7795590; inspection of current origin/main (c9cdc35987531d6f71c317693cb18cbbb0dbe8d9) shows ASCII Box still does not call UseLeaseKnownHosts.

The provider reports the Box ready. box ssh succeeds, and direct SSH with the ASCII Box key succeeds. Crabbox's exact probe fails with REMOTE HOST IDENTIFICATION HAS CHANGED, identifying the stale entry in the shared ASCII Box known_hosts file. Removing only that IP with ssh-keygen -R <ip> -f <ascii-box-known-hosts> lets the already-running crabbox run recover immediately without starting a replacement Box.

Expected

ASCII Box host trust should be scoped to the Crabbox lease, as it is for providers that call UseLeaseKnownHosts, so ephemeral IP reuse cannot poison later leases while strict host-key verification remains enabled.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed September 2, 2026, 4:05 PM ET / 20:05 UTC.

Summary
Keep open: current main constructs ASCII Box SSH targets without a lease-scoped known-hosts path, so SSH falls back to a file beside the provider-wide key. This is a small, source-proven provider-adapter bug suitable for a focused repair PR.

Reproducibility: yes. at source level: an ASCII Box target leaves KnownHostsFile empty, causing the shared SSH layer to select known_hosts beside the shared provider key. The live recycled-IP allocation was not executed in this read-only review.

Next step
Current main directly proves a narrow adapter omission, and the shared helper plus focused provider tests provide a clear one-PR repair path.

Review details

Best possible solution:

Use the existing lease-scoped known-hosts helper for every ASCII Box SSH target, including readiness/run and guarded cleanup, with regression coverage that proves separate leases cannot share the provider key directory's host trust.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: an ASCII Box target leaves KnownHostsFile empty, causing the shared SSH layer to select known_hosts beside the shared provider key. The live recycled-IP allocation was not executed in this read-only review.

Is this the best way to solve the issue?

Yes: calling the existing provider-neutral lease-scoping helper at the ASCII Box adapter boundary is the narrowest repair and preserves strict host-key verification.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9ab5b3ba7e05.

Label changes

Label changes:

• add P2: A stale host-key entry can prevent an affected ASCII Box lease from becoming usable, but the failure is limited to this provider path.
• add impact:other: The issue blocks remote command execution for a lease whose recycled public IP has a different SSH host key.
• add issue-rating: 🦞 diamond lobster: Current issue advisory state selects this label.
• add clawsweeper:source-repro: Current issue advisory state selects this label.
• add clawsweeper:queueable-fix: Current issue advisory state selects this label.
• add clawsweeper:fix-shape-clear: Current issue advisory state selects this label.

Label justifications:

• P2: A stale host-key entry can prevent an affected ASCII Box lease from becoming usable, but the failure is limited to this provider path.
• impact:other: The issue blocks remote command execution for a lease whose recycled public IP has a different SSH host key.

Evidence reviewed

Acceptance criteria:

• go test ./internal/providers/asciibox
• go test ./internal/cli

What I checked:

• Current ASCII Box target omits lease host trust: leaseFromBox builds the target and immediately starts its SSH readiness check; boxSSHTarget supplies a shared provider key but does not call UseLeaseKnownHosts or set KnownHostsFile. (internal/providers/asciibox/backend.go:349, 9ab5b3ba7e05)
• SSH fallback is provider-wide for this target: When KnownHostsFile is empty, the shared SSH layer uses a known_hosts file beside target.Key; ASCII Box derives that key from its provider CLI home, not the lease directory. (internal/cli/ssh.go:1174, 9ab5b3ba7e05)
• Existing contract and repair primitive: The documented SSH contract says reused provider IPs must not share host-key state, and UseLeaseKnownHosts sets the target path to the protected per-lease known_hosts file. (docs/features/ssh-keys.md:71, 9ab5b3ba7e05)
• Recent relevant adapter history: Recent main history includes the dynamic ASCII Box SSH endpoint change, confirming Shun Kakinoki as a recent contributor to this exact target-construction path. (internal/providers/asciibox/backend.go:520, 006094a107ad)
• Vision fit: VISION.md explicitly keeps provider-specific lifecycle rules in adapters and calls for concrete provider bug fixes with regression coverage. (VISION.md:3, 9ab5b3ba7e05)

Likely related people:

• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• shunkakinoki: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[shunkakinoki]

Additional downstream evidence from September 4: this still blocks validation with installed Crabbox revision da6484dc0ed73003ccc9850f40e4f170d7795590.

• Two validation attempts reached ssh-auth but exhausted the downstream three-minute provisioning deadline before the delegated command began.
• Direct non-interactive SSH using the same managed authentication key succeeded against the affected Box while readiness was stalled.
• Read-only inspection found different ED25519 fingerprints for that endpoint in the provider-wide and direct-SSH known-hosts stores. This supports the stale endpoint-pin diagnosis; we did not remove pins, disable host-key verification, or restart the already-stopped Box to manufacture another reproduction.

The repair is already being handled by #1785, so we will depend on that PR rather than submit an overlapping implementation. Its reported real SSH execution and retained-lease reuse proof directly cover the downstream failure mode. We understand the separate native deletion-completion hold is still unresolved; successful SSH is not cleanup proof.

A remaining diagnostic improvement worth tracking separately: a deterministic host-key mismatch should surface as a trust failure promptly, rather than remain indistinguishable from transient SSH readiness until the entire provisioning deadline. That must retain changed-key rejection and must not automatically delete or re-enroll existing pins.