feat(azure): use GA full caching for ephemeral OS disks

[jwmoss]

What Problem This Solves

Azure full caching is generally available, but Crabbox still exposes it as a separate preview disk mode.

User Impact

--azure-os-disk ephemeral now enables GA full caching in the CLI and coordinator, including direct fixed-ID leases.

Migration: replace ephemeral-preview with ephemeral in flags, config, and environment variables. The removed value returns a clear replacement error. Existing ephemeral settings now require full-caching support: at least eight vCPUs and a supported VM family. Smaller VMs and Fsv2 no longer qualify. Managed disks remain the default; ephemeral disks still cannot provide native snapshot checkpoints. No new credentials are required.

Why This Change Was Made

The preview implementation in #186 introduced ephemeral-preview and a raw ARM request because the SDK lacked the full-caching field. Azure now documents GA support, and Crabbox's pinned Go SDK exposes that field.

Use the native SDK and matching coordinator API, remove the preview request/polling workaround, and apply GA family and vCPU requirements. Fixed-ID creates retain If-None-Match: * and replay protection.

Evidence

Selected Azure evidence appears below. Account IDs, addresses, resource identities, local paths, and credential material are omitted.

• Live scripts/live-smoke.sh: passes on Linux x64, Standard_D8ads_v6, Ubuntu 26.04, in eastus. Covers provision, readiness, inspect, SSH, cache inventory, sync, command execution, and stop. ARM confirms enableFullCaching: true, Local, NvmeDisk, and ReadOnly.
• Fixed-ID replay preserves one immutable VM. All task-created Azure resources are deleted; the final resource-group existence check returns false.
• Local Docker E2E passes, including four concurrent built-CLI warmups, execution, and cleanup. Focused Azure tests and the built-CLI help contract pass with the race detector.
• Coordinator suite: 3,415 pass, 15 skip. Go vet/build, Worker and Node typechecks/builds, scoped lint/format checks, and documentation checks pass.

The full Go race gate is not a pass: 101 packages pass, but the CLI package hits localhost SSH failures and its 20-minute timeout. The static SSH failure also reproduces on unchanged upstream. One unchanged script test fails under macOS Bash 3.2; its full 44-test file passes in Linux Docker.

Live proof covers direct Linux x64. Coordinator coverage uses HTTP fixtures; Windows and ARM64 are not live-tested here. The ARM flag proves Azure accepted full caching, not completion of the background cache copy. This PR does not deploy or release Crabbox.

Sanitized Azure readback and command output

Selected fields from the raw ARM 2026-04-01 response:

{
  "size": "Standard_D8ads_v6",
  "state": "Succeeded",
  "osDisk": {
    "osType": "Linux",
    "diskSizeGB": 30,
    "createOption": "FromImage",
    "caching": "ReadOnly",
    "diffDiskSettings": {
      "enableFullCaching": true,
      "option": "Local",
      "placement": "NvmeDisk"
    },
    "managedDisk": {
      "storageAccountType": "StandardSSD_LRS"
    }
  }
}

Selected terminal output from the successful live smoke:

warmup complete total=1m33.848s
azure-full-caching-e2e
command complete in 2.27s total=8.699s

The command exits zero. Fixed-ID replay preserves the same resource ID and immutable VM ID. The final resource-group existence check returns false.

Stop reports SSH exit-255 warnings for guest-side cleanup; Azure resource deletion succeeds. The disk settings do not measure background cache-copy completion.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge. Reviewed September 25, 2026, 4:05 PM ET / 20:05 UTC.

ClawSweeper review

What this changes

The PR makes Azure ephemeral OS disks use GA full caching through the Go SDK and coordinator, enables that mode for direct fixed-ID leases, and updates SKU validation, tests, and documentation.

Merge readiness

⛔ Blocked before merge - 6 items remain

The GA Azure path is useful and has credible live CLI proof, but current main and v0.66.0 still support both older disk modes. This PR needs an explicit compatibility decision because it rejects one shipped setting and changes the provisioning requirements of another.

Priority: P2
Reviewed head: b2f67e8d6f36b2c66f48a7e39bc96ee991df9714
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Strong direct Azure proof supports the implementation, while two released-setting compatibility breaks prevent a merge-ready rating.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.
Patch quality	🦐 gold shrimp (3/6)	2 actionable review findings remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.
Evidence reviewed	7 items	Introduced disk-mode behavior: The introduced patch rejects ephemeral-preview and makes ephemeral request full caching in both the CLI and coordinator. Shipped compatibility contract: Current main documents ephemeral as partial local-disk mode and ephemeral-preview as the full-caching mode; v0.66.0 contains the same contract. Release check: The latest release still documents both accepted disk modes, so the PR changes a released setting rather than removing an unshipped preview value.
Findings	2 actionable findings	[P1] Preserve the released preview setting during upgrade [P1] Keep existing ephemeral leases eligible
Security	None	None.

How this fits together

Crabbox turns CLI flags or coordinator lease requests into Azure VM settings. The Azure provider selects a VM size and OS disk mode, then sends a VM creation request that determines whether the lease can provision and support checkpoints.

flowchart LR
  A[CLI flags or lease request] --> B[Azure disk setting]
  B --> C[VM size selection]
  C --> D[Eligibility check]
  D --> E[Azure VM request]
  E --> F[Provisioned lease]
  F --> G[Run or checkpoint]

Loading

Decision needed

Question	Recommendation
Should GA full caching become a new opt-in Azure disk mode while the two shipped modes remain compatible, or may this release break existing ephemeral and ephemeral-preview settings?	Preserve shipped modes: Add a distinct GA choice and test that existing ephemeral and ephemeral-preview settings continue to provision as before.

Why: The PR deliberately changes a released configuration contract, and the discussion records no maintainer acceptance of that upgrade impact.

Before merge

• Preserve the released preview setting during upgrade (P1) - The coordinator now rejects ephemeral-preview before lease creation, and the CLI rejects the same shipped flag or config value. Existing deployments therefore stop provisioning until an operator edits their settings. Keep an accepted compatibility path or obtain explicit approval and upgrade proof for this break.
• Keep existing ephemeral leases eligible (P1) - A released ephemeral setting previously used partial caching, including on supported smaller and Fsv2 VMs. This change forces full caching and the new SKU limits, so those existing lease requests fail or change behavior on upgrade. Add an explicit GA opt-in or secure maintainer approval with fresh-install and upgrade coverage.
• Resolve merge risk (P1) - Existing flags, config files, and coordinator defaults set to ephemeral-preview will stop lease creation until operators change them.
• Resolve merge risk (P1) - Existing ephemeral settings on smaller or Fsv2 VMs will fail provisioning, while eligible settings silently switch from partial to full caching; no upgrade acceptance or compatibility proof is recorded.
• Complete next step (P2) - Resolve the Azure disk-mode upgrade policy, repair or explicitly approve both compatibility breaks, and verify fresh and existing configurations before merge.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

• [P1] Preserve the released preview setting during upgrade — worker/src/config.ts:582-583
• [P1] Keep existing ephemeral leases eligible — internal/cli/azure.go:983-987

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Code and test delta	production +59/-288 lines; tests +173/-88 lines	The patch removes the preview request workaround and adds focused request tests, but also changes the released settings contract.

Merge-risk options

Maintainer options:

1. Keep existing configurations working (recommended)
   Introduce GA full caching as an explicit choice and cover both fresh configuration and upgrades from the two released modes.
2. Own a breaking transition
   A maintainer can approve the changed meanings after reviewing upgrade proof and the required operator migration.

Technical review

Best possible solution:

Preserve both shipped disk-mode meanings during upgrade, add an explicit GA full-caching choice, and verify fresh and existing settings through CLI and coordinator provisioning.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a feature migration. Current main and v0.66.0 clearly define the two existing settings, and the introduced validation shows where each upgrade break occurs.

Is this the best way to solve the issue?

No. The GA request path is supported by live proof, but assigning it to the shipped ephemeral value and rejecting ephemeral-preview is not the safest upgrade path without explicit maintainer approval.

Full review comments:

• [P1] Preserve the released preview setting during upgrade — worker/src/config.ts:582-583
  The coordinator now rejects ephemeral-preview before lease creation, and the CLI rejects the same shipped flag or config value. Existing deployments therefore stop provisioning until an operator edits their settings. Keep an accepted compatibility path or obtain explicit approval and upgrade proof for this break.
  Confidence: 0.96
• [P1] Keep existing ephemeral leases eligible — internal/cli/azure.go:983-987
  A released ephemeral setting previously used partial caching, including on supported smaller and Fsv2 VMs. This change forces full caching and the new SKU limits, so those existing lease requests fail or change behavior on upgrade. Add an explicit GA opt-in or secure maintainer approval with fresh-install and upgrade coverage.
  Confidence: 0.95

Overall correctness: patch is incorrect
Overall confidence: 0.91

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against fc3cb7d44ef1.

Labels

Label changes:

• add P2: This is a bounded Azure feature update with a material upgrade decision for users of optional ephemeral disks.
• add merge-risk: 🚨 compatibility: Merging would reject a released disk-mode value and change the meaning and eligible VM sizes of another.
• add proof: sufficient: Contributor real behavior proof is sufficient. The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.

Label justifications:

• P2: This is a bounded Azure feature update with a material upgrade decision for users of optional ephemeral disks.
• merge-risk: 🚨 compatibility: Merging would reject a released disk-mode value and change the meaning and eligible VM sizes of another.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.
• proof: sufficient: Contributor real behavior proof is sufficient. The changed CLI Azure VM-create path was exercised against a real Linux x64 Azure VM: the PR records the full-caching ARM readback, successful command output, fixed-ID replay, and cleanup. Coordinator request-shape fixtures supplement that proof; the record does not establish live coordinator or upgrade behavior. No stored-data schema contract changes.

Evidence

What I checked:

• Introduced disk-mode behavior: The introduced patch rejects ephemeral-preview and makes ephemeral request full caching in both the CLI and coordinator. (internal/cli/azure.go:381, b2f67e8d6f36)
• Shipped compatibility contract: Current main documents ephemeral as partial local-disk mode and ephemeral-preview as the full-caching mode; v0.66.0 contains the same contract. (docs/providers/azure.md:170, fc3cb7d44ef1)
• Release check: The latest release still documents both accepted disk modes, so the PR changes a released setting rather than removing an unshipped preview value. (docs/providers/azure.md:170, 145cd6eca3c6)
• Prior feature provenance: The earlier full-caching preview was merged in feat: add Azure ephemeral full-caching preview #186; the current PR is a distinct GA migration of that feature. (internal/cli/azure.go, 190257b0f609)
• Current provision paths: CLI Azure config normalization rejects the old value; coordinator lease normalization does likewise, including for its configured default. (worker/src/config.ts:248, b2f67e8d6f36)
• Real behavior proof: The PR body records a live Linux x64 Azure run on Standard_D8ads_v6 with an ARM readback showing enableFullCaching=true, command output, fixed-ID replay, and resource cleanup. Coordinator request coverage uses HTTP fixtures. (internal/cli/azure.go:987, b2f67e8d6f36)

Likely related people:

• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• jwmoss: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Resolve the disk-mode compatibility choice and demonstrate fresh-install and upgrade behavior for both CLI and coordinator settings.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

Thanks @jwmoss! This landed via #2567 (merged as 2d694e6 and deployed to the production coordinator, with a passing post-deploy live smoke), rebuilt on current main with your implementation and tests preserved and co-author credit in the commit. Follow-ups on top: constrained-vCPU eligibility per Microsoft's GA prerequisites, explicit coordinator rejection of the legacy ephemeral-preview value (verified live: released CLIs now get http 400 invalid_azure_os_disk ... use ephemeral), config show migration, and upgrade notes for the next release. Closing this original in favour of the superseding PR.