observability.md

Observability

Read this when you need to:

• debug a failed or slow run;
• check who used capacity this month;
• find a remote machine to SSH into for live inspection;
• correlate Actions hydration with the remote workspace.

Crabbox surfaces operational visibility through CLI commands, coordinator usage summaries, retained run history and logs, provider labels, GitHub Actions run links, and Worker logs. The single most useful habit is to keep the lease ID (cbx_...) and the run ID (run_...) together — most debugging paths start from one of those two identifiers. See Identifiers for the full ID scheme.

Lease state

Inspect a lease with status, list, and inspect:

crabbox status --id swift-crab
crabbox list --json
crabbox inspect --id swift-crab --json

Fields worth checking:

• lease ID and slug;
• owner and org;
• provider and server type;
• state (active, released, expired, failed);
• createdAt, lastTouchedAt, idleTimeoutSeconds, ttlSeconds, expiresAt;
• public address;
• SSH user and port;
• keep/delete behavior.

status accepts --wait (with --wait-timeout, default 5m) to block until the lease reaches a stable state. Provider machines are tagged with Crabbox metadata, so a cloud console can be correlated back to the lease that owns the instance.

Usage and cost

usage reports monthly cost and capacity summaries from the coordinator:

crabbox usage
crabbox usage --scope user --user alice@example.com
crabbox usage --scope org --org example-org
crabbox usage --scope all --json

Scope is user (default), org, or all; --month YYYY-MM selects the reporting month (defaults to the current UTC month). Reports include lease count, active lease count, elapsed runtime, estimated elapsed cost, reserved worst-case cost, and breakdowns by owner, org, provider, and server type. usage requires a configured coordinator. See Cost & usage for how the Worker computes rates and enforces budget caps.

Run history and logs

Coordinator-backed crabbox run creates a durable run record before leasing begins, appends lifecycle events as the CLI progresses, and finishes the run with exit code, timing, and retained command output. Inspect those records with:

crabbox history
crabbox history --lease cbx_...
crabbox history --owner alice@example.com --json
crabbox logs run_...
crabbox events run_...
crabbox attach run_...
crabbox results run_...

The CLI chooses the run ID before admission. If an admission response is lost, it can recover the same record using that ID and the original request. The coordinator accepts recovery only for the same initiating owner, organization, and request; it does not append another run.started event. Lease replacement uses the existing acknowledged attribution flow after admission, rather than changing the original create request.

Recovery never replays the remote command. The CLI refuses execution without a validated, still-starting run handle, and cancellation stops admission recovery. An older coordinator that lacks the caller-known admission route must be updated; the CLI does not fall back to anonymous record creation. Run IDs are single-invocation identities, and recovery relies on the existing record's retention. A new CLI invocation always uses a new ID.

• history lists recorded runs. Filter with --lease, --owner, --org, --state, and --limit (default 50). It is intended for command debugging, not unbounded log archival.
• logs <run-id> prints the retained remote stdout/stderr capture for a run. Use --tail N for the last N lines. Logs are stored in 64 KiB chunks with an 8 MiB cap, so very long output is truncated.
• events <run-id> prints the ordered phase and output events for a run. Filter with --type, --phase, --after N, and --limit (default 500).
• attach <run-id> follows events for a still-active run, polling every --poll interval (default 1s), so you can watch a run another CLI started.
• results <run-id> prints structured test-result summaries. See Test results.

The run ID is run_<hex>; the lease ID is cbx_<12 hex>. history, logs, events, attach, and results all accept the run ID as a positional argument or via --id.

Capturing run output locally

By default, coordinator logs and events go to the broker and live progress goes to your terminal. To redirect streams into local files:

• run --capture-stdout <path> writes remote stdout to the file and leaves coordinator logs/events plus lifecycle progress on stderr.
• run --capture-stderr <path> does the same for remote stderr.

Failed runs write a local failure bundle to .crabbox/captures/*.tar.gz by default. POSIX SSH-backed runs bundle only the current run's uploaded script file when still available, redacted env/config summaries, timing JSON, command stdout/stderr, common test/report/log paths, and a generic gateway log tail when present. Automatic capture excludes the retained .crabbox/scripts store from general report/log discovery: earlier uploads and arbitrary neighbors, including logs and XML reports, are not included. A run without an uploaded script includes no files from that store. Explicit artifact and download selections are independent, including --download-on-failure for eligible failures. Native Windows capture stays local-only. Blacksmith delegated runs bundle stdout/stderr plus timing and redacted env/config metadata. The stdout/stderr files captured inside automatic failure bundles are size-capped — pass --capture-stdout / --capture-stderr when you need a complete local stream file. Remote archive entries are confined to the bundle subtree; unsafe links and special files are omitted. --capture-on-fail is still accepted as a compatibility alias; failure bundles are saved automatically on non-zero exit regardless.

If the project capture destination is unwritable (permission denied or a read-only filesystem), Crabbox saves the bundle under its per-user state directory instead: $XDG_STATE_HOME/crabbox/captures/ when XDG_STATE_HOME is set, otherwise <user-config-dir>/crabbox/state/captures/. The failure-bundle local=... line reports the actual path. These fallback bundles persist outside the project; remove them when no longer needed. Unsafe destinations, including symlinks, and archive/read errors do not trigger this fallback. If both destinations fail, the warning includes both paths and the remote command's exit status is preserved. Explicit --capture-stdout and --capture-stderr destinations are never moved.

The bundle directory stays open and private through file creation, publication, and writing, so a concurrent replacement of its parent path cannot redirect the capture. Failed archive writes remove the incomplete bundle through that same directory.

Crabbox does not redact captured files. Treat every bundle and capture file as secret-bearing until you have reviewed it. On Unix-like hosts, Crabbox creates local captures, downloads, proofs, and failure bundles with owner-only file permissions (0600); new output directories use 0700.

To pull successful-run artifacts back without routing file bytes through the coordinator log, use --download remote=local (repeatable):

crabbox run --id swift-crab \
  --download dist/report.xml=./report.xml \
  -- pnpm test

Test results are stored as structured summaries when --junit, --results-auto, results.junit, or results.auto is configured. Optional --fail-on-test-failures / results.failOnFailures policy makes parsed JUnit failures turn an otherwise successful command into a failed run.

Successful Blacksmith runs additionally support --emit-proof <path>: when requested, Crabbox writes bounded stdout/stderr, timing, metadata, and the generated proof block as local run artifacts.

Timing and phases

--timing-json prints final timing as JSON, including sync phases and command phases. Failed runs add blockedStage and retryLikely when Crabbox can classify the likely blocker; the human-readable run summary prints the same values as blocked_stage and retry_likely.

runnerTotalMs is the CLI's observed wall time through route cleanup. runnerPhases is a timing-only breakdown whose accepted durations never exceed that total; Crabbox fills any remainder with unattributed or a delegated opaque phase. Coordinator phase vectors accept at most one positive integer duration for each of request, network_ready, bootstrap, and unattributed. Any malformed vector is discarded as a unit, after which valid legacy startup scalars can supply the breakdown. These fields are unsigned local telemetry: receipt v2 and its signing contract are unchanged.

Automatic run cleanup adds leaseStopped: true means the release owner confirmed the end of the recoverable lease, even when a terminal receipt remains locally. Retained resources and accepted but pending, failed, retry-scheduled, or otherwise unconfirmed cleanup report false and preserve failure recovery guidance. False does not certify a running or reachable resource. leaseStopError reports cleanup errors separately and may be present even after confirmed removal, for example when local finalization fails. Run finalization emits timing after cleanup and the failure digest. Its terminal order is timing record, timing JSON, local receipt persistence, then coordinator finish. Timing sink failures are terminal and are reflected in the local receipt and process exit; a failing CLI invocation can append its normal exit diagnostic. Timing artifacts lists only files already committed when the timing payload is emitted. Terminal receipt metadata is intentionally excluded because persistence happens afterward; successful persistence prints a separate artifact kind=receipt path=... bytes=... confirmation.

Commands can define their own phases by printing marker lines to stdout or stderr:

echo CRABBOX_PHASE:install
pnpm install --frozen-lockfile
echo CRABBOX_PHASE:build
pnpm build
echo CRABBOX_PHASE:test
pnpm test

Local benchmark ledger

--timing-record=default and --timing-record <path> append the final TimingReport to a local JSONL benchmark store. The default store is <CrabboxStateDir()>/timings.jsonl, which uses $XDG_STATE_HOME/crabbox when available and otherwise falls back to the user config directory's crabbox/state directory.

The ledger is explicit local state. Plain crabbox run does not write it. Recorded rows can include repo paths, remote workdirs, command display text, labels, artifact paths, and lease metadata because they preserve the timing payload. Use crabbox bench report to aggregate local observations, and treat insufficient sample counts as a prompt to collect more local evidence rather than as a provider ranking.

Timing rows, receipts, captures, and failure bundles are sensitive local correlation artifacts. They may contain repository and filesystem paths, workdirs, labels, artifact paths, lease IDs, and run IDs. Keep them private and review them before sharing.

The benchmark ledger records observed timing; it is not a deterministic budget gate. The future deterministic metric contract lives in Deterministic perf evidence and intentionally separates reproducible fuel/instruction-style counters from noisy wall-clock timings.

Forwarding live secrets

When local CRABBOX_ENV_ALLOW is set, run prints the variable names selected for forwarding plus safe metadata (whether secret-looking names are set and their value length). Values are never printed. Delegated Testbox providers report that forwarding is unsupported, because secrets belong in the provider workflow.

For one-off live secrets, avoid hand-written source boilerplate:

crabbox run \
  --env-from-profile ~/.project-live.profile \
  --allow-env API_TOKEN \
  --preflight \
  -- ./scripts/live-smoke.sh

Crabbox parses simple export NAME=value and NAME=value profile lines without executing the profile. Only names selected by --allow-env, env.allow, or CRABBOX_ENV_ALLOW are forwarded, and summaries show presence/length metadata for secret-looking names rather than values. On native Windows, the profile handoff file is uploaded as UTF-8 and imported with PowerShell UTF-8 decoding, so non-ASCII token material and paths do not depend on the system code page. See Env forwarding.

Running scripts instead of argv

Use --script <file> or --script-stdin when the remote command is more than a small argv:

crabbox run \
  --script ./scripts/provider-smoke.sh \
  --env-from-profile ~/.project-live.profile \
  --allow-env API_TOKEN \
  --timing-json

The script is uploaded under .crabbox/scripts/ in the remote workdir; only that run's uploaded file is selected for automatic failure bundles, not its directory or earlier uploads. POSIX SSH providers support this path; delegated providers reject it before reading stdin because they own command transport. Native Windows targets upload scripts too and run them through Windows PowerShell — use --shell for short snippets and --script <file.ps1> for longer runs. Crabbox writes Windows scripts as UTF-8 with a byte-order mark when the input has none, which keeps Windows PowerShell 5.1 from mojibaking non-ASCII script source:

crabbox run \
  --provider ssh \
  --target windows \
  --windows-mode normal \
  --static-host win-dev.local \
  --preflight \
  --script ./scripts/windows-smoke.ps1 \
  -- -Mode smoke

For PR debugging that should not inherit local dependency churn, run against a fresh remote checkout:

crabbox run \
  --fresh-pr example-org/my-app#123 \
  --script ./scripts/e2e-smoke.sh

Add --apply-local-patch only when the local diff should be applied on top of the PR checkout. PR URLs must be on github.com; non-GitHub and GitHub Enterprise PR URLs are rejected rather than rewritten to a public clone. The numeric shorthand uses the current repository's GitHub origin.

Remote debugging

SSH in for live process and filesystem inspection:

crabbox ssh --id swift-crab
crabbox inspect --id swift-crab --json

Useful remote checks:

crabbox-ready
test -f /var/lib/crabbox/bootstrapped
df -h
free -h
ps aux --sort=-%cpu | head

A lease created with --keep stays reachable over SSH until crabbox stop, idle expiry, or the TTL cap removes it. For one-shot E2E debugging, add --keep-on-failure: Crabbox releases successful runs normally, but on failure it prints the inspect, SSH, and stop commands for the exact failed lease and leaves idle/TTL expiry to clean it up later.

Preflight capability snapshot

For a concise pre-command capability snapshot, add --preflight:

crabbox run --id swift-crab --preflight -- pnpm test:changed

Preflight prints a target-specific capability snapshot from the same workdir the command will run in. It sources the Actions handoff env file when present and marks the workspace as raw or Actions-hydrated. A raw workspace that has Actions hydration configured prints the exact hydrate command suggestion and whether the selected provider/target supports hydration.

Preflight is a probe layer, not an installer. Missing tools print tool=missing; Crabbox does not run apt install, corepack prepare, bun install, or any other setup. Install toolchains through Actions hydration, a prebaked image, a devcontainer/Nix/mise/asdf setup, or the uploaded script/command itself.

The built-in probes cover common toolchains — git, tar, node, npm, corepack, pnpm, yarn, bun, and docker; opt-in go, cargo, cmake, uv, python, and python3 on POSIX, WSL2, and native Windows; and opt-in make on POSIX and WSL2 — plus target-specific probes such as sudo, apt, bubblewrap, powershell, execution_policy, longpaths, temp, and pwsh. Linux and WSL2 also support an opt-in raw_socket capability probe. Override the probe list per run:

crabbox run --preflight --preflight-tools python,python3 -- python3 -m pytest
crabbox run --preflight --preflight-tools default,cmake -- cmake --build build
crabbox run --preflight --preflight-tools raw_socket -- ./packet-tests

The opt-in CMake probe invokes the literal cmake --version command on POSIX, WSL2, and native Windows targets. It reports only the first output line or cmake=missing; the result is diagnostic only, so missing CMake does not block the workload or trigger installation or upgrades.

raw_socket reports direct when the execution user can open and immediately close socket(AF_INET, SOCK_RAW, IPPROTO_RAW), sudo when only the same bounded probe succeeds through sudo -n, unavailable otherwise, and probe_missing when neither python3 nor python exists. It never sends a packet or elevates the workload. This kernel capability is distinct from Python, Scapy, tcpdump, libpcap, or packet-capture-tool availability, and unsupported targets skip the probe.

Or per repository:

run:
  preflightTools:
    - default
    - cmake

Actions hydration

crabbox actions hydrate populates a lease's workspace by driving the repo's configured workflow setup over SSH by default, waiting for a ready marker. The marker path is the key local correlation point; --github-runner instead registers the box as a self-hosted runner and reports the workflow run URL.

crabbox actions hydrate --id swift-crab --workflow .github/workflows/hydrate.yml
crabbox inspect --id swift-crab --json

A hydrated run writes non-secret handoff data for later crabbox run --id swift-crab commands. Secrets and OIDC tokens stay workflow-step scoped unless the workflow intentionally writes its own short-lived handoff. See Actions hydration.

Live provider debugging

For live provider or end-to-end test runs, prefer an Actions-hydrated lease when tests need Node, pnpm, Docker services, repository secrets, or GitHub OIDC:

crabbox warmup --provider aws --class beast --keep
crabbox actions hydrate --id swift-crab --workflow .github/workflows/hydrate.yml
mkdir -p .crabbox/logs
CRABBOX_ENV_ALLOW=OPENAI_API_KEY,OPENAI_BASE_URL \
  crabbox run --id swift-crab \
  --preflight \
  --timing-json \
  --capture-stdout .crabbox/logs/live-provider.stdout.log \
  --capture-stderr .crabbox/logs/live-provider.stderr.log \
  --keep-on-failure \
  --shell 'echo CRABBOX_PHASE:install; pnpm install --frozen-lockfile; echo CRABBOX_PHASE:test; pnpm test:live'

For Blacksmith Testbox comparison runs, keep secrets in the Testbox workflow environment. Crabbox reports that CRABBOX_ENV_ALLOW forwarding is unsupported, because Blacksmith owns command execution:

CRABBOX_ENV_ALLOW=OPENAI_API_KEY \
  crabbox run --provider blacksmith-testbox \
  --blacksmith-workflow .github/workflows/ci-check-testbox.yml \
  --blacksmith-job test \
  --preflight \
  -- pnpm test:live:providers

Worker logs

When the coordinator path fails before SSH is established, check Worker logs and Durable Object errors. The symptoms usually fall into a few groups:

• auth failure;
• cost limit rejection;
• provider quota or capacity rejection;
• provider API failure;
• Durable Object alarm or state-transition bug.

Keep the lease ID, owner, org, provider, class, and request time on hand when comparing CLI output to Worker logs. See How it works and the coordinator reference for the request flow.

Gaps

Crabbox observability is sufficient for maintainer operations but is not yet a full analytics product. Notably missing:

• alerting on budget or failure-rate thresholds;
• a dashboard UI.