Skip to content

Latest commit

 

History

History
165 lines (142 loc) · 13.2 KB

autodev-runbook.md

File metadata and controls

165 lines (142 loc) · 13.2 KB

autodev runbook

autodev turns a labelled GitHub issue into a PR written by Claude Code in a sandboxed LSF job, revises it on request, and merges it after a human approves. Design: agentic-dev-infrastructure.md.

How people use it

you do autodev does
add autodev:go to an issue (needs write access) runs an agent round, opens PR autodev/issue-N, labels the issue autodev:review
review with Request changes, or comment /revise <what> (write access) runs another round on the same branch (up to max_revisions)
approve the PR (or comment /approve <sha> under a personal token, after the latest round's summary) squash-merges that exact commit, closes the issue (a change request you made earlier on the same head is withdrawn by your approval; anyone else's still blocks)
close the PR stops; re-add autodev:go to start over (closing the issue does not stop a running workflow)
sees autodev:needs-human read the failure comment, fix, re-add autodev:go (an open PR is resumed)

Trust boundaries:

  • Only people with write access can trigger (:go), approve, or request changes, and only their comments reach the agent's prompt. The issue body is included as-is, so edits to it after labelling are a known ceiling.
  • Agent commits touching .github/ are refused.
  • Permanent failures stop the round immediately; transient GitHub/csub/git errors retry for up to 6 h.
  • Agent summaries and log tails are posted with secret-shaped strings (sk-ant-…, ghp_…, JWTs, the worker's own GitHub token) replaced by [redacted]; that is a pattern filter, not a guarantee.
  • The Temporal dev server's gRPC port and web UI are unauthenticated: anyone with a login on the host can start, signal or terminate workflows. Run autodev on a host you don't share with untrusted users.

Run it as yourself (testing)

  1. Prerequisites P1-P4 in the plan. csub is pinned to krokicki/csub branch deploy-all (has the claude request field); pixi install gets it.
  2. cp mia.yaml.example mia.yaml, keep the autodev service and name it autodev-<you> in services (the deployment is the service name, so your labels, branches and workflows stay apart from the service and from colleagues), and set its paths.work_root under your csub allowed_roots. Keep work_root and paths.state_dir apart: work_root must not be an ancestor of state_dir, because work_root is bound read-write into the sandbox, and <state_dir>-host (the broker log) and temporal.state_dir from mia.yaml (the Temporal binary and database) must stay outside every writable bind.
  3. gh auth token > ~/.config/autodev/token && chmod 600 ~/.config/autodev/token (or a fine-grained PAT with Contents, Pull requests and Issues read/write, and Checks and Commit statuses read), and set github.token_file: ~/.config/autodev/token. The sandboxed worker reads the token from this file (the sandbox masks .env); host-side commands like autodev check also accept .env / $GITHUB_TOKEN.
  4. pixi run mia exec autodev check: the temporal row is BAD until step 5 has started the dev server; fix any other BAD row first.
  5. pixi run mia exec autodev init-labels, then pixi run mia up. It starts, in tmux, a Temporal dev server (session temporal) and the pixi env's csub-broker (reading your ~/.config/csub/broker.toml policy) on the host, and the worker inside the sandbox. Re-run pixi run mia exec autodev check: now every row ok.
  6. Open a small issue ("add a --version flag"), add autodev-<you>:go, watch pixi run mia attach <service>.

Manage the services listed in mia.yaml: pixi run mia ps, mia logs <service> [worker|broker] (-n N, -f to follow), mia logs temporal, mia attach <service>, mia down [<service>...], mia restart [<service>...], mia up [<service>...]. Without a service name each command acts on everything, Temporal first on the way up and last on the way down.

Files: the Temporal CLI binary, temporal.db and its log live in temporal.state_dir from mia.yaml (default ~/.local/state/temporal); the broker log is in <state_dir>-host/logs, which the sandbox never binds; worker logs are in <state_dir>/logs.

buffer-api (mia-buffers serve) keeps its SQLite ledger at <local_root>/service/buffers.sqlite (host-local disk, never NFS) and writes backups to <state_dir>/backups/ every keeper.minutes; worker writes need the token in service.token_file. If the ledger is missing but <state_dir>/backups/ holds backups, serve refuses to start and names the newest: copy it to <local_root>/service/buffers.sqlite to restore it. After the first mia up, fill the registry: mia exec mia-buffers catalog sync (the catalog records; --ref <sha> for a particular commit) and mia exec mia-buffers datasets sync (the prepared rows, from storage_root).

Under a personal token you are both the bot and the reviewer, so GitHub won't let you approve your own PR: comment /approve <first 7+ chars of the head sha> instead. Only /approve comments posted after the bot's latest round summary count, so a short prefix approved earlier cannot approve a later head.

Run it as the group service

This is the per-account checklist. Where the pieces run in production (Temporal in Kubernetes, workers under systemd on a cluster-attached host) is docs/deployment.md.

  1. Service account on the cluster with rootless podman / bwrap set up (agentic-sandbox README), a csub broker installed for it (csub deploy/INSTALL.md), and an LSF billing group in its broker policy.
  2. Broker policy (~svc/.config/csub/broker.toml): [agents] claude = ["api.anthropic.com", "claude.ai", "platform.claude.com"]; allowed_hosts ⊇ agent.allow_hosts; allowed_roots contains paths.work_root; the CPU [queues.<name>] tables and cpu_short_queue/cpu_default_queue are named after queues bqueues lists (at Janelia short and local; a made-up name fails every submit with No such queue). Config rejects GitHub hosts in agent.allow_hosts, and agent.extra_mounts must be absolute and end in :ro.
  3. Claude for the service account: claude auth login as that account (a team seat). agent.api_key_env is refused by mia up for now.
  4. GitHub App owned by the org, installed on the repo, permissions: Contents RW, Pull requests RW, Issues RW, Checks R, Commit statuses R, Metadata R. Not Workflows, not Administration. Private key at ~svc/.config/autodev/app.pem (mode 600); github.auth: app, app_id, installation_id, private_key_path. Step by step: docs/github-app.md.
  5. Branch protection / ruleset on main: 1 approving review required, stale approvals dismissed on push, the App not in the bypass list. This is the real gate; autodev's own approval check is a pre-check. Add .github/CODEOWNERS (inside .github/, so agents cannot edit it) covering autodev/, mia_ops/ and .github/, so changes to the gates need a named human.
  6. As the service account: pixi run mia exec autodev init-labels, pixi run mia up, then pixi run mia exec autodev check.

The judges: from shadow to active

Every human gate can have a judge (spec docs/specs/2026-11-09-autonomous-gates.md): a second model, read-only, over the gate's evidence. It earns authority in three steps, per gate, and nothing turns on by itself.

  1. Off (judge.<gate>.mode: off, the default): no judge runs.

  2. Shadow (mode: shadow): set it in mia.yaml (a judge.model other than the deployment's agent.model is a precaution, not a rule), pixi run mia restart. The judge runs on every live round: autodev's review judge after each pushed head (a PR comment headed "Review judge", no /approve line), mia-data's crop judge after each passing set (a comment under the crop table). Both write an assessment by: judge; the person's verdict still decides. Watch the Decisions tab of the buffers page: per gate, how many subjects were judged and whether the newest human verdict agrees. Build an evaluation set from the human-reviewed rounds on disk and score the judge offline:

    pixi run mia exec mia-data collect-judge-set /path/to/eval-crops
    pixi run mia judge eval crops /path/to/eval-crops            # per category: agree, false approve, false reject, abstain
    

    Twenty live rounds without a false approval, and none on the evaluation set, is the bar the spec sets (autonomy.min_shadow_rounds).

  3. Active (mode: active, not available yet): the judge's approve authorizes within policy; Revoke on the dataset's or the spec's page withdraws it at any time, mia pause stops every workflow at its next job.

Known gaps (deliberate, for later)

  • No cancel command: temporal workflow terminate -w <id> stops a workflow but leaves its job; csub kill <job>.
  • The agent's tests need tools inside the job (e.g. pixi on path_prepend + extra_mounts, conda/pypi in allow_hosts); nothing installs them for you.
  • CI failures after approval only escalate; there is no automatic fix-CI round.
  • Claude sessions are not resumed across rounds; each round starts fresh from the PR diff + feedback.
  • Temporal is a dev server under the running account; a shared production server is a later step.
  • autodev/workflows.py and autodev/decide.py are workflow code: changing their decisions or command order breaks in-flight workflows on replay. test_replay_recorded_history replays a recorded run (tests/fixtures/autodev_issue_history.json) and fails on such a change. If the change is intentional, guard it with workflow.patched() (there is no Worker Versioning yet) so runs in flight keep their old path, then re-record: AUTODEV_RECORD_HISTORY=1 pixi run pytest tests/test_autodev_workflows.py -k recorded_history.
  • [bot] logins are treated as having no permission (never looked up), so no bot can trigger, approve or revise.
  • The live end-to-end run (issue, PR, /revise, /approve) has not been done yet.

mia-data

Prerequisites P1-P4 (plan): P1 repos mia-workflow-data and mia-snippets with the GitHub App installed; P2 shared dirs for state_dir, work_root, storage_root (all under paths) visible from compute nodes; P3 csub broker policy ([broker] in broker.toml: env_allow includes "MIA_HARNESS_*"; allowed_roots covers harness.dir, work_root and storage_root, none of them under $HOME, which csub never mounts; [agents], data hosts, redirect probe; CPU queue tables named after real bqueues queues; mia exec mia-data check names what is missing); P4 see the plan.

  1. cp mia.yaml.example mia.yaml (if you have not), fill in the shared paths, the mia-data block, harness.dir and a github.token_file (or app key). paths.work_root and paths.storage_root take a path or {path, url}; with a url (the folder in Fileglancer) the timeline entries link the round's workspace and the dataset's crops folder. The data repo needs at least one commit on its default branch: the galleries branch is created through the trees API, which answers 409 on an empty repo.
  2. pixi run mia exec mia-data check verifies repos, Temporal, csub, dirs and the harness pin (and that cd harness && pixi install was done in harness.dir). Run on the host, mia exec mia-data check and mia exec mia-data test-pr need the mia-data broker: CSUB_TRANSPORT=unix CSUB_SOCKET=/scratch/$USER/mia-data/csub.sock pixi run mia exec mia-data check.
  3. pixi run mia up starts Temporal (unless mia.yaml says temporal.managed: false, in which case it only checks that the configured cluster answers), then each service: it ensures the service's namespace, starts its broker and sandboxed worker, and applies its schedule. Re-running it while a session exists only re-applies that service's schedule, so a run that failed partway is repaired by running it again. Uses temporal from PATH; a managed server downloads one into temporal.state_dir/bin otherwise. Manage with mia ps, mia logs mia-data [worker|broker] (-n N, -f), mia attach mia-data, mia down mia-data, mia restart mia-data (needed after a config change; wait for running rounds to finish first). mia down with no service stops everything, Temporal last; naming a service never touches Temporal.
  4. pixi run mia exec mia-data init-labels creates the data:* stage labels the mirror sets. Datasets are started from the buffers page (Start); an issue, a copy of the dataset's timeline, appears in the data repo the first time its dataset has something to say.
  5. pixi run mia exec mia-data test-pr <pr> --binding <file> --record <id> (all required) runs the crop check on a snippet PR's head with that binding and catalog record, and reports on the PR.
  6. Buttons on a dataset's page: Start starts a round; Approve publishes the crops under review (and merges the snippet PR that made them); Reject… asks for your feedback and sends it to the agent fixing the snippet, and the next crops come back for review; Cancel the round ends a round in progress.