Skip to content

Worktree isolation exists only as markdown guidance - all spawned agents share one working tree with no lease or lock #1717

Open

Description

@swigerb

Summary

Squad documents git worktree isolation for parallel agents in several templates and skills, but no runtime code creates, leases, or cleans up a worktree. Every spawned agent runs in the same working tree with no lock or mutex. This appears to be a regression against #525 / #529, which were closed as implemented.

Evidence

The guidance is markdown only. Every git worktree add in the repo lives in a template or skill file handed to the model as a prompt:

packages/squad-sdk/templates/issue-lifecycle.md:136
packages/squad-cli/templates/issue-lifecycle.md:136
packages/squad-sdk/templates/worktree-reference.md:108-109
packages/squad-cli/templates/worktree-reference.md:108-109
packages/squad-sdk/templates/skills/git-workflow/SKILL.md:83-84
packages/squad-cli/templates/skills/git-workflow/SKILL.md:83-84

No source file creates one. In packages/**/src/, every worktree hit is either a comment or a detection helper for the case where Squad is already running inside one:

  • squad-cli/src/cli/core/detect-squad-dir.ts:47 - detection fallback
  • squad-cli/src/cli/core/init.ts:210-216 - warns if a worktree is detected
  • squad-cli/src/cli/commands/doctor.ts:548 - resolves hooks path correctly under worktrees
  • squad-sdk/src/resolution.ts:180,218,285 - .git-file handling

packages/squad-sdk/src/platform/ralph-commands.ts - the file #529 targeted - contains no worktree reference at all.

Spawns share one directory. watch/agent-spawn.ts:190-200 passes a single cwd straight to spawn(), and every caller passes context.teamRoot. execute.ts:135 builds one prompt containing all eligible issues and execute.ts:103 instructs the agent to "spawn agents for ALL actionable issues simultaneously" - concurrent work, one tree.

There is no lease, lock file, or mutex guarding the tree. #1014 ("detect concurrent background spawns on a shared worktree ... real-world git-race incident observed") documents this failure mode occurring in practice.

Impact

  • Concurrent agents run git checkout / git commit against a shared index. Interleaving produces cross-contaminated branches, lost edits, and commits containing another agent's changes.
  • The failure is nondeterministic and only shows up under the parallelism the prompts actively encourage, which makes it hard to reproduce and easy to misattribute to the model.
  • Isolation currently depends on the model choosing to follow markdown. That is a reasonable way to express preference, but not a way to enforce a safety invariant.

Proposal

Move isolation from prompt guidance into the spawn flow:

  1. When dispatching more than one work item concurrently, create git worktree add <path> -b squad/<id>-<slug> <base> per item and set that path as the child's cwd.
  2. Hold a lease file per worktree so a second process cannot claim the same item, and so crashed runs can be detected and reclaimed.
  3. Serialise any operation that must touch the shared tree behind a single mutex.
  4. Clean up on completion, with git worktree prune recovery for the crash case.
  5. Fall back to sequential execution in the same tree when worktree creation is unavailable, rather than silently running parallel agents in one tree.

Keeping the markdown guidance is still useful for the in-iteration behaviour, but the invariant itself should hold whether or not the model reads it.

Activity

  1. swigerb commented on Sep 16, 2026

    @swigerb
    ContributorAuthor

    Confirming this is still present in v0.12.0, and adding one detail that I think sharpens the "isolation is markdown only" framing, plus a mitigation that works today.

    The guidance can be absent entirely, not merely unenforced

    watch/capabilities/execute.js prefixes the watch agent's charter to every spawned session:

    // Load Ralph's charter to give the spawned session full specialist context.
    let charterPrefix = '';
    try {
        const charter = await loadAgentCharter('ralph', context.teamRoot);
        charterPrefix = `You are an AI agent on a software development team.\n\nYOUR CHARTER:\n${charter}\n\nADDITIONAL CONTEXT:\n`;
    }
    catch {
        // Fall back gracefully if no charter exists (e.g., fresh setup)
    }
    const fullPrompt = charterPrefix + prompt;

    loadAgentCharter throws when .squad/agents/ralph/charter.md is missing, and the catch is silent. A team that has never created that file gets charterPrefix = ''.

    That matters for this issue because it means the isolation guidance is not just unenforced, it can be entirely absent at runtime. squad init does not appear to scaffold a charter for the watch agent, and the roster template lists Ralph with an em dash in the charter column rather than a path, so the gap is easy to have without noticing. There is no warning: the session simply starts with no standing instructions at all.

    So the current state is arguably a step below "depends on the model choosing to follow markdown". For anyone who has not hand-written that file, the model is not shown the markdown in the first place.

    A one-line debugLog in that catch, or a doctor check for a missing charter on the configured watch agent, would at least make it visible.

    Mitigation for anyone hitting this now

    Writing .squad/agents/<watch-agent>/charter.md with explicit worktree instructions does get honoured, because of the prefix above. Verified against v0.12.0:

    const c = await loadAgentCharter("ralph", repoRoot);   // returns the full file

    It is still instruction-level and inherits every limitation described in the issue, so it is a workaround rather than a fix. But it is a meaningfully better starting point than an empty prefix, and it costs one file.

    Corroborating the failure mode

    I hit exactly what #1014 describes, on a private repo, with --max-concurrent 2. An agent wrote its changes into the shared team root rather than a worktree and left the integration branch with an uncommitted tree that did not compile against its own HEAD. The work itself was fine once recovered onto a branch, but it was close to being discarded as local debris, because nothing distinguishes an interrupted agent's output from someone's abandoned experiment.

    Worth noting the concurrency setting was my own error: the repo's config pinned maxConcurrent to 1 and its notes explained why. The point is only that the guard rail is a comment in a config file, and raising a number silently removes it.

    On the proposal

    Creating the worktree in the spawn flow and passing that path as the child cwd looks right to me. Two things I would add from the recovery side:

    • Clean up on success only. Removing the worktree after a failed or timed-out run destroys uncommitted work, which is precisely the state a crashed agent leaves behind. Leaving it and reporting the path is recoverable.
    • Commit before the timeout kills the child, or at least leave the tree intact for a human. A partial commit is recoverable; a pruned worktree is not.
  2. swigerb commented on Sep 16, 2026

    @swigerb
    ContributorAuthor

    Separate from the worktree question, but found while investigating it: spawnAgent discards everything the agent says, which makes any spawn-side failure invisible.

    watch/agent-spawn.ts:

    const cp = execFile(cmd, safeArgs, { cwd, timeout, maxBuffer, shell: IS_WINDOWS }, (err) => {
        if (err) { ... resolve({ success: false, error: msg }); }
        else { resolve({ success: true }); }
    });

    execFile's callback is (error, stdout, stderr). Taking only err throws both streams away. When the child exits 0 having done nothing, success: true propagates, the round reports normally, and there is no record anywhere of what the agent actually did.

    What that hid

    A watch loop here was reporting a healthy round every 8 minutes:

    [verbose] Execute: 1 total issues, 1 eligible
    [verbose]   → #NNN: "..." [squad, go:yes, ...]
    [verbose]   executed: 0
    [verbose] Round 1 complete (11.9s)
    

    Eligible, dispatched, exit 0, no error, no branch, no PR, nothing. Four rounds of that before I stopped assuming it was working.

    I patched the callback locally to tee both streams to a file. The spawned session's complete output turned out to be:

    ===== STDOUT (33 chars) =====
    Ready for the development task.
    
    ===== STDERR =====
    Changes    +0 -0
    Tokens     ↑ 24.7k (24.7k written) • ↓ 98 (86 reasoning)
    

    It had read the prompt, spent 86 reasoning tokens, acknowledged it, and exited. copilot -p is one-shot, so the instructions it was waiting for were never coming.

    The cause in my case was self-inflicted: I had written a charter for the watch agent, and execute.ts prefixes it as YOUR CHARTER: ... followed by ADDITIONAL CONTEXT: Ralph, Go! .... The model read the whole thing as a briefing. Removing the charter and re-running, the same round was still going at 170 seconds instead of returning at 13, which is the control.

    That is my problem to solve. The reportable bug is that it took a local patch to node_modules to discover it. Nothing in the logs, at any verbosity, distinguished "agent did the work" from "agent said hello and exited".

    Suggestion

    Capture both streams and act on them:

    }, (err, stdout, stderr) => {
        vlog.log(`agent stdout (${stdout?.length ?? 0} chars)`);
        if (stdout) vlog.log(stdout.slice(0, 2000));
        if (stderr) vlog.warn(stderr.slice(0, 2000));
        ...
    });

    Two cheap additions on top:

    • Persist it. A file under .squad/agent-output/ per spawn makes a failed overnight run diagnosable the next morning instead of gone.
    • Flag the trivial-output case. An agent that exits 0 having written fewer than a few hundred characters and touched no files has almost certainly not done the work. Surfacing that as a warning would have saved this whole investigation.

    Happy to open a PR for the callback change if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wave1:P9Wave 1 PR group P9wave:1-nextWave 1 — immediate next sprint

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions