feat(herd): sessions that outlive your terminal (PRD 0009) (#342)

* feat(herd): sessions that outlive your terminal, and a roster that says which one wants you

Implements PRD 0009. `moshcode start claude -d` now runs the engine in a
runtime that outlives the pit and hands the prompt straight back, so the
pit stops being a one-thing-at-a-time shell: `/ps` shows what is running
and what state each session is in, `/attach` steps into one, detaching
leaves it running.

Two substrates behind one interface, because "requires tmux" was the wart
worth removing rather than documenting:

  tmux  a single named server (socket `moshcode`, not tabs.mjs's per-pid
        one). Full fidelity — resizing, scrollback, native attach.
  pty   no tmux: script(1) allocates the pty, the child is detached with
        its stdin on a FIFO held O_RDWR so it never sees EOF when the pit
        exits, and attach replays the transcript and relays keystrokes.
        Sized from inside via stty; a later resize cannot reach it.
  none  foreground passthrough, exactly as today, said once.

Semantic state (working/blocked/done/idle/unknown) with herdr's rule that
each session has ONE authority: a live hook report suppresses screen
classification entirely, and expires so a crashed agent cannot read
`working` forever. Screen rules ship beside each engine's install spec,
are anchored to things a terminal draws rather than English words, and
are overridable in ~/.moshcode/herd/rules.json when they rot.

`blocked` can page the operator through the existing notify()/ask()
fan-out, with the reply typed back into the session that was waiting —
the part herdr structurally cannot do. Only transitions notify, so a
session sitting blocked does not page every five seconds.

One surface for humans and agents: every verb takes --json, `wait` exits
0/2/3 so scripts can branch, and moshscript gets herdStart/herdPrompt/
herdWait/herdRead/herdList/herdKill as values rather than exit codes.

Two bugs the survival test caught, both of which would have shipped as
"finished agents report gone": tmux's remain-on-exit was set in a second
call that a fast command could beat, now one invocation via tmux's `;`;
and the pty substrate could not tell a finished agent from a rebooted
box, now the session's own shell records its exit.

Named `herd`, not the `runtime`/`agent <verb>` the PRD first proposed —
`agent` is already an alias of `agents` and a test pins that, and
`runtime` is what src/runtime.mjs already means. PRD updated to match.

Deferred and stated in the PRD: auto-installing the status hook into each
engine (the protocol ships and works), pointing ttyd at a real session,
and scrollback replay across reboots.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(herd): stop `wait` exiting instantly — an unref'd poll timer let node exit under it

CI caught this; a green local suite did not. Thirteen tests in
herd-cli.test.mjs came back `cancelledByParent`, which was the test
runner's way of reporting that the file's process had died mid-run.

The cause is a real bug in shipped behaviour, not a test artifact. The
poll timer in waitFor() was unref'd, copying the instinct from mirror.mjs
where a background nicety must never hold the process open. Here that is
exactly backwards: `wait` and `watch` exist to keep the process alive. An
unref'd timer means node finds nothing scheduled between polls and exits.
It does not hang — it is worse. `moshcode wait api --timeout 1h` returned
in a millisecond, exit 0, having waited for nothing, and any script
branching on it would have read that as "the agent is ready".

Measured before: waitFor resolved never and the process exited after 1ms.
After: it waits the full duration and settles. End to end, `moshcode wait`
now blocks the whole timeout and exits 2, exits 0 the moment the state
matches, and exits 3 for a session that does not exist.

Also un-unref's the liveness poller in the pty substrate's attach, which
was the same trap one step from being sprung: the follow timer beside it
is already unref'd, so an attach whose stdin did not hold the loop open
would have exited the instant it started. It is cleared on detach, so it
never outlives the attach either.

The regression test runs waitFor in its own process with nothing else
pending, because the test runner itself keeps the event loop alive and
hides the bug completely. Verified it fails against the old code before
keeping it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: ralyodio

README.md

@@ -28,6 +28,12 @@ or miss one that does. A test fails the build when it drifts.
 |---|---|---|
 | `moshcode agents` | engines | list engines or launch one autonomously |
 | `moshcode start` | engines | launch an engine with its native defaults |
+| `moshcode herd` | runtime | run agent sessions that outlive this terminal |
+| `moshcode ps` | runtime | list herd sessions and what each one is doing |
+| `moshcode attach` | runtime | attach this terminal to a herd session |
+| `moshcode kill` | runtime | end a herd session |
+| `moshcode wait` | runtime | block until a session is blocked, done, or idle |
+| `moshcode restore` | runtime | rebuild the herd's sessions after a reboot |
 | `moshcode install` | engines | install an engine or workflow tool |
 | `moshcode uninstall` <br>`remove` | engines | take an engine or workflow tool off this machine |
 | `moshcode upgrade` <br>`update` | engines | update moshcode, engines, or tools |
@@ -102,6 +108,103 @@ is shorthand for `moshcode start claude`. In the TUI, use `/agents <engine>` for
 autonomous mode or `/start <engine>` for raw mode. Running `moshcode agents` or
 `/agents` without an engine still lists engines and their install status.
 
+## The herd — sessions that outlive your terminal
+
+Every launch above hands an engine the whole terminal and waits. That is why
+they feel native, and it is also why the pit can only do one thing at a time and
+why closing the lid kills the work.
+
+The herd inverts it. Add `-d` and the session runs in a runtime that outlives
+the pit, so you get your prompt back immediately:
+
+```sh
+moshcode start claude -d --name api      # runs in the background, prompt returns
+moshcode agents codex -d                 # autonomous, and still detached
+moshcode ps                              # who is running, and who wants you
+moshcode attach api                      # step in; Ctrl-b d steps back out
+moshcode kill api                        # end it
+```
+
+Close the terminal, drop the SSH link, come back tomorrow — `moshcode ps` still
+answers, and `moshcode attach` puts you back inside. In the pit the same verbs
+are `/ps`, `/attach`, `/kill`, and the roster prints on the way in.
+
+### Which one needs you
+
+Every session carries a state: `working`, `blocked`, `done`, `idle`, or
+`unknown`. `blocked` means a human decision is the only thing missing.
+
+```
+  api       claude  blocked   ~/src/coinpay        12m
+  web       codex   working   ~/src/ugig.net        4m
+  audit     opencode  done    ~/src/moshpit-dns     1h
+```
+
+State comes from one authority per session, never two. An engine that reports
+through a lifecycle hook (`moshcode herd report <name> <state>`) is believed and
+its screen is not second-guessed; everything else is classified from the bottom
+of its screen. Nothing recognisable reads `unknown`, which is a safe answer —
+detection never gates a launch. Patterns that go stale can be fixed in
+`~/.moshcode/herd/rules.json` without waiting for a release.
+
+Blocked can also come and find you, using the same notification fan-out as
+`notify()`/`ask()`:
+
+```sh
+moshcode herd notify on --ask            # email/SMS/Slack/Telegram/push
+moshcode herd start claude --name watch  # then run `moshcode herd watch` in the herd
+```
+
+With `--ask`, whatever you reply is typed into the session that was waiting.
+
+### Driving it from a script or another agent
+
+There is no second API — every verb takes `--json`, and that is what a machine
+reads. `wait` exists to be branched on: exit `0` matched, `2` timed out, `3` no
+such session.
+
+```sh
+moshcode herd start claude --name api --json
+moshcode herd prompt api "port the auth routes" --wait
+moshcode herd read api --lines 40
+moshcode wait api --state blocked --timeout 1h
+```
+
+moshscript gets the same surface as values rather than exit codes, which is what
+makes fan-out practical:
+
+```js
+herdStart("claude", { name: "api" });
+herdStart("codex",  { name: "web" });
+herdPrompt("api", "port the auth routes");
+herdPrompt("web", "port the dashboard");
+await herdWait("api"); await herdWait("web");
+say(herdRead("api", { lines: 20 }));
+```
+
+### After a reboot
+
+```sh
+moshcode restore --dry-run       # what would come back
+moshcode restore --resume        # and ask each engine to reopen its conversation
+```
+
+This brings back the *shape* — the sessions, in their directories, on their
+engines. The processes are new. Work that was in flight is not still running,
+and `--resume` only reaches engines that have a resume flag of their own.
+
+### What it runs on
+
+`tmux` when the box has it: real resizing, scrollback, native attach. Without
+tmux, sessions run under `script(1)` with their input on a FIFO — they work and
+they persist, but their size is fixed when they start. With neither, launches
+stay in the foreground and say so once. moshcode does not turn a soft dependency
+into a hard one, so `-d` never fails; at worst it degrades and tells you what
+would fix it.
+
+The session manifest and every transcript are written `0600`: engine argv and
+engine output both carry secrets.
+
 ### Parallel pit tabs
 
 At the mosh prompt, `/new` opens and switches to another independent moshcode

bin/moshcode.mjs

@@ -28,6 +28,8 @@ import { createPrd, listPrds, authoringPrompt } from "../src/prd.mjs";
 import { loginAuto, whoami, logout } from "../src/auth.mjs";
 import { tui } from "../src/tui.mjs";
 import { consoleCommand } from "../src/console.mjs";
+import { herdCommand, herdStart, splitDetachArgs } from "../src/herd-cli.mjs";
+import { detectSubstrate, substrateNote } from "../src/herd.mjs";
 import { dnsCommand } from "../src/dns.mjs";
 import { templateCommand } from "../src/templates.mjs";
 import { serveCommand } from "../src/serve.mjs";
@@ -117,6 +119,23 @@ function printEngineStatus(json = false) {
 }
 
 async function launchEngine(key, engine, args, { agentMode = false } = {}) {
+  const { detach, name, rest: engineArgs } = splitDetachArgs(args);
+  if (detach) {
+    const substrate = detectSubstrate();
+    if (substrate) {
+      const code = herdStart([
+        key, ...(name ? ["--name", name] : []), ...(agentMode ? ["--agent"] : []), ...engineArgs,
+      ]);
+      if (code) process.exitCode = code;
+      if (!process.stdin.isTTY || process.env.MOSHCODE_NESTED === "1") return;
+      return tui();
+    }
+    // R2: degrade, loudly, once — and then still do the thing that was asked
+    // for. A launch that refuses because the box has no tmux would be a worse
+    // answer than a launch that works and ends with this terminal.
+    console.error(`⚠ ${substrateNote(null)}`);
+  }
+  args = engineArgs;
   if (agentMode) {
     const note = `agent mode: ${key} ${agentLaunchArgs(engine).join(" ")}`;
     console.error(engine.agentsView
@@ -314,6 +333,18 @@ async function main() {
     const [key, engine] = resolved;
     return launchEngine(key, engine, rest.slice(1));
   }
+  // The herd (PRD 0009). `herd` is the namespace; the five verbs people reach
+  // for most often are also top-level, because `moshcode ps` is what someone
+  // types when they want to know what is running and nobody should have to
+  // learn a namespace to ask that.
+  if (cmd === "herd") {
+    process.exitCode = (await herdCommand(rest)) || 0;
+    return;
+  }
+  if (["ps", "attach", "kill", "wait", "restore"].includes(cmd)) {
+    process.exitCode = (await herdCommand([cmd === "ps" ? "ps" : cmd, ...rest])) || 0;
+    return;
+  }
   if (cmd === "tools") {
     const asJson = rest.includes("--json");
     printStatus(toolStatus(), asJson);

prd/0009-persistent-agent-runtime.md

@@ -2,14 +2,14 @@
 openprd: "0.2"
 id: "0009"
 title: "Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents"
-status: Draft
+status: Accepted
 authors:
   - anthony@profullstack.com
 created: 2026-08-09
 updated: 2026-08-09
 repo: https://github.com/moshcoder/moshcode
-discussion:
-implementation:
+discussion: https://github.com/moshcoder/moshcode/pull/341
+implementation: src/herd.mjs, src/herd-state.mjs, src/herd-cli.mjs
 tags: [runtime, sessions, agents, tui, notify]
 supersedes:
 superseded-by:
@@ -203,17 +203,26 @@ entries in a new `runtime` group:
 
 | command | what it does |
 |---|---|
-| `moshcode runtime` | start / inspect / stop the background runtime |
-| `moshcode ps` | list live sessions with state |
+| `moshcode herd` | the namespace: status, start, prompt, read, send-keys, report, notify, watch, prune, stop |
+| `moshcode ps` | list sessions with state |
 | `moshcode attach <name>` | attach to a session |
 | `moshcode kill <name>` | end a session |
 | `moshcode wait <name>` | block until a state transition |
 | `moshcode restore` | rebuild sessions from the manifest |
-| `moshcode agent <verb>` | start / prompt / read / send-keys / stop |
 
-TUI equivalents follow the existing convention: `/ps`, `/attach <name>`,
-`/kill <name>`, `/restore`. `/agents <engine>` keeps its meaning and simply
-gains `--name` and a detachable session underneath it.
+The namespace is `herd`, not the `runtime` / `agent <verb>` this document first
+proposed. Two reasons, both found while building it. `agent` is already a
+registered alias of `agents` in `PIT_COMMANDS`, and a test pins
+`suggest("agent") === "agents"` — so `moshcode agent start` would have meant two
+different things depending on where it was typed. And `runtime` is what
+`src/runtime.mjs` already calls the moshscript interpreter. The five verbs
+people reach for most are top-level anyway, which is what the original table was
+really asking for: nobody should have to learn a namespace to ask what is
+running.
+
+TUI equivalents follow the existing convention: `/herd`, `/ps`, `/attach <name>`,
+`/kill <name>`, `/wait`, `/restore`. `/agents <engine>` and `/start <engine>`
+keep their meaning and simply gain `-d` / `--name`.
 
 **The pit's front door changes.** Today `moshcode` prints a banner and a prompt.
 With anything running it prints the herd first:
@@ -301,3 +310,53 @@ exactly like today. No repeated nagging, no failure.
 - **Scope.** Phases 1–3 are independently shippable and should ship that way.
   Phase 1 alone — sessions that survive the terminal — is the bulk of the value
   and does not require a single line of state detection.
+
+## Implementation Notes
+
+Written after the build, so the document and the code agree.
+
+**A second substrate, which this PRD did not ask for.** R2 promised only to
+degrade gracefully without tmux. That was not good enough: `/new` already
+required tmux and it is the wart people notice. So there are two substrates
+behind one interface — tmux when the box has it, and otherwise `script(1)` with
+the session's stdin on a FIFO, reusing the capability detection `pty.mjs`
+already does. The FIFO is opened `O_RDWR` before the spawn so the child is its
+own writer and never sees EOF when the pit exits, which is the whole trick. Its
+one real limit: nothing outside a pty can ioctl its master, so the size is fixed
+at launch (set from inside by `stty`) and a later resize does not reach it.
+`MOSHCODE_HERD=pty` forces it, which is how the fallback is tested on a box that
+has tmux.
+
+**Two bugs the survival test caught**, both of which would have shipped as
+"finished agents report `gone`". tmux's `remain-on-exit` was being set in a
+second call, and a fast command finishes before that process starts — fixed by
+making the session and its option one invocation using tmux's `;` argument. And
+the pty substrate could not tell "the agent finished" from "the box rebooted",
+since both are a dead pid — fixed by having the session's own shell record its
+exit code on the way out.
+
+**Delivered:** R1–R12 and R14. Both substrates are covered by an integration
+test that starts a session in one process, exits it, and talks to the session
+from another.
+
+**Not delivered, deliberately:**
+
+- **R7 tier-1 hook installation.** The protocol ships and works —
+  `moshcode herd report <name> <state>` takes authority, suppresses screen
+  classification entirely while it is live, and expires so a crashed agent
+  cannot read `working` forever. What is not built is auto-installing that call
+  into each engine's hook config via the `plugins.mjs` / `skills.mjs` fan-out.
+  Until then tier 1 is opt-in and tier 2 carries the roster.
+- **R13, the browser as a real client.** `console.mjs` still points ttyd at a
+  shell rather than at `moshcode attach <name>`, and `mirror.mjs` keeps its
+  documented blind spot. The runtime it would attach to now exists, so this is a
+  small follow-up rather than a design question.
+- **R15, scrollback replay.** P2 and opt-in in this document; still the right
+  call not to write engine output across a reboot by default.
+
+**Rules will rot, and that is planned for.** The shipped patterns are
+conservative and anchored to things a terminal draws — brackets, selectors, line
+anchors — never bare English words, and a test asserts that. `unknown` is
+common and safe. `~/.moshcode/herd/rules.json` lets a rotted pattern be fixed on
+the box it rots on, and a malformed entry there loses that pattern rather than
+the file.

prd/README.md

@@ -24,5 +24,5 @@ Start one with `moshcode prd "<idea>"` (TUI: `/prd`).
 | [0006](0006-help.md) | --help | Draft |
 | [0007](0007-profullstack-site-init.md) | Generate batteries-included Profullstack sites for Moshpit names | Draft |
 | [0008](0008-ticker-research-and-plugin-marketplace.md) | Bring equity research into the pit, and ship the pit's slash commands as a plugin | Draft |
-| [0009](0009-persistent-agent-runtime.md) | Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents | Draft |
+| [0009](0009-persistent-agent-runtime.md) | Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents | Accepted |
 <!-- PRD-INDEX:END -->