Skip to content

feat(directsdk): session_unlisted_versions: honor, self-test or force on a Claude Code release the tested list does not name - #414

Open
metahacker wants to merge 179 commits into
NousResearch:sid/ddsdk-transportfrom
metahacker:feat/directsdk-unlisted-versions
Open

metahacker wants to merge 179 commits into
NousResearch:sid/ddsdk-transportfrom
metahacker:feat/directsdk-unlisted-versions

Conversation

@metahacker

@metahacker metahacker commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #402 and merges after it (#402 is stacked on #361); only 20 commits are new here, with two merges of #402's head that had no conflict (74a5ae5, main 72eb3b8 merged in, then 4b16c5c); the rest is #402.

Kept mode runs only on the Claude Code releases listed in plugin.yaml (session_versions). On any other release every call runs per-call until the list names it. This adds one setting for those releases, default unchanged.

The setting

session_unlisted_versions, read from the serving profile's config like the other session settings:

Value On a release not on the list
honor (default) Every call runs per-call, with one log line. This is today's behaviour and the rule the review of #361 set.
self-test The plugin tests that binary itself, offline and in the background. Pass: it keeps processes on it. Fail, timeout, or a test that cannot run: per-call, with the reason logged.
force It keeps processes on any release, with one warning.

Any other value is honor. Under self-test and force, every kept step is still checked before upstream, as on a listed release. A kept process's first request is also held to per-call's own build, as its first resumed request always is. A release whose kept process fails that check runs per-call for the rest of the Hermes process. So a release that frames a request differently costs a restart into per-call, not different bytes.

Each of these is logged once, never per call: the mode on an unlisted release, a self-test's pass, its fail with the reason, and a setting value that is none of the three.

What a self-test runs

selftest.py, in a child Python process, against a loopback upstream that answers the same way every time. It runs one Hermes-shaped conversation twice, per-call and kept:

  • several tool rounds, two calls in one step;
  • an error result, a whitespace-only result and a large result, each followed by a /steer;
  • a reply that leads with a whitespace-only block;
  • then a compacted history (an empty block, a cache mark on a block that is not the last) and a new turn.

Every kept request must equal per-call's, byte for byte. One process must serve the steps, apart from the expected restarts. The binary must report the version it is tested as. Those are the shapes the review of #402 at fc71f5d broke: the test fails against the plugin at fc71f5d ("request 4 of 13 differs") and at 74accf4 (no process kept), and passes on every listed release.

Time per release
Linux, 16 vCPU, each of 2.1.292 to 2.1.326 7.8 to 9.9 s
macOS, Apple M4, 2.1.292 and 2.1.326 4.8 s and 4.3 s

No call waits for it: until a verdict exists, calls run per-call. One test runs at a time per machine. A failing run is tried once more before the fail is kept. The verdict belongs to that binary's files, the plugin version and the platform, so an update in place, another install or a wrapper is tested anew. It is kept under Hermes' home (cache/claude-subscription-directsdk-experimental/session-selftest.json).

What it is kept away from. The child gets a throwaway home, Claude config directory and Hermes home, and a placeholder API key. It runs with every plugin setting at its default, so no profile or managed Hermes setting reaches it. On Linux, where unprivileged user namespaces are allowed, it runs in network and pid namespaces of its own: loopback is its only network, and everything in it ends with it. Elsewhere (macOS, Windows, a Linux that refuses namespaces), Claude Code's traffic is pointed at loopback by its settings only; a binary that ignored them could reach the network, holding no login. The child is bounded in time (240 s) and memory, and it ends when Hermes exits or dies. It also ends what it started; without namespaces, that means every process it saw start.

What a pass proves: those shapes, on this machine, for this binary. It does not stand in for the list, which names the releases the whole session suite passed on.

Tests

tests/test_directsdk_session_unlisted.py: 36 tests, 60 cases. They cover:

  • each mode on a listed and an unlisted release;
  • a pass on the real binary, and a fail on a stand-in whose kept requests differ by one byte;
  • a binary that reports another version, a timeout, the memory bound;
  • a cached verdict, a corrupt cache and a FIFO at the cache path;
  • two agents asking at once, another Hermes process testing;
  • a binary replaced mid-test;
  • Hermes exiting or killed mid-test, with and without namespaces;
  • loopback-only inside the namespaces;
  • a profile claude_config_dir that must not reach the test;
  • the mode changed while a process is kept;
  • force on a release whose kept first request differs.

Each fails with its fix removed: 30 mutations, each caught (logs in the evidence). They also run in evals/ci.sh.

Gate at 6654561 2.1.292 2.1.322 2.1.323 2.1.324 2.1.325 2.1.326 2.1.327 2.1.328
Session test files with the new file, Linux 190 passed 190 passed 190 passed 190 passed 190 passed 190 passed 190 passed 190 passed
macOS 26.6.2 arm64, the new file 57 passed, 3 skipped not run not run not run not run 57 passed, 3 skipped not run not run

Our hosts did not run the full plugin suite or evals/ci.sh for this branch at 6654561 (at b641ed0 they passed: 773, and exit 0 with 133, on 2.1.292 and 2.1.326; #402's head passed both at 74a5ae5). Upstream CI at 6654561 (run https://github.com/NousResearch/hermes-official-plugins/actions/runs/44624965851): the plugin passes on ubuntu-latest (757 passed, 88 skipped; evals/ci.sh session tests 130 passed, 3 skipped) and macos-latest (756 passed, 88 skipped; 130 passed, 3 skipped). It fails on windows-latest in evals/ci.sh: test_a_line_the_plugin_does_not_know_restarts_the_kept_process[event-toolu_00_1-event], where the bytes differ only in device_id. The full suite there passed (728 passed, 116 skipped).

At 514b63d, the merge of #402's 4b16c5c (2.1.329 listed; #411's test-harness commit, which writes the device id into the tests' throwaway home up front): on Linux, against hermes-agent main 32dbc1dc, the session test files with the new file 154 passed on 2.1.329; that is the only run on our hosts at this head. Upstream CI at 514b63d (run https://github.com/NousResearch/hermes-official-plugins/actions/runs/44685885407): the plugin passes on ubuntu-latest (757 passed, 88 skipped; evals/ci.sh session tests 130 passed, 3 skipped), macos-latest (756 passed, 88 skipped; 130 passed, 3 skipped) and windows-latest (728 passed, 116 skipped; 114 passed, 19 skipped).

The 3 skipped on macOS are the namespace cases, which need Linux.

Reviews

  • Grok: 5 findings, all fixed (cleanup at Hermes exit, temp directories, the shared working directory, two tests).
  • A fresh-context Claude, twice. First pass: 7 findings and 3 test gaps. Second pass, on the fixes: 3 problems, 3 fixes without a test of their own, and a flaky test. All fixed:
    • the verdict tied to the binary, and the reported version checked;
    • one test per machine, and a failing run tried again;
    • no reused pid's group killed, and temp directories removed;
    • a namespace failure reported with its reason;
    • an unlisted release whose kept launch never matches runs per-call after the first refusal.
  • GPT (codex review), three passes over the range:
    • Hermes exit and an unwritable cache;
    • a binary replaced mid-test, and descendants the child had seen;
    • a probe that raised kept its max_sessions place, and a test's watchdog outlived it.
      All fixed.
  • One gpt-6-astra adversarial pass: 11 findings:
    • a managed Hermes config could reach the child: fixed;
    • the proxy is no network boundary: namespaces on Linux, the limit stated above;
    • a pass on one model enabled unchecked first requests on others: fixed by holding the first kept request to per-call's build;
    • lifecycle and cache items: fixed;
    • a process orphaned between two 250 ms samples, without namespaces: not fixed, stated above.

Not covered

alt-glitch and others added 30 commits November 10, 2026 05:03
…ess (NousResearch#329)

The plugin started one claude per model call with no limit. A delegate_task
fan-out (up to 10 children by default) plus auxiliary calls started 10+
processes at the same moment, about 250 MB each, all on one subscription,
with Claude Code's own retries off, so a burst could trip the plan's rate
limits for every caller together.

Every turn now takes a place in one process-wide pool before its claude
starts (the extra-usage check included) and gives it back only once that
process has exited, on the fast-return drain thread too, so a crashed or
killed native never holds one. The pool lives under one sys.modules key
because Hermes imports the plugin once per profile home; a per-module pool
would let a multiplexed gateway run max_concurrent per profile.

A call past the limit waits in arrival order within its own timeout. It is
not failed and not sent to a fallback provider. Cancel ends the wait within
0.1 s without spawning. While waiting it yields the plugin's existing empty
liveness chunk once a second; without it Hermes' stale timer would kill the
waiting call and requeue it at the back. A wait over 2 s is logged (one
warning a minute, info after), since a model provider has no door to the
status line.

/usage, the picker handshake and auth status take no place: they send no
Messages request, end within seconds, and /usage must answer while a
fan-out holds every place.

Setting: max_concurrent in plugin.yaml config_schema, default 4, read per
call from the serving profile.
Appends from several processes to one file are not atomic on Windows; the
Windows lane lost one fake's start line (4 of 5 recorded, all 5 calls served).
…rrency

feat(directsdk): cap concurrent Claude Code processes
…iction during a park falls back to per-call bytes
…rounds, compaction, edit, park) and replay-cost bench
… texts merge into one wire message; one process per per-call step
…pped client's process ends with it

With max_sessions 1, three agents starting together kept three processes: the cap was checked under _live_lock but the
place was taken only when the launched native joined _live. A _Reservation now holds the place from the check.

The idle timer held the client's bound _end_session, so a client Hermes dropped without close() stayed referenced, and
its process with it, for the idle hour. The timer calls through a WeakMethod and a finalizer ends the process a
collected client still held. _keep refuses to publish on a client closed mid-step; settled() asks the reader thread
instead of poll(), which reaped the child behind the reader's group kill.
…vious mode's drains, budgets each turn by its own rounds
…Hermes' frames is refused before upstream and replayed
…rocess state together; a pid that vanishes mid-check counts as gone
…the parked row's pid for survival; a bare claude resolves on PATH
…ssion ends it; ownership moves under both locks

Review M2 on NousResearch#361 (and Greptile's "Cancellation misses the kept process"). The ownership-transfer change is the
reviewer's patch, kept together with the weak idle timer. The test pauses a step just before the hand-over, stops
the client from another thread and expects the cancellation error, no kept session, no process and no place held.
…equest; control writes are bounded like frames

Review M3 on NousResearch#361 (and Greptile's "Tool replies bypass the timeout"). The bound on _write is the reviewer's patch:
one timer per control write, as send() already has per frame. A write that gives up on the reader thread now reaches
the step as the timeout error too, not as bad output. The test parks a call on the real CLI, stops the child, brings
a 250,000 byte result with a 0.2 s timeout and expects the timeout error with nothing left running; the same result
on a healthy child arrives in full on the same process.
…and its place back at once

Review m1 on NousResearch#361 (and Greptile's "Dead processes consume slots"). The reader thread ends the session through the
client when native exits, and release() no longer joins the reader from the reader. A process that exits just before
it becomes the session is ended by the hand-over. Tests: kill the idle child, the place is free and a second agent
with max_sessions 1 keeps a process; kill it at the hand-over, the step returns and no place stays held.
Review m3 on NousResearch#361 (and Greptile's "Failed launches leave relays"). The launch directory is made inside the relay's
cleanup scope, as the reviewer suggested. The test spawns nothing: _launch_dir raises OSError and the relay's thread,
its listener and its place against max_sessions must all be gone.
…against max_sessions back

Follow-up to review M1 on NousResearch#361. The place was taken under the lock but released only inside _launch's own cleanup,
so a relay that could not start kept it for the life of the process. _run now gives it back however the launch ends
(the reviewer's release on every outcome), and _launch still hands the place from the reservation to native in one
step under the lock, so the count never reads one high. The test fails the relay with OSError and expects no place held.
…stream; the extension check has its own case

Review m5 on NousResearch#361. tests/test_admission_session_drift.py is the reviewer's regression, added as written in the review.
On this branch a history that does not line up with Hermes' frames is refused first, and that test's drift case has
no frames to line up with, so either refusal answers it. The new case in test_admission_session_history.py changes
the model on rows that do line up: it fails when the extension check stops refusing and passes without the
alignment refusal.
…ls/ci.sh

Review m4 on NousResearch#361 (and Greptile's "Session tests miss CI"). A runner without claude on PATH skips every test in
tests/test_directsdk_session*.py and the suite still exits 0. ci.sh already installs the pin for the evals, so it
runs those three files with CLAUDE_CODE_BIN set to it, on Linux and macOS as the evals do.
…bug level

launch, edited, results, steer, exited or diverged: resumes() answers with the reason instead of None, and one
_restart() kills, releases and logs. The divergence line moves from info to debug with the others. The session test
asserts no restart line at debug level during the conversation, and the reason for its edit of the opening turn.
…s with per-call's bytes

resumes() has a third answer for the expected results followed by one text block: the step writes the steer as a
user frame, then delivers the results. Native reads its stdin in order, so it sends one request with both, the steer as
its own system row (the same wording on 2.1.292, 2.1.322 and 2.1.323). canonical_history() puts that row back as the
text block after the results, in the newest frame and in native's history after it, and refuses a request with a steer
row it cannot match, so a Claude Code that words it otherwise restarts into per-call bytes. A steer of more than one
text block restarts at once. The real-CLI tests (one and three parked calls, both restarts) run in evals/ci.sh.
…rted and per-call

--steer 50,200 has Hermes deliver a steer after those tool rounds and adds a session_restart mode, session mode with the
steer not carried (as before), so the step's cost is reported three ways and each session mode's bytes are compared
with per-call's.
…; the steer bench and its numbers

The transport list says what a steer does on a kept process and what it costs at round 200, kept, restarted and
per-call; the steering paragraph no longer says the provider adds nothing; ci.sh's session files include _steer.py.
metahacker and others added 30 commits November 19, 2026 11:07
Native's limit message names a clock time ("resets 3:45pm") that core
cannot parse, so core never knew when a used-up window reopens. The
relay now keeps the reset of a 429 whose unified status is "rejected",
and the error carries it as body["resets_at"], the field core reads for
its fallback bench and limit notice.
…ession-steer: main's fixes carried onto the kept process

NousResearch#445 lands 15 fixes on main. Where a kept process takes the same path, it gets the same fix:
close() no longer refuses or cancels a call, it ends the kept process (NousResearch#378); a delegated child's
5m cache tier is in the launch env and so in what a kept process is matched on (NousResearch#416); an incomplete
upstream answer keeps its transport phrase and plan-window reset (NousResearch#396, NousResearch#387); and one stdin watchdog
per request serves a kept step's frames too (NousResearch#419). The watchdog ends with its step, so a kept
process between steps holds neither a thread nor its client. One-shots mount no MCP server here, so
the frozen-copy test checks that a one-shot names none instead of the inert helper.
…rozen copy is deleted: plugin.yaml is read at import
… process leaves the step's answer standing; the process still ends
…ate_limit_event is read as it arrives, and a step it reports nothing on feeds the last state
…d child's 5m cache tier and for a vendor-prefixed model id
…ce, so a client dropped right after a step is collected and its kept process ends
…o feat/directsdk-unlisted-versions: no conflict
…n suite, evals/ci.sh and the bench passed on it
…userID into their throwaway home, so per-call and kept carry one device id on Windows too
…ed, the session tests' userID written up front) into feat/directsdk-unlisted-versions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-decision Awaiting an owner policy or product decision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants