From 6f6dd71e3e0bdd0bcfc70269fc83128476cbaa20 Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Tue, 4 Aug 2026 17:36:20 -0700 Subject: [PATCH 1/6] docs: training on rollouts from an external agent harness How to turn token capture on, what comes back, which per-rollout metrics to read first, and how a training framework redirects the write to its own transport or reads the records back over HTTP. Signed-off-by: Ananth Subramaniam --- .../external-agent-harnesses.mdx | 270 ++++++++++++++++++ .../latest/pages/training-tutorials/index.mdx | 6 + 2 files changed, 276 insertions(+) create mode 100644 fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx new file mode 100644 index 0000000000..7b9d77da0f --- /dev/null +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -0,0 +1,270 @@ +--- +title: "Training on External Agent Harnesses" +description: "Capture exact token ids and log probabilities when the agent harness drives its own model calls." +position: 6 +--- + +Some agent harnesses run their own model-calling loop. Gym starts them and points them at a model +endpoint, but it does not mediate the calls they make: the harness decides when to call the model, +what to send, and what to do with the reply, and hands back a finished transcript. The Claude Code +CLI is the reference case, driving a multi-turn loop over Anthropic Messages. + +Two things follow. Gym never sees the individual calls as they happen, and the transcript that comes +back carries no token ids, because the wire formats these harnesses speak have no field for them. + +RL trains on token ids. Re-tokenizing the returned text produces a sequence that differs from what +the policy actually sampled, by an amount nobody measured. Token capture records the exact ids inside +the model server, where they still exist, keyed to the rollout that produced them, and rebuilds a +rollout's calls into one contiguous response. + +## When you need it + +The question is not where the harness runs, it is who makes the model calls and whether token ids +survive the round trip. + +| Your agent | What you need | +|---|---| +| Calls the model server through Gym and returns Responses items carrying token ids | Nothing. Train as usual. | +| Drives its own calls and returns text, or a dialect with no field for token ids | Token capture, described below. | +| Drives its own calls but returns token ids in a shape Gym does not read | Token capture, and open an issue so the shape can be read directly. | + +## Turning it on + +Two settings turn capture on. Omit either and the run completes without error while teaching the +model nothing. + +**1. Enable capture and give it node-local storage.** Writer and reader are on the same node, so a +shared filesystem adds latency for nothing and lets two shards write the same file. + +```yaml +env: + nemo_gym: + token_id_capture: + enabled: true + dir: /tmp/nemo_gym_token_id_captures +``` + +Everything run-wide about capture lives in that one block, and it is validated as a whole at +startup. A typo in a key is an error rather than a run that looks configured and silently teaches +the model nothing. Leaving the other settings in place with `enabled: false` is fine, so a config +can carry a directory and toggle capture per run. + +**2. Opt the agent in.** The per-agent flag scopes capture to harnesses that need it, so a native +agent in the same run is left alone. + +```yaml +responses_api_agents: + claude_code_agent: + token_id_capture: true +``` + +Capture reads the ids off the served response, so the inference server has to be returning them. +For vLLM that means a tokenizer: + +```yaml +policy: + generation: + vllm_cfg: + skip_tokenizer_init: false +``` + +Sampling parameters also have to be pinned server-side. Harnesses built for interactive serving +generally send none, so an unset parameter becomes the engine's own default rather than the value +your policy is optimized under. Set `sampling_overrides` on the model server to your trainer's +generation config. + + +One setting that is not about capture, but that decides whether there is anything worth capturing. + +A **tool-call parser** turns the model's tool-call syntax into structured calls the harness can +dispatch. Without one, the harness sees ordinary text, never calls a tool, and every rollout is a +single model call. Capture still works perfectly and there is nothing to chain. This is configured +on the inference server, for example `tool_parser: hermes` under +`http_server_serving_chat_kwargs`, and the right value depends on the model. + +The failure is silent, so check `n_calls` on the first run rather than the reward. + + + +## What you get back + +Each rollout's model calls are stitched into a single Responses payload whose `output` items are +contiguous: every item's `prompt_token_ids` is the running sequence, and its `generation_token_ids` +is what the policy sampled at that step. Gym replaces the rollout's `response.output` with these +items, so a trainer reads `response.output` the same way for a native agent and an external harness. + +The loss mask follows from that structure rather than being shipped separately. Prompt positions are +context, generation positions are trainable. + +## What to watch on a first run + +Read these before the reward curve. A rollout can look healthy with a moving reward while most of it +was never trained on. + +Gym attaches a metrics dict to each rollout under `_ng_token_capture`. Aggregate these across a step +in whatever your framework already reports. + +| Key | Expect | If it is wrong | +|---|---|---| +| `n_calls` | above 1 | The harness never called a tool. Usually a missing tool parser. | +| `chains` | 1 | The rollout split. Part of it is not reaching the optimizer. | +| `delivered_fraction` | 1.0 | Sampled tokens were captured but not delivered. | +| `quarantined_calls` | 0 | Two calls could not be told apart, so neither was used. | +| `empty_generation_calls` | 0 | The output budget or a content filter is truncating generations. | +| `mask_sample` | absent | The rollout lost a call and must not be trained on. | + +`n_calls` deserves particular attention. A value of exactly 1 means the agentic path was never +exercised, and every other key will look correct. + +A rollout with no metrics dict at all was never rebuilt: its model calls were not correlated, so +nothing was captured for it. + +## Integrating a training framework + +Gym owns the record shape and the code that builds a record. Where the record goes is yours. + +### The interfaces + +Two protocols in `nemo_gym.token_id_capture.protocols`. The module deliberately imports no web +framework, cluster runtime or tensor library, so an inference worker can import it without pulling in +Gym's server stack. + +```python +class TokenSink(Protocol): + async def put(self, entry: TokenEntry) -> None: ... + def mark_incomplete(self, rollout_id: str, model_call_id: str = "") -> None: ... + +class TokenSource(Protocol): + async def tokens_for(self, rollout_id: str) -> list[TokenEntry]: ... + async def drop(self, rollout_id: str) -> None: ... +``` + +`put` must be durable before it returns. A reader that runs after the rollout has to see the record, +and that guarantee is what allows records to be deleted once consumed. + +`mark_incomplete` is the only signal that a rollout lost a call. The model call itself still +succeeds, so a sink that drops it makes an incomplete rollout look complete. + +A transport with no delete operation implements `drop` as a no-op and lets whoever owns the storage +retire the records. + +### Writing somewhere other than Gym's file store + +Name a class implementing `TokenSink` and Gym builds it instead of the file store. The capture path +itself does not change: the model server still assembles the record, but the token arrays go to your +transport rather than to disk, and never ride back through an HTTP response. + +```yaml +env: + nemo_gym: + token_id_capture: + enabled: true + sink: my_pkg.sinks:MyDataPlaneSink # module.path:ClassName + sink_kwargs: + endpoint: ${oc.env:MY_DATAPLANE_URL} + shard: ${oc.select:cluster_shard,0} + rebuild_response: false +``` + +`sink_kwargs` is passed to the constructor, so a sink can take the endpoint, client or credentials +it needs rather than reaching for ambient state. Use `${oc.env:VAR}` for anything secret rather than +writing it into the config. Kwargs the constructor cannot accept are an error at startup, as is a +sink that does not implement `mark_incomplete`, which would otherwise make a rollout that lost a +call look complete. + +`sink` replaces the store, so a `dir` alongside it is never read. That is a warning rather than an +error, since nothing is lost, but expect no files on disk. + +`rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected; +only the read-back after each rollout stops, because you do that yourself through your `TokenSource` +whenever you want the records, and you retire them yourself with `drop`. Without it Gym keeps +reading a store nothing wrote to, and reports every healthy rollout as a failed rebuild. + +Leaving `enabled` off is a different thing entirely: it disables capture, so an external harness +yields rollouts with no token ids and nothing to train on. Use that only for evaluation. + + +Configure the sink rather than installing one from a launcher script, or set it up so that +`install_token_sink` runs when the app module is imported. + +`install_token_sink` sets a process global. A model server with `num_workers > 1` is launched by +uvicorn with an app string and `workers=N`, and uvicorn spawns those workers, re-importing the app +module rather than inheriting the launcher's memory. A sink installed by a parent process therefore +does not exist in any worker. Capture then falls back to the file store, or writes nothing at all +when no `dir` is set, and logs no error in either case. + +The configured sink is constructed inside each worker at app startup, so it does not have this +problem. + + +### Reading records back + +`TokenCaptureStore` satisfies both protocols and is the default, so a reader sitting alongside the +store passes the store itself as its `TokenSource`. That is the case for `gym eval run` and for a +trainer colocated with the model server, which is why the store's directory should be node-local. + +A framework staging records through its own transport reads them back through its own +`TokenSource`, which lives wherever that transport does. Nothing about reading is node-local. + +What any source owes is an honest `is_incomplete`. It is how a consumer learns that a rollout lost a +model call, and the records that did arrive can stitch into a chain that looks perfectly contiguous +while missing a turn. A source that always answers `false` will train on such a rollout without +knowing. + +Gym ships no HTTP reader. Nothing needed one, and a route serving records alone could not answer +`is_incomplete`, so a client of it would have had to answer `false` and inherit exactly that +problem. + +### Driving rollouts yourself + +`gym eval run` finalizes each record for you. A framework that calls `run_examples` directly does +not go through that path, so it calls the same function on each finished record: + +```python +from nemo_gym.token_id_capture.delivery import finalize_rollout_token_capture + +finalize_rollout_token_capture(result) +``` + +This rebuilds `response.output`, attaches the build metrics, and retires the consumed records. It +mutates the record in place and never raises. Pass the global config and the store directories as +the second and third arguments if you have them already; otherwise it resolves both itself. + +A rollout that could not be rebuilt is flagged with `mask_sample: true` at the top of its record. +Drop those from the loss: the trajectory is missing a turn, or two candidate generations could not +be told apart, and training on it is silently off-policy. + +### Rollout ids + +Capture keys each record by rollout id, which Gym derives from a run request's task and rollout +indices. That assumes each dispatch gets a distinct pair. If your loop restarts numbering, for +instance running the same indices once per training step, the derived id repeats and two dispatches +share one capture key. + +Set `_ng_rollout_id` on the run body to key them yourself: + +```python +row["_ng_rollout_id"] = f"step{step}.{task_index}-{rollout_index}" +``` + +The id becomes a URL path segment, so it is limited to letters, digits, dots, dashes and +underscores, starting with a letter or digit. An id outside that is refused rather than rewritten. + +## Extensions + +### Sampling pin + +`sampling_overrides` on the model server forces the sampling parameters on every request, overriding +whatever the harness sent. Generation KL error is the metric that tells you whether it is working. + +## Limitations + +**One trajectory per rollout.** A harness that forks sub-agents or retries a call produces a tree of +model calls. Gym delivers the chain carrying the most sampled tokens and reports the rest through +`delivered_fraction`, rather than dropping it silently. Training on the full tree needs a trainer +contract that accepts one. + +**Harness calls outside the rollout.** A harness may generate a conversation title or a +context-compaction summary. Those are real policy output and are currently trained on. A compaction +summary is long enough that it can outweigh the rollout it summarizes, which shows up as +`chains_per_rollout_mean` above 1 and `delivered_fraction_mean` below 1.0. diff --git a/fern/versions/latest/pages/training-tutorials/index.mdx b/fern/versions/latest/pages/training-tutorials/index.mdx index e562d342a8..5be06627a6 100644 --- a/fern/versions/latest/pages/training-tutorials/index.mdx +++ b/fern/versions/latest/pages/training-tutorials/index.mdx @@ -26,6 +26,12 @@ Example GRPO training on instruction following and reasoning environments. unsloth single-gpu 30 min + +Train on rollouts from a harness that drives its own model calls and returns no token ids, such as the Claude Code CLI. + +token capture agent harnesses + + Example DAPO training on math and agentic environments using VeRL, with single and multi-environment support. From 5af4260a8d007b2bbb8c2e88b10b33d735f985eb Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Tue, 18 Aug 2026 00:02:03 -0700 Subject: [PATCH 2/6] docs: align external harness guide with capture lifecycle Use one-line prose and describe source freezing, durable handoff, and conditional retirement with the current public APIs. Signed-off-by: Ananth Subramaniam --- .../external-agent-harnesses.mdx | 177 ++++++------------ 1 file changed, 53 insertions(+), 124 deletions(-) diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx index 7b9d77da0f..bf7a69f3b2 100644 --- a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -4,23 +4,15 @@ description: "Capture exact token ids and log probabilities when the agent harne position: 6 --- -Some agent harnesses run their own model-calling loop. Gym starts them and points them at a model -endpoint, but it does not mediate the calls they make: the harness decides when to call the model, -what to send, and what to do with the reply, and hands back a finished transcript. The Claude Code -CLI is the reference case, driving a multi-turn loop over Anthropic Messages. +Some agent harnesses run their own model-calling loop. Gym starts the harness and points it at a model endpoint, but Gym does not mediate the model calls. The harness decides when to call the model, what to send, and how to handle each reply. It returns a finished transcript. The Claude Code CLI is the reference case and drives a multi-turn loop over Anthropic Messages. -Two things follow. Gym never sees the individual calls as they happen, and the transcript that comes -back carries no token ids, because the wire formats these harnesses speak have no field for them. +Gym does not see the individual calls as they happen. The returned transcript carries no token ids because these harness wire formats have no field for them. -RL trains on token ids. Re-tokenizing the returned text produces a sequence that differs from what -the policy actually sampled, by an amount nobody measured. Token capture records the exact ids inside -the model server, where they still exist, keyed to the rollout that produced them, and rebuilds a -rollout's calls into one contiguous response. +RL trains on token ids. Re-tokenizing the returned text can produce a sequence that differs from the sequence sampled by the policy. The size of that difference is unknown. Token capture records the exact ids inside the model server, where they still exist. It keys the ids to the rollout that produced them and rebuilds the rollout's calls into one contiguous response. ## When you need it -The question is not where the harness runs, it is who makes the model calls and whether token ids -survive the round trip. +The harness location does not determine whether token capture is required. What matters is who makes the model calls and whether token ids survive the round trip. | Your agent | What you need | |---|---| @@ -30,11 +22,9 @@ survive the round trip. ## Turning it on -Two settings turn capture on. Omit either and the run completes without error while teaching the -model nothing. +Two settings turn capture on. If either setting is missing, the run completes without an error but provides no captured tokens for training. -**1. Enable capture and give it node-local storage.** Writer and reader are on the same node, so a -shared filesystem adds latency for nothing and lets two shards write the same file. +**1. Enable capture and give it node-local storage.** The writer and reader are on the same node. A shared filesystem adds unnecessary latency and can let two shards write the same file. ```yaml env: @@ -44,13 +34,9 @@ env: dir: /tmp/nemo_gym_token_id_captures ``` -Everything run-wide about capture lives in that one block, and it is validated as a whole at -startup. A typo in a key is an error rather than a run that looks configured and silently teaches -the model nothing. Leaving the other settings in place with `enabled: false` is fine, so a config -can carry a directory and toggle capture per run. +All run-wide capture settings live in this block, which is validated at startup. A typo in a key raises an error instead of producing a run that appears configured but provides no captured tokens for training. The other settings can remain in place when `enabled: false`, so one config can retain the directory and toggle capture per run. -**2. Opt the agent in.** The per-agent flag scopes capture to harnesses that need it, so a native -agent in the same run is left alone. +**2. Opt the agent in.** The per-agent flag scopes capture to harnesses that need it. Native agents in the same run remain unchanged. ```yaml responses_api_agents: @@ -58,8 +44,7 @@ responses_api_agents: token_id_capture: true ``` -Capture reads the ids off the served response, so the inference server has to be returning them. -For vLLM that means a tokenizer: +Capture reads token ids from the served response, so the inference server must return them. For vLLM, that requires a tokenizer: ```yaml policy: @@ -68,41 +53,28 @@ policy: skip_tokenizer_init: false ``` -Sampling parameters also have to be pinned server-side. Harnesses built for interactive serving -generally send none, so an unset parameter becomes the engine's own default rather than the value -your policy is optimized under. Set `sampling_overrides` on the model server to your trainer's -generation config. +Sampling parameters must also be pinned on the server. Harnesses built for interactive serving generally do not send them. An unset parameter therefore uses the engine default instead of the value used to optimize the policy. Set `sampling_overrides` on the model server to the trainer's generation config. -One setting that is not about capture, but that decides whether there is anything worth capturing. +One setting does not control capture, but it determines whether the rollout contains multiple calls worth capturing. -A **tool-call parser** turns the model's tool-call syntax into structured calls the harness can -dispatch. Without one, the harness sees ordinary text, never calls a tool, and every rollout is a -single model call. Capture still works perfectly and there is nothing to chain. This is configured -on the inference server, for example `tool_parser: hermes` under -`http_server_serving_chat_kwargs`, and the right value depends on the model. +A **tool-call parser** converts the model's tool-call syntax into structured calls that the harness can dispatch. Without a parser, the harness sees ordinary text and does not call a tool. Every rollout then contains one model call. Capture still works, but there are no calls to chain. Configure the parser on the inference server, such as `tool_parser: hermes` under `http_server_serving_chat_kwargs`. The correct value depends on the model. -The failure is silent, so check `n_calls` on the first run rather than the reward. +This failure is silent. Check `n_calls` on the first run before relying on the reward. ## What you get back -Each rollout's model calls are stitched into a single Responses payload whose `output` items are -contiguous: every item's `prompt_token_ids` is the running sequence, and its `generation_token_ids` -is what the policy sampled at that step. Gym replaces the rollout's `response.output` with these -items, so a trainer reads `response.output` the same way for a native agent and an external harness. +Each rollout's model calls are stitched into a single Responses payload with contiguous `output` items. Each item's `prompt_token_ids` contains the running sequence. Its `generation_token_ids` contains the tokens sampled by the policy at that step. Gym replaces the rollout's `response.output` with these items, so a trainer reads `response.output` the same way for native agents and external harnesses. -The loss mask follows from that structure rather than being shipped separately. Prompt positions are -context, generation positions are trainable. +The loss mask follows from this structure instead of being sent separately. Prompt positions provide context. Generation positions are trainable. ## What to watch on a first run -Read these before the reward curve. A rollout can look healthy with a moving reward while most of it -was never trained on. +Read these metrics before the reward curve. A rollout can appear healthy because its reward changes even when most of the rollout never reaches training. -Gym attaches a metrics dict to each rollout under `_ng_token_capture`. Aggregate these across a step -in whatever your framework already reports. +Gym attaches a metrics dictionary to each rollout under `_ng_token_capture`. Aggregate these metrics across a step through the training framework's existing reporting path. | Key | Expect | If it is wrong | |---|---|---| @@ -113,21 +85,17 @@ in whatever your framework already reports. | `empty_generation_calls` | 0 | The output budget or a content filter is truncating generations. | | `mask_sample` | absent | The rollout lost a call and must not be trained on. | -`n_calls` deserves particular attention. A value of exactly 1 means the agentic path was never -exercised, and every other key will look correct. +Pay particular attention to `n_calls`. A value of exactly 1 means the agentic path was never exercised, even though every other key can look correct. -A rollout with no metrics dict at all was never rebuilt: its model calls were not correlated, so -nothing was captured for it. +A rollout without a metrics dictionary was never rebuilt. Its model calls were not correlated, so the rollout has no captured tokens. ## Integrating a training framework -Gym owns the record shape and the code that builds a record. Where the record goes is yours. +Gym defines the record shape and builds each record. The training framework controls where the record goes. ### The interfaces -Two protocols in `nemo_gym.token_id_capture.protocols`. The module deliberately imports no web -framework, cluster runtime or tensor library, so an inference worker can import it without pulling in -Gym's server stack. +The `nemo_gym.token_id_capture.protocols` module defines two protocols. The module imports no web framework, cluster runtime, or tensor library. An inference worker can therefore import it without loading Gym's server stack. ```python class TokenSink(Protocol): @@ -139,20 +107,15 @@ class TokenSource(Protocol): async def drop(self, rollout_id: str) -> None: ... ``` -`put` must be durable before it returns. A reader that runs after the rollout has to see the record, -and that guarantee is what allows records to be deleted once consumed. +`put` must make the record durable before it returns. A reader that runs after the rollout must see the record. This guarantee allows the exact frozen snapshot to be retired after durable downstream handoff. -`mark_incomplete` is the only signal that a rollout lost a call. The model call itself still -succeeds, so a sink that drops it makes an incomplete rollout look complete. +`mark_incomplete` is the durable signal that a rollout lost a call. The model call still succeeds. A sink that drops this signal makes an incomplete rollout look complete. -A transport with no delete operation implements `drop` as a no-op and lets whoever owns the storage -retire the records. +A transport without a delete operation implements `drop` as a successful no-op. The storage owner remains responsible for record retention. ### Writing somewhere other than Gym's file store -Name a class implementing `TokenSink` and Gym builds it instead of the file store. The capture path -itself does not change: the model server still assembles the record, but the token arrays go to your -transport rather than to disk, and never ride back through an HTTP response. +Configure a class that implements `TokenSink`, and Gym constructs it instead of the file store. The capture path does not change. The model server still assembles the record, but the token arrays go to the configured transport instead of disk. They do not return through an HTTP response. ```yaml env: @@ -166,105 +129,71 @@ env: rebuild_response: false ``` -`sink_kwargs` is passed to the constructor, so a sink can take the endpoint, client or credentials -it needs rather than reaching for ambient state. Use `${oc.env:VAR}` for anything secret rather than -writing it into the config. Kwargs the constructor cannot accept are an error at startup, as is a -sink that does not implement `mark_incomplete`, which would otherwise make a rollout that lost a -call look complete. +Gym passes `sink_kwargs` to the constructor, so a sink can receive the required endpoint, client, or credentials instead of reading ambient state. Use `${oc.env:VAR}` for secrets instead of writing them into the config. Unsupported constructor arguments cause a startup error. A sink that does not implement `mark_incomplete` also causes a startup error because it could otherwise make a rollout with a missing call look complete. -`sink` replaces the store, so a `dir` alongside it is never read. That is a warning rather than an -error, since nothing is lost, but expect no files on disk. +`sink` replaces the file store, so a `dir` configured alongside it is not used. This condition produces a warning instead of an error because no data is lost. No capture files appear on disk. -`rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected; -only the read-back after each rollout stops, because you do that yourself through your `TokenSource` -whenever you want the records, and you retire them yourself with `drop`. Without it Gym keeps -reading a store nothing wrote to, and reports every healthy rollout as a failed rebuild. +`rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected. Only the read-back after each rollout stops. The training framework freezes records through its `TokenSource` and retires the consumed snapshot with `drop` after durable handoff. Without this setting, Gym tries to read from a source that received no records and reports every healthy rollout as a failed rebuild. -Leaving `enabled` off is a different thing entirely: it disables capture, so an external harness -yields rollouts with no token ids and nothing to train on. Use that only for evaluation. +Leaving `enabled` off disables capture. An external harness then produces rollouts without token ids and no token data for training. Use this configuration only for evaluation. -Configure the sink rather than installing one from a launcher script, or set it up so that -`install_token_sink` runs when the app module is imported. +Configure the sink instead of installing it from a launcher script. Programmatic installation must run inside the serving process. -`install_token_sink` sets a process global. A model server with `num_workers > 1` is launched by -uvicorn with an app string and `workers=N`, and uvicorn spawns those workers, re-importing the app -module rather than inheriting the launcher's memory. A sink installed by a parent process therefore -does not exist in any worker. Capture then falls back to the file store, or writes nothing at all -when no `dir` is set, and logs no error in either case. +`install_token_sink` sets a process global. A model server with `num_workers > 1` launches uvicorn with an app string and `workers=N`. Uvicorn spawns workers that re-import the app module instead of inheriting the launcher's memory. A sink installed by the parent process therefore does not exist in any worker. Capture then falls back to the file store. If no `dir` is set, the worker has no local destination. -The configured sink is constructed inside each worker at app startup, so it does not have this -problem. +Gym constructs the configured sink inside each worker at app startup, which avoids this process-boundary problem. ### Reading records back -`TokenCaptureStore` satisfies both protocols and is the default, so a reader sitting alongside the -store passes the store itself as its `TokenSource`. That is the case for `gym eval run` and for a -trainer colocated with the model server, which is why the store's directory should be node-local. +`TokenCaptureStore` implements both protocols and is the default. A reader beside the store uses the store as its `TokenSource`. This arrangement applies to `gym eval run` and to a trainer colocated with the model server. The store directory should therefore be node-local. -A framework staging records through its own transport reads them back through its own -`TokenSource`, which lives wherever that transport does. Nothing about reading is node-local. +A framework that stages records through its own transport reads them through its own `TokenSource`. That source can run wherever the transport runs. Reading through a custom source is not restricted to the model server node. -What any source owes is an honest `is_incomplete`. It is how a consumer learns that a rollout lost a -model call, and the records that did arrive can stitch into a chain that looks perfectly contiguous -while missing a turn. A source that always answers `false` will train on such a rollout without -knowing. +Every source must return an accurate `incomplete` value in its frozen snapshot. This value tells the consumer that a rollout lost a model call. The records that arrived can form a contiguous-looking chain while still missing a turn. A source that always reports `false` can cause training to use that incomplete rollout. -Gym ships no HTTP reader. Nothing needed one, and a route serving records alone could not answer -`is_incomplete`, so a client of it would have had to answer `false` and inherit exactly that -problem. +Gym does not provide an HTTP reader. No current integration requires one. A reader must return an atomic snapshot that includes both the entries and the incomplete state. ### Driving rollouts yourself -`gym eval run` finalizes each record for you. A framework that calls `run_examples` directly does -not go through that path, so it calls the same function on each finished record: +`gym eval run` finalizes each record. A framework that calls `run_examples` directly does not use that path, so it must call the same function for each finished record: ```python -from nemo_gym.token_id_capture.delivery import finalize_rollout_token_capture - -finalize_rollout_token_capture(result) +from nemo_gym.token_id_capture.delivery import ( + finalize_rollout_token_capture, + retire_rollout_token_capture, +) + +built = await finalize_rollout_token_capture(result, source) +await downstream.put(result) # Must be durable when this returns. +await retire_rollout_token_capture(rollout_id, source, built) ``` -This rebuilds `response.output`, attaches the build metrics, and retires the consumed records. It -mutates the record in place and never raises. Pass the global config and the store directories as -the second and third arguments if you have them already; otherwise it resolves both itself. +This function freezes the capture snapshot, rebuilds `response.output`, and attaches the build metrics. It mutates the record in place and never raises. The caller supplies the `TokenSource` as the second argument. The function does not retire the snapshot. Retirement occurs only after durable downstream handoff. -A rollout that could not be rebuilt is flagged with `mask_sample: true` at the top of its record. -Drop those from the loss: the trajectory is missing a turn, or two candidate generations could not -be told apart, and training on it is silently off-policy. +A rollout that cannot be rebuilt is flagged with `mask_sample: true` at the top level of its record. Exclude these rollouts from the loss. The trajectory is missing a turn, or two candidate generations could not be distinguished. Training on such a trajectory is off-policy. ### Rollout ids -Capture keys each record by rollout id, which Gym derives from a run request's task and rollout -indices. That assumes each dispatch gets a distinct pair. If your loop restarts numbering, for -instance running the same indices once per training step, the derived id repeats and two dispatches -share one capture key. +Capture keys each record by rollout id. Gym derives this id from the run request's task and rollout indices. This scheme assumes that each dispatch receives a distinct pair. If a training loop restarts numbering, such as reusing the same indices in each training step, the derived id repeats and two dispatches share one capture key. -Set `_ng_rollout_id` on the run body to key them yourself: +Set `_ng_rollout_id` on the run body to provide a distinct key: ```python row["_ng_rollout_id"] = f"step{step}.{task_index}-{rollout_index}" ``` -The id becomes a URL path segment, so it is limited to letters, digits, dots, dashes and -underscores, starting with a letter or digit. An id outside that is refused rather than rewritten. +The id becomes a URL path segment. It can contain letters, digits, dots, dashes, and underscores, and it must start with a letter or digit. Gym rejects an invalid id instead of rewriting it. ## Extensions ### Sampling pin -`sampling_overrides` on the model server forces the sampling parameters on every request, overriding -whatever the harness sent. Generation KL error is the metric that tells you whether it is working. +`sampling_overrides` on the model server applies the configured sampling parameters to every request and overrides values sent by the harness. Generation KL error indicates whether the overrides are working. ## Limitations -**One trajectory per rollout.** A harness that forks sub-agents or retries a call produces a tree of -model calls. Gym delivers the chain carrying the most sampled tokens and reports the rest through -`delivered_fraction`, rather than dropping it silently. Training on the full tree needs a trainer -contract that accepts one. +**One trajectory per rollout.** A harness that forks sub-agents or retries a call produces a tree of model calls. Gym delivers one chain and reports omitted sampled tokens through `delivered_fraction` instead of dropping them silently. Training on the full tree requires a trainer contract that accepts a tree. -**Harness calls outside the rollout.** A harness may generate a conversation title or a -context-compaction summary. Those are real policy output and are currently trained on. A compaction -summary is long enough that it can outweigh the rollout it summarizes, which shows up as -`chains_per_rollout_mean` above 1 and `delivered_fraction_mean` below 1.0. +**Harness calls outside the rollout.** A harness may generate a conversation title or a context-compaction summary. These calls are policy output and can be selected for training. A compaction summary can be long enough to outweigh the rollout it summarizes. This pattern appears as `chains` above 1 and `delivered_fraction` below 1.0 in `_ng_token_capture`. From 81253693179b26e551809001b5fead0e969a2ee7 Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Tue, 18 Aug 2026 11:26:27 -0700 Subject: [PATCH 3/6] docs: clarify token source process ownership Document worker-local sink construction and consumer-local source injection without coupling their virtual environments. Signed-off-by: Ananth Subramaniam --- .../external-agent-harnesses.mdx | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx index bf7a69f3b2..131da2b385 100644 --- a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -113,9 +113,13 @@ class TokenSource(Protocol): A transport without a delete operation implements `drop` as a successful no-op. The storage owner remains responsible for record retention. -### Writing somewhere other than Gym's file store +### Connecting a framework-owned transport +<<<<<<< ours Configure a class that implements `TokenSink`, and Gym constructs it instead of the file store. The capture path does not change. The model server still assembles the record, but the token arrays go to the configured transport instead of disk. They do not return through an HTTP response. +======= +The training framework owns the transport and its client configuration. Gym owns the model-server processes, so each server worker must construct a framework-provided `TokenSink` and `LineageStore` proxy inside that worker. The configured class paths are worker factory descriptors, not shared Python objects and not transport implementations owned by Gym. +>>>>>>> theirs ```yaml env: @@ -133,8 +137,19 @@ Gym passes `sink_kwargs` to the constructor, so a sink can receive the required `sink` replaces the file store, so a `dir` configured alongside it is not used. This condition produces a warning instead of an error because no data is lost. No capture files appear on disk. +The framework separately constructs its `TokenSource` in the trainer or rollout-consumer process. That process may use another virtual environment or actor because the source and sink are independent clients of the same transport. + +```python +source = TransferQueueTokenSource(queue_handle) +built = await finalize_rollout_token_capture(result, source) +await durable_handoff(built) +await retire_rollout_token_capture(result["_ng_rollout_id"], source, built) +``` + `rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected. Only the read-back after each rollout stops. The training framework freezes records through its `TokenSource` and retires the consumed snapshot with `drop` after durable handoff. Without this setting, Gym tries to read from a source that received no records and reports every healthy rollout as a failed rebuild. +When Gym's rollout collector owns rebuilding over a framework transport, install the framework-created source in that collector process with `install_token_source` before collection starts. The default file-backed path needs no installation because Gym constructs a `TokenCaptureStore` from `token_id_capture.dir`. + Leaving `enabled` off disables capture. An external harness then produces rollouts without token ids and no token data for training. Use this configuration only for evaluation. From 8a67c279815a2f53052d75572d31bb41db0e6c6f Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Tue, 18 Aug 2026 11:26:44 -0700 Subject: [PATCH 4/6] fix(docs): remove ownership merge markers Keep the framework transport ownership guidance without references to downstream lineage support. Signed-off-by: Ananth Subramaniam --- .../pages/training-tutorials/external-agent-harnesses.mdx | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx index 131da2b385..e78dd4ab37 100644 --- a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -115,11 +115,7 @@ A transport without a delete operation implements `drop` as a successful no-op. ### Connecting a framework-owned transport -<<<<<<< ours -Configure a class that implements `TokenSink`, and Gym constructs it instead of the file store. The capture path does not change. The model server still assembles the record, but the token arrays go to the configured transport instead of disk. They do not return through an HTTP response. -======= -The training framework owns the transport and its client configuration. Gym owns the model-server processes, so each server worker must construct a framework-provided `TokenSink` and `LineageStore` proxy inside that worker. The configured class paths are worker factory descriptors, not shared Python objects and not transport implementations owned by Gym. ->>>>>>> theirs +The training framework owns the transport and its client configuration. Gym owns the model-server processes, so each server worker constructs a framework-provided `TokenSink` proxy from the configured class path. The configured class is a worker factory descriptor, not a shared Python object or a transport implementation owned by Gym. ```yaml env: From 49ea459eeb0f2e1eeb54bca9956cc9f8b4f869be Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Fri, 21 Aug 2026 10:04:27 -0700 Subject: [PATCH 5/6] docs: align external harness guide with merged delivery APIs Document the actual async sink/source signatures, versioned snapshot retirement, static capture selection, and caller-owned durable handoff after #2126 merged. Signed-off-by: Ananth Subramaniam --- .../external-agent-harnesses.mdx | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx index e78dd4ab37..ebb947440a 100644 --- a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -44,6 +44,10 @@ responses_api_agents: token_id_capture: true ``` +For a training run that captures every configured agent, set `token_id_capture.all_agents: true` in the run-wide block instead of repeating the agent flag. This overrides agent-level opt-ins but does not enable capture by itself. Keep both `enabled` and `all_agents` false in evaluation configs. + +An opted-in agent adds `/training-token-capture` to its rollout-correlated model-server URL. The model server uses that segment to distinguish training capture from ordinary requests on the same endpoint, then strips it before API routing. It does not intercept or change the request body. + Capture reads token ids from the served response, so the inference server must return them. For vLLM, that requires a tokenizer: ```yaml @@ -100,18 +104,24 @@ The `nemo_gym.token_id_capture.protocols` module defines two protocols. The modu ```python class TokenSink(Protocol): async def put(self, entry: TokenEntry) -> None: ... - def mark_incomplete(self, rollout_id: str, model_call_id: str = "") -> None: ... + async def mark_incomplete(self, rollout_id: str, model_call_id: str = "") -> None: ... + async def close(self) -> None: ... class TokenSource(Protocol): - async def tokens_for(self, rollout_id: str) -> list[TokenEntry]: ... - async def drop(self, rollout_id: str) -> None: ... + async def freeze(self, rollout_id: str) -> TokenCaptureSnapshot: ... + async def drop(self, rollout_id: str, *, snapshot_id: str, version: int) -> bool: ... + async def close(self) -> None: ... ``` -`put` must make the record durable before it returns. A reader that runs after the rollout must see the record. This guarantee allows the exact frozen snapshot to be retired after durable downstream handoff. +`put` must make the record durable before it returns. A reader that runs after the rollout must see every acknowledged record. This guarantee allows the consumer to freeze one complete view after the harness finishes. `mark_incomplete` is the durable signal that a rollout lost a call. The model call still succeeds. A sink that drops this signal makes an incomplete rollout look complete. -A transport without a delete operation implements `drop` as a successful no-op. The storage owner remains responsible for record retention. +`freeze` returns one atomic snapshot containing entries, incomplete state, `snapshot_id`, and version. It is idempotent for an unchanged rollout. A write that races the snapshot must advance the observable version. + +`drop` conditionally retires only the supplied snapshot identity and version. It returns `false` if state changed after `freeze`, preserving a late write instead of deleting evidence the consumer never saw. A transport without a delete operation returns `true` without deleting data, and its storage owner remains responsible for retention. + +`close` releases client resources. Gym closes clients it constructs, but it does not close a caller-installed source. ### Connecting a framework-owned transport @@ -142,7 +152,7 @@ await durable_handoff(built) await retire_rollout_token_capture(result["_ng_rollout_id"], source, built) ``` -`rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected. Only the read-back after each rollout stops. The training framework freezes records through its `TokenSource` and retires the consumed snapshot with `drop` after durable handoff. Without this setting, Gym tries to read from a source that received no records and reports every healthy rollout as a failed rebuild. +`rebuild_response: false` tells Gym to stop after the write. Correlation and capture are unaffected. The training framework freezes, reconstructs, durably hands off, and conditionally retires the snapshot through its `TokenSource`. Set `rebuild_response: true` only when Gym's rollout collector owns that sequence; the collector process then requires an installed source or the default file store. When Gym's rollout collector owns rebuilding over a framework transport, install the framework-created source in that collector process with `install_token_source` before collection starts. The default file-backed path needs no installation because Gym constructs a `TokenCaptureStore` from `token_id_capture.dir`. @@ -164,11 +174,11 @@ A framework that stages records through its own transport reads them through its Every source must return an accurate `incomplete` value in its frozen snapshot. This value tells the consumer that a rollout lost a model call. The records that arrived can form a contiguous-looking chain while still missing a turn. A source that always reports `false` can cause training to use that incomplete rollout. -Gym does not provide an HTTP reader. No current integration requires one. A reader must return an atomic snapshot that includes both the entries and the incomplete state. +Consume records through `TokenSource.freeze`. Reading entries alone is insufficient because safe masking and retirement also require the snapshot's incomplete state, identity, and version. ### Driving rollouts yourself -`gym eval run` finalizes each record. A framework that calls `run_examples` directly does not use that path, so it must call the same function for each finished record: +`gym eval run` finalizes each record and retires successful evidence only after the output row is durable. A framework that calls `run_examples` directly does not use that path, so it must perform the same sequence for each finished record: ```python from nemo_gym.token_id_capture.delivery import ( @@ -181,7 +191,7 @@ await downstream.put(result) # Must be durable when this returns. await retire_rollout_token_capture(rollout_id, source, built) ``` -This function freezes the capture snapshot, rebuilds `response.output`, and attaches the build metrics. It mutates the record in place and never raises. The caller supplies the `TokenSource` as the second argument. The function does not retire the snapshot. Retirement occurs only after durable downstream handoff. +`finalize_rollout_token_capture` freezes the source snapshot, rebuilds `response.output`, and attaches build metrics. It mutates the record in place and does not retire evidence. `retire_rollout_token_capture` conditionally drops only the snapshot that was rebuilt, and only after the caller establishes its durability boundary. Failed and masked builds remain available for diagnosis. A rollout that cannot be rebuilt is flagged with `mask_sample: true` at the top level of its record. Exclude these rollouts from the loss. The trajectory is missing a turn, or two candidate generations could not be distinguished. Training on such a trajectory is off-policy. From 018a37abe5338f271f526f578c54320b5a7d3b3f Mon Sep 17 00:00:00 2001 From: Ananth Subramaniam Date: Fri, 21 Aug 2026 10:38:56 -0700 Subject: [PATCH 6/6] docs: describe capture interfaces structurally Clarify that framework adapters satisfy Gym's sink and source contracts by method shape without importing or inheriting from the protocol definitions. Signed-off-by: Ananth Subramaniam --- .../pages/training-tutorials/external-agent-harnesses.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx index ebb947440a..2c3d5de6dd 100644 --- a/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx +++ b/fern/versions/latest/pages/training-tutorials/external-agent-harnesses.mdx @@ -99,7 +99,7 @@ Gym defines the record shape and builds each record. The training framework cont ### The interfaces -The `nemo_gym.token_id_capture.protocols` module defines two protocols. The module imports no web framework, cluster runtime, or tensor library. An inference worker can therefore import it without loading Gym's server stack. +Gym describes the sink and source as structural protocols. Framework adapters do not inherit from these definitions or import them at runtime. They implement the same method signatures, and Gym consumes the resulting objects by that method shape. The definitions below are the reference contract. ```python class TokenSink(Protocol):