feat(vllm-model): on-policy sampling pin via sampling_overrides

[ananthsub]

On-policy sampling pin in the vLLM model server

Part of the token-id capture stack for #1824. Replaces #2127, which could not be retargeted after four commits were inserted into the stack (GitHub does not permit changing the base branch of a PR that is part of a stack).

Problem

An external agent picks its own sampling settings when it calls the model. During training the generation worker checks that requests use the trainer's sampling settings, so a temperature or top_p the agent chose that doesn't match will either fail the run or bias the recorded rollout.

What it adds

• sampling_overrides in vllm_model: set temperature and top_p on every chat request before generation, replacing whatever the agent sent.
• The values come from plain config keys (policy_generation_temperature, policy_generation_top_p) with on-policy defaults. Gym doesn't know anything about the trainer; the trainer writes its settings into those keys (see the matching NeMo-RL change) and Gym just reads them.

Notes for review

• vllm_model never references NeMo-RL. Any training framework can drive this by filling in those config keys.
• This change stands on its own; it's grouped with the stack because it's part of making external-agent training work.

Tests

No isolated unit test — the vllm_model server test runs in its own environment. Verified in the 2-GPU Megatron GRPO run: sampling matched the trainer and the run stayed on-policy (generation KL ~0.04).

Stack

One commit per PR, each based on the previous branch (bottom of the stack targets main):

1. feat(token-id-capture): capture training tokens from external harnesses #2124 capture core — store, sink, read route
2. feat(token-id-capture): chain a rollout's calls into one response #2125 trajectory builder and consumer
3. feat(token-id-capture): deliver rebuilt trajectories safely #2126 uniform delivery, per-agent scoping, retention
4. feat(token-id-capture): resolve each call's parent at request time #2180 resolve each call's parent at request time
5. feat(vllm-model): supply the previous call's exact training tokens #2181 supply the engine the previous call's exact tokens
6. feat(token-id-capture): add an optional bearer token to the read route #2182 require a bearer token on the token read route
7. feat(vllm-model): on-policy sampling pin via sampling_overrides #2183 on-policy sampling pin (this PR)

All seven are drafts. #2179 (harness side calls) and #2128 (Claude Code example)
were closed and dropped from the stack; #2183 supersedes #2127.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ananthsub]

Not actually merged. When the stack was reordered to put the sampling pin at the bottom, this PR's base branch (ananthsub/tokidcap/read-route) came to contain this commit, so GitHub marked the PR merged and auto-deleted the head branch. Nothing landed on main.

Superseded by #2190, which carries the same commit with main as its base.