feat(token-id-capture): resolve each call's parent at request time

[ananthsub]

Resolves each model call's parent at request time from entries already committed by TokenSink.put. It does not create a second lineage write path.

Multi-worker publication and resolution

sequenceDiagram
    participant H as Harness
    participant W1 as Model worker 1
    participant B as Shared token backend
    participant W2 as Model worker 2

    H->>W1: first request without model-authored history
    W1->>B: resolve(request items)
    B-->>W1: ROOT
    W1->>W1: run inference and stamp ROOT and continuation metadata
    W1->>B: TokenSink.put(TokenEntry)
    B-->>W1: record durable and resolver-visible
    W1-->>H: response A

    H->>W2: continuation echoing response A
    W2->>B: fetch newly committed entry metadata
    W2->>W2: refresh bounded metadata-only index
    W2->>W2: fingerprint lookup and context-digest verification
    alt one token-identity candidate
        W2->>B: lazily load winning call tokens
        B-->>W2: winning TokenEntry and ancestors when needed
        W2->>W2: verify cumulative digest and mark record RESOLVED
    else no safe unique candidate
        W2->>W2: record UNRESOLVED(reason)
    end
    W2->>W2: run inference and stamp the child decision
    W2->>B: TokenSink.put(child TokenEntry)
    W2-->>H: response B

Loading

ROOT, RESOLVED, and UNRESOLVED are the only outcome stages. Diagnostic reasons are persisted for analysis but do not weaken reconstruction behavior.

Summary

• Persists the request-time resolution outcome, verified parent identity, diagnostic reason, fingerprint version, and compact continuation lookup metadata on each TokenEntry (schema version 3).
• Keeps TokenSink.put(entry) as the single durable publication boundary. LineageStore is a read-only view over sink-committed entries and has no served-path record() operation.
• Extracts IncrementalLineageStore for file, queue, KV, or service adapters. Backends implement entry refresh and lazy load hooks while inheriting Gym's canonical matcher, locking, bounded LRU, and digest checks.
• Maintains metadata-only lineage nodes and loads cumulative token arrays only for the winning parent. Cache eviction causes a cold refetch rather than loss of lineage correctness.
• Uses a fixed set of striped in-process locks to serialize same-rollout refresh without growing lock metadata per rollout. The file backend also uses the token store's per-rollout flock for cross-process visibility.
• Collapses retries only when candidates have identical cumulative token identity. Candidates with different digests remain ambiguous and resolve UNRESOLVED.
• Canonically hashes structured request content, tool-call and result identities, and multimodal blocks with versioned, length-delimited fields. Unsupported items and mismatched fingerprint versions fail closed; golden vectors pin the cross-dialect contract.
• Reconstructs ROOT as a valid chain start and follows only verified RESOLVED links. UNRESOLVED, a digest conflict, or missing resolution metadata creates a masked fragment. Prefix matching is retained only to recover a recorded RESOLVED link whose direct parent is absent from the current build.
• Requires a custom sink to have a resolver over the same backend namespace unless the operator explicitly accepts unresolved continuations. Multi-worker startup also requires process-shared resolver behavior.
• Exposes worker-level resolution and capture-failure counters and preserves per-record reasons for offline diagnosis.

Depends on #2341. Required by #2181.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

left a comment but dont think it is blocking, more of a unlikely edge case

[ananthsub]

/claude review

[claude]

SHIP WITH CARE

Request-time parent lineage for token capture: a shared lineage store resolves which recorded call a request continues (fingerprint over model-authored turns, verified by a conversation digest), stamps parent_call_id/cum_len/digest on each TokenEntry, and the builder prefers the verified link over prefix matching. The design is sound and the test coverage is genuinely strong — dialect-agnostic fingerprinting, ambiguous-retry refusal, tree-forks, eviction bounds, and cross-process file resolution are all exercised with assertions on real behavior. The verifier-adjacent contract ("degrade to fallback, never a wrong answer") is the right one, and parent_digest_mismatch correctly quarantines to avoid merging two attempts.

Two findings, both inline, neither a hard blocker but both worth resolving before this drives training data at scale:

1. builder.py:152 (RISK) — parent_call_id_missing quarantines the node + subtree instead of falling back to _infer_parent, contradicting the module docstring and silently dropping chains the old builder kept (e.g. when the recorded parent was an empty-generation call, which is filtered out of entries before by_call_id is built). Fall back on the missing-parent reason; reserve quarantine for digest-mismatch.

2. base_responses_api_model.py:1410 (RISK) — the now-default FileLineageStore does read-all + full-file-rewrite + double-fsync per model call, embedding full cumulative_token_ids, giving O(calls² × context) write amplification on the hot path — on the shared/Lustre filesystems store.py already flags as a throughput cliff. And sink.parent_tokens (the only consumer of those persisted tokens) is set but never read. Drop cumulative_token_ids from the file records until a consumer exists; the builder verifies via the TokenEntry digest, not the store.

Also confirm the num_workers>1 guard has been exercised with a real multi-worker rollout, not just unit tests — the CLAUDE.md bar for capture/training changes is real rollouts, and the file-lock + read-after-write assumption across uvicorn workers is exactly the kind of thing green unit tests do not prove.

[ananthsub]

/claude review

[claude]

Review incomplete: the PR base SHA changed during review (captured merge-base 89b175cfa31ef13444421aeddf0767c6e63d4676, now b37601cc37cee71e153360f5722cf6613b4c1233). The immutable diff I analyzed no longer matches the current PR revision, so ./review-context/check-pr-revision.sh fails the pre-publish check. No inline findings are being posted to avoid anchoring comments to stale lines. Please re-trigger the review against the current head so it runs on an up-to-date diff.

[ananthsub]

/claude review