Skip to content

Harden the agentic pilot pipeline before the next paid run #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aktasbatuhan merged 2 commits into from
Sep 17, 2026
Merged

Harden the agentic pilot pipeline before the next paid run #65

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
310113b
Harden the agentic pilot pipeline before the next paid run
aktasbatuhan Sep 17, 2026
b8082ad
Stop the study control from clamping the agent's output budget
aktasbatuhan Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions benchmarks/claude-experiment-handoff.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Claude execution handoff

Updated 2026-09-17. This document prepares execution; it does not launch a job.
Claude owns setup and execution. Codex reviews the frozen design, exceptions and
result artifacts. Preserve the user's existing work and all historical evidence.

## Start here

1. Read this file, `flex-agentic/budget-curve-v1.md` and its matching JSON.
2. Inspect `git status`. The repaired implementation is currently local work;
do not assume a fresh remote clone contains it. Transfer a reviewed commit or
explicit source bundle to GCP. Record its commit or SHA-256 inventory. Include
the new `agentic_safety.py`, spec and qualification scripts. Do not transfer
unrelated working-tree changes or credentials in the bundle.
3. Run the relevant offline checks below. Do not spend inference credit to debug
imports, broken graders, or missing datasets.
4. Use the isolated runtime and commands in the canonical protocol. Keep the
Python environment, source files, spec, execution mode and output directory
unchanged after sealing. Keys are loaded from a local `keys.json` by the
runner; provision it securely on the VM, restrict access, never print it or
include it in source/evidence archives.

## Flex pilot: exact scope

- Model: `deepseek-ai/DeepSeek-V4.1-Flash`, Doubleword realtime versus Flex.
- Workload: ten frozen retail tasks, two trials, three agent allowances
($0.01/$0.03/$0.08), paired within task/trial/budget. 120 planned episodes.
- One fresh continuous episode per allowance. No repeated independent attempts,
hidden-grader feedback, or oracle selection of a winning answer. A retry/search
policy would be a separate protocol amendment before new paid data.
- Keep the standard-tier simulator fixed. Agent allowance excludes simulator
spend; total experiment spend and total cost per success include it.
- $10 is the inference ceiling for probes, smoke and continuation together,
not $10 per stage. Agent and auxiliary role caps are $6 and $4. Do not raise
caps or create a fresh spend ledger to get around an exhausted reserve.
- GCP compute is additional. Account for it against the user's remaining total
authorization; this document grants no new compute allowance. Record VM type,
region, uptime and actual/estimated compute charges separately. Shut down the
experiment VM when work is finished, after copying evidence safely.

## Required order and stop conditions

1. **Offline preparation.** Install the pinned benchmark checkout and dependencies
in the isolated runtime. Verify data hashes and clean harness revision. Run
reference qualification with the selected spec and save
`reference-qualification.json` in the intended run directory. It must cover
exactly the selected tasks and match the target Python environment. Reference
tool replay alone is not a successful end-to-end agent/grader smoke test.
2. **Paid API qualification.** Use the `probe` command. It sends two requests per
agent tier and two for the simulator. Re-running `probe` spends again. Inspect
tool correctness, cache-read usage, costs and actual tier evidence separately
for each arm. Never remove markers or switch to Responses to obtain a tier
echo at the expense of prompt caching.
3. **Resolve the known blocker.** Cached Doubleword Chat Completions previously
omitted tier echoes. The current runner accepts response echoes only; there
is no implemented billing-receipt verifier. A price-derived cost, requested
tier, dashboard aggregate or hand-edited `tier_confirmed` field is not a fix.
If echoes remain absent, stop and return the probe evidence for a design/code
decision. Do not repeatedly probe or bypass the gate. A provider receipt
verifier would need implementation and tests before another scaled run.
4. **Paired smoke.** Run `--count 2` only after qualification passes. Inspect both
official grader artifacts, traces, errors, budget stops and costs. Record
observed wall time and projected remaining runtime. Missing checkpoints,
unexplained zero-call episodes or missing charges are not successful smoke.
5. **Continuation.** Continue in the same directory only after the smoke is
sound. Keep one lane and the frozen pair ordering. Missing episodes stay
pending. An interrupted paid episode without an outcome requires evidence
reconciliation, not an automatic retry or deletion of its spend entry.

A 402, infrastructure failure, contradictory tier, invalid provider evidence,
or failed cache gate means stop and diagnose. Report the exact error and artifact
path. Do not adapt task selection, scoring, model, budget or output caps after
seeing results and silently pool the changed run with the original.

## Offline checks and report

Run from the repository root, before freezing the target benchmark runtime:

```sh
uv run pytest tests/test_agentic_study.py tests/test_agentic_gateway.py tests/test_agentic_run.py tests/test_agentic_worker.py tests/test_agentic_safety.py tests/test_cache_policy.py tests/test_qualify_agentic_study.py tests/test_tier_equivalence.py -q
```

Use the canonical runbook for environment preparation, reference replay, probes
and paid commands. Afterwards, generate the descriptive budget report offline:

```sh
.compound/venvs/flex-retail/bin/python -m compound.agentic_study report --spec benchmarks/flex-agentic/budget-curve-v1.json --outcomes artifacts/budget-curve-v1/outcomes.jsonl --out artifacts/budget-curve-v1/report-final.json
```

The report writer refuses to overwrite an existing output. Use a new report
filename when recomputing; preserve earlier artifacts. It separates each budget
and reports agent and auxiliary costs separately. For total inference cost per
success, combine those costs only when both have complete coverage; include
failed attempts. Account for qualification overhead separately in the complete
experiment bill. Do not present the cost subtotals as a reconciled total when
charges are unresolved. Repeated-trial intervals need task clustering; this
pilot does not establish equivalence or non-inferiority.

## Evidence package for Codex review

Return one directory or archive, with a short `REVIEW.md` index containing:

- Exact code identity, launch commands, GCP region/machine, start/end times,
Python version and frozen packages; any deviations and their timing.
- `spec.json`, `plan.json`, `run-manifest.json`, rate snapshot and reference
qualification. Include the dataset/harness hashes and revisions, not secrets.
- Complete `calls.jsonl`, `spend.json`, `probes.json`, `outcomes.jsonl`,
`budget-stops.jsonl` if present, and execution amendments if present.
- All `attempts/` artifacts (official grader outputs, trajectories, outcomes,
`failure.json`) and `worker-logs/`. Include interrupted attempts too.
- Descriptive report, completion counts by tier/budget, unresolved charges,
tier-unverified calls, cache-hit evidence, provider failures and infrastructure
failures. Distinguish reported from token/rate-derived cost.
- Total inference spend including probes, held/unsettled reservations and GCP
cost. List which conclusions are supported, inconclusive or blocked.

Do not send thousands of log lines in chat. Send the artifact location, a small
results table, total spend, and a concise list of anomalies. Codex will review
pairing/coverage, graders, cache/tier provenance, budget accounting, censoring
and whether the claims follow from the evidence. No automatic publication.

## Fireworks experiment: preparation only

Goal: compare the same DeepSeek V4.1 Flash workload across Fireworks, Doubleword
realtime and Doubleword Flex, measuring quality, cost and time together.
The [published article](https://fireworks.ai/blog/DeepSeek-V4.1-Flash-Astra)
is context, not an executable configuration supplied by this repository.

Before execution, document task IDs, harness commit, prompts/tools, reasoning
settings, sampling controls, retries/termination, grader, runtime images, cache
policy and cost boundaries. If the original setup is unavailable, propose an
independent comparison with our pinned harness and label it accordingly. Do not
call SWE-bench/mini-SWE-agent a DeepSWE replication. Do not silently replace the
quality comparison with token-price arithmetic.

There is currently no frozen runnable Fireworks spec or approved launch budget
in this handoff. Return a concrete proposed spec, adapter gaps and a conservative
smoke-first budget for review. Do not add Astra or a batch arm by assumption.

## Prompt to start a Claude session

> Read benchmarks/claude-experiment-handoff.md and the linked canonical Flex
> protocol. Prepare the GCP execution setup and complete the offline checks.
> Use my existing spending authorization only; do not increase it. Execute the
> paid qualification and paired smoke only when their documented prerequisites
> are satisfied. Stop on missing tier evidence or infrastructure problems,
> preserve the ledger and return the evidence. Continue the pilot only after a
> sound smoke, without changing its frozen design. Keep Fireworks at preparation
> stage pending a reviewed spec. Return the evidence package and REVIEW.md for
> Codex to review; do not publish findings.
111 changes: 23 additions & 88 deletions benchmarks/experiments-next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,96 +1,31 @@
# Two experiments to run next
# Experiment entry point

Both are written after the caching fix of 2026-09-15. Every paid path now sends
a `cache_control` marker on hosts whose caching is opt-in, and
`tests/test_cache_policy.py` fails the build if one does not. Neither experiment
below starts until a two-episode probe shows `cached_tokens > 0` in
`calls.jsonl`.
Use [the Claude execution handoff](claude-experiment-handoff.md) as the starting
point. It identifies the current specifications, launch gates, spending bounds,
and evidence required for independent review.

Measured inputs both plans rely on, from the aborted 913-episode run:
## 1. Budget-first Flex pilot

| Quantity | Value |
|---|---|
| Agent tokens per retail episode | 65,031 input, 4,161 output (16:1) |
| Agent cost per episode, uncached | $0.0060 |
| Agent cost per episode, cached (measured, 79% hit) | $0.0021 |
| Simulator cost per episode, DeepSeek V4.1 Flash | $0.0014 |
Canonical protocol: [budget-curve-v1](flex-agentic/budget-curve-v1.md).
Canonical specification: [budget-curve-v1.json](flex-agentic/budget-curve-v1.json).

## 1. Flex tier APIs benchmark
The primary question is delivered task success at a fixed agent-dollar allowance,
comparing realtime and Flex within the same model and workload. Time and delivery
failures remain secondary outcomes. The first pilot is descriptive and limited to
DeepSeek V4.1 Flash on Doubleword; it is not an all-provider ranking.

**Question.** Across providers that sell a cheaper asynchronous tier, what does
that tier actually cost you in wall-clock time, reliability and task success?
The pilot answered this for cost and latency on five routes and failed to answer
it for quality, because per-cell samples of six flip about a fifth of the time.
The old breadth/depth budgets, uncached pilot extrapolations and equivalence
recommendation formerly on this page are superseded. Historical specs and raw
results remain available; do not overwrite them or use them as the new launch
configuration. The equivalence analyzer is not the budget-curve reporter.

**Why it is not just a rerun.** Three things changed. Caching is on, which moves
the cost axis by roughly 65% on Doubleword and makes the earlier cost numbers
unusable. The simulator is DeepSeek V4.1 Flash rather than GPT-5.6 Sol, which
moves absolute pass rates. And the equivalence protocol in
`tier-equivalence.md` replaces the old difference test.
## 2. Fireworks comparison

**Shape.** Two layers, because they have different sample requirements:
This is a separate design, not an executable experiment yet. See the handoff's
Fireworks section. Exact replication needs the original harness and evaluation
configuration. A different harness must be labelled an independent comparison.

- *Breadth*, for the map: all five routes on both tiers, roughly 80 attempts per
route, enough to pin a median cost and a median duration. About $19 with the
old simulator costs, less now. This regenerates the positioning map with one
simulator and one caching regime, which the current map does not have.
- *Depth*, for the quality claim: one route pair at 88 tasks x 7 trials, 1232
episodes, using the frozen analysis in `scripts/analyze_tier_equivalence.py`.
DeepSeek on Doubleword is the only affordable candidate: about **$4.30** now
($0.0021 + $0.0014 per attempt), against $17 for GLM and $157 for Astra.

**Do not** buy depth on more than one or two routes. Simulation puts power at 81%
for a +/- 7.5 point margin at 616 per arm; below that the answer is
"inconclusive", which is not worth paying for.

## 2. Fireworks claim replication on Doubleword

**The claim.** Fireworks published DeepSeek-V4.1-Flash at 74.34% pass@1 on
DeepSWE for $0.430 per task, against GPT-6 Astra at 74.12% for $6.524, and call
it 15x cheaper at equal quality. Their arithmetic reproduces exactly from their
published token mix of 148,741 uncached input, 36.7M cached input and 211,513
output tokens. It is an honest, auditable post.

**The gap worth attacking.** It is a claim about a model, not about a host. The
same model is served by seventeen providers, and on Fireworks' own token mix:

| Host | $/task | vs Fireworks | vs Astra |
|---|---:|---:|---:|
| DeepSeek direct | 0.2595 | 1.66x cheaper | 25.1x |
| GMICloud | 0.3632 | 1.18x | 18.0x |
| DeepInfra | 0.3771 | 1.14x | 17.3x |
| Fireworks | 0.4295 | - | 15.2x |
| Doubleword batch 24h | 0.4428 | 1.03x more | 14.7x |
| Doubleword async | 0.4868 | 1.13x more | 13.4x |
| Doubleword realtime | 0.5167 | 1.20x more | 12.6x |

Fireworks is fourth. Because 99.6% of input is cached on this workload, the
cache-read rate dominates the bill, and Doubleword charges $0.010/M against
Fireworks' $0.007 and DeepSeek's $0.003. The headline would be 1/25th on
DeepSeek's own endpoint.

**Do not replicate the quality half.** Their reported model gap is 0.22 points,
their own stated run-to-run variation is 1.4 to 3.2 points, and at 500 tasks the
standard error on a 74% pass rate is 2.0 points. Their full run cannot separate
those models and neither could ours. Paying for that is buying a known
"inconclusive".

**Replicate the economics instead**, which is the load-bearing part: does the
99.6% cache-hit rate hold off Fireworks, and what is the real token mix per host?
Near-deterministic, so 30 to 50 tasks is enough.

| Scope | 50 tasks | 100 | 500 |
|---|---:|---:|---:|
| Fireworks + both Doubleword tiers | $72 | $143 | $717 |
| Add DeepSeek direct | $85 | $169 | $847 |
| Add the GPT-6 Astra baseline | +$326 | +$652 | +$3,262 |

Skip the Astra arm; their table already reports it and it is the least contested
number in the post.

**Prerequisites.** DeepSWE needs Docker and the grader, so this one wants a VM,
about $10. We have SWE-bench Verified wired up, not DeepSWE, so the honest
framing is "their workload shape on our harness", not "we reproduced DeepSWE".
And Doubleword must run on chat completions with markers, or it shows up 11x
worse ($5.66 per task against $0.52) for a reason that has nothing to do with
Doubleword.
Do not treat a rate-card calculation on another provider's token mix as measured
cost, quality, cache performance, or a reliable funding estimate. The prior
cost-ranking and 30-to-50-task recommendation on this page are withdrawn as
execution guidance. Do not launch this experiment under the Flex pilot's $10 cap.
5 changes: 5 additions & 0 deletions benchmarks/flex-agentic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Flex agentic pilot

For the next experiment start with [the Claude execution handoff](../claude-experiment-handoff.md)
and [the budget-first protocol](budget-curve-v1.md).
The configurations described below are historical. The current runner requires
fresh cache/tool/tier qualification and refuses to adopt an unsealed old run.

Four models, five provider/model routes, two service tiers, and 15 fixed tasks:
150 planned episodes, one trial per task and route/tier. This is an integration
pilot, not a sufficiently powered model ranking. Paid execution runs on an isolated
Expand Down
Loading
Loading