harbor: run a task tree from disk, and a multi-model grid runner - #60
Merged
Merged
Conversation
FrontierSWE v2 (released 2026-09-02) ships 34 Harbor-schema tasks but not its runner: px-eval is 'to be added once the public release is ready'. Harbor's own --path takes a task or dataset directory, so a checkout plus a commit is enough to run them. - build_command/sweep_harbor/CLI take --task-path as an alternative source to --dataset, and refuse both or neither rather than guessing. - scripts/cloud/gcp-fswe-grid.sh: one VM per serving host, all started in the same minute, each sweeping both models over the same tasks. Task images do not depend on the model, so each VM builds once and sweeps the models over the built environments. pd-standard boot disks: pd-balanced counts against SSD_TOTAL_GB (500/region), which cannot hold eight VMs. - scripts/analyze_grid.py: pools per-call ledgers by (model, route) across tasks, tests each pinned arm against the unpinned one with Holm correction, and prints whether the host ranking survives changing the model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…l raises One grid fans the same --host-model mapping out to every arm, and only the Doubleword arms of that grid have a Doubleword provider, so 'doubleword=<id>' on an OpenRouter-only arm must not be an error. Caught live: six of eight arms in the FrontierSWE grid died on 'keys match no provider' before their first call. A key that names no known provider kind is still a typo and raises, so a misspelling cannot silently leave an arm on the wrong model id. Also makes the grid runner's clone idempotent so an arm can be relaunched in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…tatus OpenRouter's endpoint listing reports its own belief about whether a host is up. Without our own upstream key we sit on OpenRouter's shared rate-limit pool for that host, where a listed-up endpoint can return 429 on every call for one model while serving another fine. Measured on the live listing for z-ai/glm-5.3-flash: 17 of 23 hosts answered, and DeepInfra, Baseten, Makora, DigitalOcean, Fireworks and io.net returned 429 despite being listed up. Six of eight arms in tonight's grid would have been picked from that listing. --probe sends one 8-token pinned call per host and prints what actually happened, with the served provider echoed back on success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…lled total A host that reports no cost per call must not print $0.0000: a null is not a measured zero, and a zero in a cost column reads as 'this host is free'. Doubleword is that host, so the grid analyzer now prints -- for it. dw_cost_attribution.py supplies the missing number the honest way: dw usage gives the BILLED total per model for a window (measured by Doubleword's own meters), and the split across the realtime and flex tiers comes from each tier's share of the prompt tokens our ledger recorded, since dw usage reports by model and not by tier. The output labels which half is measured and which is derived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…easons The rollup table has no column for the per-call fields that explain it. Routing spread says what the unpinned arm actually did; pin verification says whether a pinned arm stayed put, which is the claim a pinned run rests on; reasoning-token share and finish reasons explain cost and truncation that token totals alone do not. First look at the FrontierSWE grid: reasoning is 39-61% of completion tokens depending on the model, which is a cost driver invisible in a plain input/output split. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
The question is not how a host did but whether the host you would pick stays the host you would pick when the model changes, so every panel puts both models on the same host row. A bar pair that disagrees is a host-model pair, not a host property. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…n model
Agentic workloads resend a growing transcript every turn, so the prompt cache
sets the input bill. Whether a cache works turns out not to be a property of
the host: it is a property of (host, model, opt-in). Measured 2026-09-03
through the pinning proxy, growing prefix, steady state over turns 2-6:
doubleword/realtime and doubleword/flex, both tiers identical
deepseek-ai/DeepSeek-V4-Flash-0731 0.0% unmarked 97.7% marked
zai-org/GLM-5.3-Flash 0.0% unmarked 0.0% marked
So the opt-in trap is real (0% for a stock client either way), and separately
Doubleword reports no cached tokens at all for GLM 5.3 Flash while caching
DeepSeek V4 Flash almost completely on the same endpoint. That explains both
Doubleword arms of the FrontierSWE grid reading 0% cache.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
Cost is the headline the chart existed to show and it was missing. Label precision now follows the panel's own scale: dollars per million tokens are sub-unit and would all print as 0, while a percentage does not want three decimals. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…ories An arm directory can hold trees kept as evidence, such as the rate-limited ledgers of a host swapped out mid-grid. Pooled into the results table a fully 429'd host reads as a well-behaved arm with a 0% incomplete rate, because a 429 is an error and not a hang. Only ledgers directly under an 'out' directory are results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
…erence The reference page is generated from --help at build time, so adding CLI flags made it stale and CI caught it. Also writes up what --probe is for: OpenRouter's STATUS column is its own belief about an endpoint, and six of twenty-three hosts listed as up for GLM 5.3 Flash returned 429 on a real call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
Dividing one billed total by token share necessarily hands both tiers the same rate, so the equal per-tier figures are a property of the arithmetic, not evidence that flex and realtime cost the same. Doubleword's published flex rate is below its realtime rate. Someone reading the table would otherwise quote the equality as a finding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FrontierSWE v2 ships 34 Harbor-schema tasks but not its runner (
px-evalis unreleased). Harbor's--pathtakes a local task or dataset directory, so a checkout plus a pinned commit is enough to run them.--task-pathoncompound-bench harbor, threaded throughbuild_commandandsweep_harbor; passing both a dataset and a path, or neither, raises.scripts/cloud/gcp-fswe-grid.sh: one VM per serving host, all started within the same minute so congestion is shared, each sweeping both models over the same tasks. Images are built once per VM and reused across models.scripts/analyze_grid.py: pools ledgers by (model, route), Holm-corrected tests against the unpinned arm, and a cross-model ranking table.Verified: a single-VM probe built
crash-proof-flash-filesystemin ~4 minutes and ran terminus-2 against it end to end. Full test suite green.🤖 Generated with Claude Code
https://claude.ai/code/session_01HiNXJfaq4WnmsHJywPhpR9