docs(harbor): pin harbor==0.22.0 and make the pin checkable - #341
Open
vaibhavdabas16 wants to merge 1 commit into
Open
docs(harbor): pin harbor==0.22.0 and make the pin checkable#341vaibhavdabas16 wants to merge 1 commit into
vaibhavdabas16 wants to merge 1 commit into
Conversation
Every Harbor invocation in docs/harbor.md and src/README.md pinned harbor==0.15.0. Upstream is 0.22.0, six releases on: readers following these docs installed a stale Harbor, and readers who already had a current one found our pin fighting their install. Bump all eight pins to 0.22.0 and say in the docs what that claim rests on, because "smoke-test the adapter output against current Harbor" is the actual ask in TIGER-AI-Lab#294 -- a pin nobody has exercised is how we got here. Verified against harbor 0.22.0: - All 129 tasks from `clawbench-harbor-adapt` construct as harbor.models.task.task.Task with no errors, and Task.is_valid_dir accepts every generated directory. - Every flag these docs use still exists on `harbor run`: -p, -a, -m, --ak, --env-file, --ve, --jobs-dir, -n, -k, -r, --timeout-multiplier and the four per-phase multipliers. Not verified, and the docs say so: no end-to-end `harbor run` against live sites. That needs Docker, a browser container per trial and judge credentials. Loading proves the format is accepted, not that a sweep passes. One correction to the issue. It reads 0.15's `harbor run -p <path>` as having "given way to" hub-published `-d org/dataset@version`. Both exist in 0.22.0 -- `-d` is an addition for hub datasets, not a replacement -- so the local-path commands in these docs were never broken by the version drift, only the pin was. Hub publishing stays in TIGER-AI-Lab#331. tests/test_harbor_version_compatibility.py re-runs the task-loading half of that check. Harbor is not a ClawBench dependency, so it skips unless Harbor is importable: uv run --with harbor==<new-version> pytest \ tests/test_harbor_version_compatibility.py It converts the real V2 corpus rather than a fixture -- a hand-built task would only prove the fixture loads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #294 (ask 1). Ask 2 — whether to publish the V2 dataset to the Harbor hub — is a maintainer decision and stays in #331.
Eight Harbor invocations across
docs/harbor.mdandsrc/README.mdpinnedharbor==0.15.0. Upstream is 0.22.0. Readers following our docs installed a six-releases-stale Harbor; readers who already had a current one found our pin fighting their install.Bumping the number is the easy half. The issue's actual ask is "smoke-test
clawbench-harbor-adaptoutput against Harbor 0.22 and note the supported range" — a pin nobody has exercised is how we got here — so this PR does the check and leaves behind a way to repeat it.Verified against harbor 0.22.0
clawbench-harbor-adaptconstruct asharbor.models.task.task.Taskwith no errors, andTask.is_valid_diraccepts every generated directory.harbor run:-p,-a,-m,--ak,--env-file,--ve,--jobs-dir,-n,-k,-r,--timeout-multiplierand the four per-phase multipliers.Not verified, and
docs/harbor.mdsays so: no end-to-endharbor runagainst live sites. That needs Docker, a browser container per trial and judge credentials. Loading proves the format is accepted; it does not prove a sweep passes.One correction to the issue
Both exist in 0.22.0:
-dis an addition for hub datasets, not a replacement. So the local-path commands in these docs were never broken by the version drift — only the pin was. Worth stating plainly, because "our documented commands don't work on current Harbor" and "our pin is stale" call for different amounts of work, and it is the second one.Making the next bump checkable
tests/test_harbor_version_compatibility.pyre-runs the task-loading half of the check. Harbor is not a ClawBench dependency, so it skips unless Harbor is importable:It converts the real V2 corpus through
harbor_adapter.main()rather than a hand-built fixture — a synthetic task would only prove the fixture loads. What has to hold is that the corpus we tell people to convert produces a dataset Harbor accepts.A second test records the installed Harbor version and skips with a clear message when it differs from the one the docs claim, so a mismatch is visible in test output rather than silently passing under a version nobody documented.
Verification
213 passed, 5 skipped— the new file skips, no change tomain's count. (test_host_tasks.py::…[v1-lite]fails onmaintoo:v1-litesymlinks check out as text on Windows. Unrelated.)2 passed in 5.44sagainstharbor==0.22.0.Merge notes
Docs and one new test file only; no source changes.
CHANGELOG.mdconflicts textually with my other open PRs (#339, #340); trivial either way.