Skip to content

docs(harbor): pin harbor==0.22.0 and make the pin checkable - #341

Open
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:docs/harbor-version-pin
Open

docs(harbor): pin harbor==0.22.0 and make the pin checkable#341
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:docs/harbor-version-pin

Conversation

@vaibhavdabas16

Copy link
Copy Markdown
Contributor

Closes #294 (ask 1). Ask 2 — whether to publish the V2 dataset to the Harbor hub — is a maintainer decision and stays in #331.

Eight Harbor invocations across docs/harbor.md and src/README.md pinned harbor==0.15.0. Upstream is 0.22.0. Readers following our docs installed a six-releases-stale Harbor; readers who already had a current one found our pin fighting their install.

Bumping the number is the easy half. The issue's actual ask is "smoke-test clawbench-harbor-adapt output against Harbor 0.22 and note the supported range" — a pin nobody has exercised is how we got here — so this PR does the check and leaves behind a way to repeat it.

Verified against harbor 0.22.0

  • All 129 tasks from clawbench-harbor-adapt construct as harbor.models.task.task.Task with no errors, and Task.is_valid_dir accepts every generated directory.
  • Every flag these docs use still exists on harbor run: -p, -a, -m, --ak, --env-file, --ve, --jobs-dir, -n, -k, -r, --timeout-multiplier and the four per-phase multipliers.

Not verified, and docs/harbor.md says so: no end-to-end harbor run against live sites. That needs Docker, a browser container per trial and judge credentials. Loading proves the format is accepted; it does not prove a sweep passes.

One correction to the issue

0.15's local-path form harbor run -p ./harbor-datasets/clawbench-v2 has given way to hub-published datasets addressed as harbor run -d org/dataset@version

Both exist in 0.22.0:

| --path      -p  <path>  Path to a local task or dataset directory
| --dataset   -d  <str>   Dataset name@version (e.g., 'dataset@1.0')

-d is an addition for hub datasets, not a replacement. So the local-path commands in these docs were never broken by the version drift — only the pin was. Worth stating plainly, because "our documented commands don't work on current Harbor" and "our pin is stale" call for different amounts of work, and it is the second one.

Making the next bump checkable

tests/test_harbor_version_compatibility.py re-runs the task-loading half of the check. Harbor is not a ClawBench dependency, so it skips unless Harbor is importable:

uv run --with harbor==<new-version> pytest tests/test_harbor_version_compatibility.py

It converts the real V2 corpus through harbor_adapter.main() rather than a hand-built fixture — a synthetic task would only prove the fixture loads. What has to hold is that the corpus we tell people to convert produces a dataset Harbor accepts.

A second test records the installed Harbor version and skips with a clear message when it differs from the one the docs claim, so a mismatch is visible in test output rather than silently passing under a version nobody documented.

Verification

  • Default suite: 213 passed, 5 skipped — the new file skips, no change to main's count. (test_host_tasks.py::…[v1-lite] fails on main too: v1-lite symlinks check out as text on Windows. Unrelated.)
  • With Harbor present: 2 passed in 5.44s against harbor==0.22.0.

Merge notes

Docs and one new test file only; no source changes. CHANGELOG.md conflicts textually with my other open PRs (#339, #340); trivial either way.

Every Harbor invocation in docs/harbor.md and src/README.md pinned
harbor==0.15.0. Upstream is 0.22.0, six releases on: readers following
these docs installed a stale Harbor, and readers who already had a
current one found our pin fighting their install.

Bump all eight pins to 0.22.0 and say in the docs what that claim rests
on, because "smoke-test the adapter output against current Harbor" is
the actual ask in TIGER-AI-Lab#294 -- a pin nobody has exercised is how we got here.

Verified against harbor 0.22.0:

- All 129 tasks from `clawbench-harbor-adapt` construct as
  harbor.models.task.task.Task with no errors, and Task.is_valid_dir
  accepts every generated directory.
- Every flag these docs use still exists on `harbor run`: -p, -a, -m,
  --ak, --env-file, --ve, --jobs-dir, -n, -k, -r, --timeout-multiplier
  and the four per-phase multipliers.

Not verified, and the docs say so: no end-to-end `harbor run` against
live sites. That needs Docker, a browser container per trial and judge
credentials. Loading proves the format is accepted, not that a sweep
passes.

One correction to the issue. It reads 0.15's `harbor run -p <path>` as
having "given way to" hub-published `-d org/dataset@version`. Both exist
in 0.22.0 -- `-d` is an addition for hub datasets, not a replacement --
so the local-path commands in these docs were never broken by the
version drift, only the pin was. Hub publishing stays in TIGER-AI-Lab#331.

tests/test_harbor_version_compatibility.py re-runs the task-loading half
of that check. Harbor is not a ClawBench dependency, so it skips unless
Harbor is importable:

    uv run --with harbor==<new-version> pytest \
      tests/test_harbor_version_compatibility.py

It converts the real V2 corpus rather than a fixture -- a hand-built task
would only prove the fixture loads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(harbor): docs pin harbor==0.15.0 but upstream is 0.21.0 with a changed dataset interface

2 participants