feat(tier3): upgrade Harbor to 0.22.0

[chrisknvidia]

Closes #79

Summary

• Upgrade the pinned and locked Tier 3 backend from Harbor 0.13.2 to the current stable release, Harbor 0.22.0.
• Adapt command construction, secure Docker and trusted-local execution, installed-agent lifecycles, task staging, multi-step rewards, result collection, attempt accounting, and unified reports to Harbor 0.22 contracts.
• Keep native task selectors, logical dataset IDs, task display names, and structural attempt ordinals separate and fail closed on ambiguous identity or incomplete results.
• Preserve paired-baseline isolation by rejecting skill-controlled setup, healthchecks, prior trajectories, step workdirs/tests, verifier controls, and collect hooks where they could contaminate standard grading; retain Harbor-native ownership in custom_only mode.
• Harden diagnostics and artifacts against credential/userinfo leakage, bound report/error payloads, validate finite timeout values, and make mixed metric contracts and unavailable telemetry explicit.
• Bring local OpenCode NVIDIA Build execution into parity with Harbor 0.22's instruction/error lifecycle.

Harbor v0.22.0 is the latest upstream release as of 2026-08-26: https://github.com/harbor-framework/harbor/releases/tag/v0.22.0

Verification through ca3e731

The full local suites, packaging gates, and real Harbor runs below were collected on implementation snapshot 8b0110f. Follow-ups a783c2a and ca3e731 are limited to a behavior-preserving helper rename plus persistence assertion and a portable test invocation; focused local regressions and the complete exact-head GitHub matrix cover those diffs.

Automated

• Python 3.12 full suite: 6378 passed, 19 skipped, 9 deselected in 594.56s.
• Isolated Python 3.13 full suite with all extras and editable source: 6378 passed, 19 skipped, 9 deselected in 578.52s.
• Real-process integration slice for Docker sidecar cleanup, redaction, interrupted lifecycle, and local streaming: 7 passed in 52.58s.
• Report-focused suites: 350 passed, 4 skipped; direct report regression slice: 11 passed.
• Packaging/boundary suite: 241 passed in 39.56s; public benchmark gate passed.
• Wheel and sdist build plus strict Twine checks passed. Isolated wheel smoke installs passed on Python 3.12.12 and 3.13.11, including CLI, doctor, packaged templates, and harbor.__version__ == "0.22.0".
• Ruff lint passed; all 28 changed Python files pass Ruff format checking; git diff --check passed. The repository-wide format check still lists 18 files, all byte-identical to origin/main.
• Bandit high-severity scan: zero findings. pip-audit: no known vulnerabilities (the unpublished local skillevaluator package is skipped). Path-aware Gitleaks scans passed for the tracked diff and all 30 branch commits.
• Fern docs validation: 0 errors, 1 warning; redirect validation was unavailable without Fern authentication.
• Post-review CodeQL follow-up: the name-based sensitive-data heuristic was removed by renaming an integer-only attempt helper from trusted to structural; persistence coverage now also proves that all runner-internal underscore metadata is omitted from reward.json. Focused identity tests: 3 passed; long stop-on-pass merge path: 1 passed; broader collector suites: 176 passed.
• Windows Tier 2 exposed a CI-only harness bug: Windows resolved bash to the WSL launcher instead of a POSIX shell. The isolation test now invokes the staged evaluator with sys.executable -I on every OS while separately asserting the generated shell contract; the full isolation/judge slice passed with 459 passed, 2 skipped.
• All 30 commits have DCO sign-offs.
• Exact-head GitHub validation passed all 15 checks: Python 3.12/3.13, Windows Tier 2 (1292 passed, 30 skipped), native-Windows fail-closed, macOS Tier 2/Tier 3, RHEL 8, packaging, dependency review, DCO, Gitleaks, the Security CodeQL job, and the separate GitHub CodeQL gate. Alert fix: upgrade langchain-core for CVE-2026-45134 #10 is fixed by fresh analysis with zero results; it was not dismissed or suppressed.

Real Harbor 0.22 integration paths

1. Native Harbor BYOT / Docker / Codex / custom-only / skip baseline

   • Real Docker agent execution, step-local verifier, collection, retained jobs, and report completed in 220.5s.
   • Deliberately distinct selector (selector-attempt2), logical entry ID (logical-attempt9), and display name (publisher/display-attempt7) remained distinct.
   • The single step and custom reward scored overall=1.0 and native_quality=1.0; the staged task contained only the intended step-local test overlay.

2. Generated dataset / Docker / Codex / runtime setup / unified reports

   • Blocking top-level validation completed with exit 0 in 4m38s using custom_only and --skip-baseline.
   • A non-credential runtime_env value reached the verifier, and two pre-agent setup commands persisted an independently checked artifact before agent execution.
   • Custom metrics canary_protocol=1.0 and runtime_setup=1.0; 1/1 expected/scored attempt; no baseline or lift was synthesized.
   • JSON, HTML, and Markdown reports were generated and inspected. JSON records Tier 3 succeeded, overall 1.0, and both custom metrics; HTML contains both evaluator cards; Markdown records the overall passed validation and completed Tier 3 stage. The Tier 3 verdict is correctly neutral because this was custom-only, skip-baseline evidence with no lift.

3. Native Harbor BYOT / local / OpenCode / NVIDIA Build / custom-only

   • Required macOS Seatbelt sandbox, network explicitly enabled, and ambient agent-key inheritance disabled.
   • Real OpenCode 1.2.26 execution, multi-step ATIF conversion, step-local verification, collection, and report completed with exit 0 in 84.6s.
   • 1/1 expected/scored attempt; overall=1.0, native_quality=1.0; logical attempt accounting correctly recorded actual attempt 1 despite attempt-like selector/ID text.
   • ATIF v1.7 retained 9 steps and real totals of 110,717 input, 84,480 cached, and 3,568 output tokens. No matching local process or container remained after completion.

4. Generated dataset / Docker / Codex / paired baseline / standard + custom grading

   • Docker runtime preflight passed on a real model request.
   • After making the negative-control prompt terminate deterministically, controlled paired runs completed both agent arms and the with-skill arm scored all standard metrics plus its custom metric.
   • The external standard accuracy judge returned malformed non-JSON for the baseline, including once with the stronger model selected through the new verifier-only judge override. SkillEvaluator correctly treated the reward as unscoreable and failed closed, so current-head paired lift is not claimed as fully scored.

Exact provider-key scans found no configured provider credential in retained real-run artifacts. Temporary E2E fixtures/results were removed from the worktree after inspection; no fixture or generated runtime token is included in this branch.

Review

• Independent Review Guru passes found no remaining P1/P2 implementation defect.
• Review covered native identity and attempt spoofing, baseline contamination, standard/custom verifier boundaries, local and Docker lifecycle cleanup, option interactions, finite timeouts, error/truncation accounting, mixed-contract report truth, step/subagent trajectories, token semantics, secret/userinfo redaction, and regression behavior.
• A separate issue-to-diff audit confirmed the implementation covers issue [FEA]: Upgrade Tier 3 Harbor backend from 0.13.2 to 0.20+ #79's Harbor 0.22 compatibility surfaces and identified the runtime-setup and Markdown-report proof gaps; both are now covered by the successful generated Docker run above.
• Two independent follow-up reviews confirmed the CodeQL finding was a function-name heuristic rather than persisted sensitive data and recommended the behavior-preserving rename plus persistence-boundary assertion now included in a783c2a.

Remaining verification gaps

• Managed Harbor cloud backends were not exercised because no isolated approved target and credentials were available; the configured Kubernetes context is shared.
• Native Windows execution was not available locally. Windows behavior remains fail-closed by design and awaits exact-head CI confirmation.
• The current-head paired default-plus-custom path is not a fully scored lift success because the external baseline accuracy judge emitted malformed JSON as described above. Harbor execution, artifact collection, and fail-closed reward handling completed, but a publishable baseline score was not produced.
• Docker/Codex did not expose numeric Harbor token totals; reports correctly leave that telemetry unavailable. The local/OpenCode run above provides real token-metric coverage.

Draft status

This PR remains intentionally draft. Exact-head remote CI passed all 15 checks on ca3e731. Do not mark ready or merge until maintainer review is complete.

[rng1995]

One blocking merge-integration issue remains. Exact-head CI and DCO are green, but the collector conflict with current main must be resolved semantically before this can merge.

[rng1995]

@chrisknvidia Gentle ping when you have a chance: the remaining review thread is still waiting for an update or response. Please take a look when you can.

[rng1995]

Requesting changes after a full review of the Harbor 0.22 integration at f77b752. I reproduced nine issues: three P1 correctness/runtime blockers and six P2 compatibility, security, and reporting defects. Details and concrete fixes are inline. The 17 existing checks are green, but they do not exercise these paths. Separately, main advanced to d979722 during review and the PR now conflicts in CHANGELOG.md.

[rng1995]

Re-reviewed the complete PR at 173bea9 after merging current main and addressing all nine findings. The unsupported cua-cloud, opensandbox, and hf-sandbox modes now fail closed; canonical ranking and custom evidence precedence are aligned; positional compatibility and mode-scoped runtime environment validation are restored; unsafe trial roots retain distinct logical identities; empty Docker environments work; and Compose YAML limits are enforced before object construction. All review threads are resolved, the local full suite passed (7,265 passed, 20 skipped, 9 deselected), release/docs/security checks passed, all 17 GitHub checks are green, and GitHub reports the PR mergeable.