feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges

[kweinmeister]

Summary

When running Tier 3 evaluations with higher concurrency or across multi-case datasets, LLM judge grading (accuracy, goal_accuracy, and behavior_check) can fail in two ways:

1. Judge models with internal reasoning tokens or verbose output can emit preamble text or exhaust their output token budget before closing the JSON object, causing _parse_json_response to fail even with max_tokens=4096.
2. Parallel Harbor trial containers hitting the same judge endpoint at the end of a wave can trigger HTTP 429 rate limits or transient 502/503/504 gateway errors.

This change updates both the host judge path (src/skillevaluator/inference/client.py, src/skillevaluator/tier3/eval_core/llm_judge.py) and the container verifier (src/skillevaluator/tier3/harbor/templates/eval.py) to:

• Pass explicit response_schema and schema_name definitions (ACCURACY_JUDGE_SCHEMA, GOAL_ACCURACY_JUDGE_SCHEMA, BEHAVIOR_CHECK_JUDGE_SCHEMA) to the LLM client (response_format for OpenAI-compatible endpoints and output_config for Anthropic endpoints).
• Automatically downgrade once without the schema constraint (response_format={"type": "json_object"} for OpenAI-compatible, or prompt-based JSON for Anthropic) if a provider returns HTTP 400 or 422, logging a warning and memoizing the target so subsequent judge calls do not repeat failed schema requests.
• Retry HTTP 429 and transient 5xx/connection errors using exponential backoff with full jitter and Retry-After header parsing, configurable via SKILL_EVAL_LLM_MAX_RETRIES, SKILL_EVAL_LLM_RETRY_BASE_DELAY, and SKILL_EVAL_LLM_RETRY_MAX_DELAY.

Verification

• I am familiar with the Contributing Guidelines
• Added or updated focused tests
• Updated documentation for user-visible changes
• Ran make lint
• Ran make test
• Ran make build
• Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

• No user-visible release note needed
• Updated CHANGELOG.md

[rng1995]

@kweinmeister Two issues remain in the new retry/schema behavior: SDK retries multiply the configured budget, and unrelated HTTP 400/422 responses permanently disable structured output for the target. Details and reproductions are inline.

Local validation: 879 focused tests passed. Separate reproductions using the real OpenAI/Anthropic SDKs with mock HTTP transports exposed these gaps. GitHub reports all 17 checks passing. Please address the two cases and add transport-level regressions before approval.

[kweinmeister]

@rng1995 Thanks for the review and the clear reproductions. Both issues have been resolved in commit 7280545:

1. Outer loop owns retry budget: Passed max_retries=0 to both OpenAI and Anthropic constructors and exposed http_client on LLMClient. Verified via mock HTTP transports that max_retries=0 sends 1 request and max_retries=3 sends 4 requests for both providers.
2. Selective schema fallback & deferred memoization: Restricted schema failure detection to genuine unsupported-option error payloads in both host and container template runtimes, and deferred _SCHEMA_UNSUPPORTED_TARGETS memoization until the prompt-only fallback confirms success. Verified that two context_length_exceeded errors followed by a valid prompt maintain [True, True, True] schema flags.

All 6,519 tests and make lint / package build pass cleanly. Ready for another look!

[rng1995]

@kweinmeister Thanks for addressing both earlier findings. I verified that SDK retries are disabled and that schema fallback only caches a genuine unsupported-option response after a successful downgrade. Both existing threads can remain resolved.

One integration issue remains: the normal Harbor runner drops the three retry environment variables before staging and launching the verifier, so the documented settings are ignored there. Details and the reproduction are inline. Please forward those controls through the runner and add an integration regression before approval.

Validation on 248d129: 764 focused tests passed locally, plus independent SDK/schema and environment-forwarding probes. GitHub reports all 17 checks passing; the PR has no merge conflicts.

[rng1995]

[P2] Forward retry settings through the Harbor runner

Adding these names to the adapter allowlist does not forward them from the host: runner._provider_environment() only copies the judge-model overrides, and _harbor_subprocess_environment() filters out the remaining ambient variables. With host settings MAX_RETRIES=0, RETRY_BASE_DELAY=0.1, and RETRY_MAX_DELAY=0.5 (all prefixed SKILL_EVAL_LLM_), the real runner produces neither verifier task entries nor subprocess values for these keys; _resolve_eval_retry_config() consequently returns (3, 1.0, 30.0). A normal Tier 3 run therefore makes extra requests even when retries are explicitly disabled, and ignores the requested delay limits. Please explicitly forward these three operator-owned settings through the runner's provider/verifier environment so both staging and execution receive them. Add a regression starting from the host environment and exercising the runner path; the current test manually supplies runtime_env, bypassing the missing step.

[kweinmeister]

Good catch. The Harbor runner's provider environment helper now picks up the three SKILL_EVAL_LLM_* retry variables from the host environment so they get passed into both the staged task.toml verifier section and the Harbor subprocess environment.

I added an integration test in tests/test_harbor_task_toml_retry_env.py that sets those retry variables on the host, runs through the Harbor runner and task staging, and verifies that the verifier resolves (0, 0.1, 0.5), along with unit tests in tests/test_harbor_runner_environment.py. I also ran an end-to-end Tier 3 smoke test to confirm the staged task.toml and live LLM judge evaluation pick up the settings as expected.

[kweinmeister]

@rng1995 Thanks for catching that. I pushed an update to forward the three retry environment variables through the Harbor runner into both task staging and the Harbor subprocess environment, added an integration regression test covering the full path from os.environ through verifier config resolution, and verified it with an end-to-end Tier 3 smoke test. Ready for another look when you have a chance.