Bug: shipped Anthropic provider config in providers.json is incompatible with real Claude models

[sahiee-dev]

Problem

providers.json ships an anthropic provider entry (models claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5) with "structured_output": "json_object", and evaluator.py/pdf.py always send temperature and top_p in the request body. Using a real ANTHROPIC_API_KEY against this config fails on the very first request:

400 {"error":{"code":"invalid_request_error","message":"`temperature` is deprecated for this model.","type":"invalid_request_error","param":null}}

After dropping temperature, top_p also fails the same way:

400 {"error":{"code":"invalid_request_error","message":"`top_p` is deprecated for this model.","type":"invalid_request_error","param":null}}

And response_format.type: "json_object" is rejected outright:

400 {"error":{"code":"invalid_request_error","message":"response_format.type: Input should be 'json_schema'","type":"invalid_request_error","param":null}}

Switching to "json_schema" then requires a strict flag and a schema with additionalProperties: false set recursively — which EvaluationData.model_json_schema() (a plain Pydantic export) doesn't produce, so that path also 400s:

400 {"error":{"code":"invalid_request_error","message":"response_format.json_schema.schema: For 'object' type, 'additionalProperties' must be explicitly set to false","type":"invalid_request_error","param":null}}

Net effect: the anthropic entry in providers.json cannot currently complete a single request end-to-end against the real API.

Repro

DEFAULT_MODEL=claude-sonnet-5
ANTHROPIC_API_KEY=<real key>
python score.py ./resume/sample.pdf

Fails with the 400 above on both the GitHub-project-selection call and the final evaluation call.

Fix that worked

• Added a supports_temperature flag (default true) to the provider config, threaded through config.provider_for() → llm_utils.initialize_llm_provider() → OpenAICompatibleProvider, and skip sending temperature/top_p when it's false.
• Set "structured_output": "none" for the anthropic entry so no response_format is sent at all — the existing prompt-only JSON instructions (and extract_json_from_response's markdown-fence stripping) are enough; claude-sonnet-5 follows them reliably.

With those two changes, a full score.py run completes end-to-end against real claude-sonnet-5.

[ayo0la]

I can independently confirm the config half of this. providers.json ships temperature: 0.1 and top_p: 0.9 for claude-opus-4-8 and claude-sonnet-5, but both parameters were removed on Claude Opus 4.7 and later — the API rejects them outright, which matches the ```temperature is deprecated for this model 400 you hit. `claude-haiku-4-5` is the only one of the three shipped models that still accepts them, so it's the sole entry that should keep those keys.

OpenAICompatibleProvider.chat() only adds each key when it's present in options, so the sampling half needs no new flag — omitting them from the per-model config in providers.json is enough, and it keeps the fix declarative rather than adding a provider-level boolean. The structured_output half does need a change, as you found.

@sahiee-dev are you planning to open a PR for this? You have the working fix and the live repro, so it's yours if you want it. If you'd rather not, I'm happy to pick it up — I'd credit your diagnosis in the PR. Just don't want us both writing the same patch.

[sahiee-dev]

Thanks for confirming, and good catch on claude-haiku-4-5 still accepting the sampling params, you're right that dropping the keys per-model in providers.json is simpler than a provider-level flag.

I've patched this locally for my own use and it's working, but I don't have the bandwidth to put together a proper PR right now. Go ahead and raise one, happy to see this fixed either way.