fix(qa): prevent grader truncation

[skyfrostmage]

The QA grader inherits the default 4096 output-token ceiling, which on Gemini is shared with dynamic thinking, so long grading JSON is cut off mid-object, fails to parse, and score/summary come back empty.

Thread an optional max_tokens through the LLM factory and have the QA path request a generous ceiling (16384) with a bounded thinking budget on the Google branch, so a full grade completes. The param is optional; non-QA callers are unaffected.

Fixes #636

---

Summary by cubic

Fixes QA grader truncation on Gemini by raising the output token limit and using model-compatible thinking settings, now applied to both Google and Vertex paths. Adds stricter prefix-based model matching to avoid unsupported configs; only the QA path opts in.

• Bug Fixes
  • Thread optional max_tokens through LLM factory methods.
  • For Google and Google Vertex, apply max_tokens and choose thinking by model-family prefix (2.5 -> thinking_budget=4096, 3 -> thinking_level="low", others omit); covered by tests, including settings wiring for GoogleLLMSettings and GoogleVertexLLMSettings.
  • QA workflow sets QA_MAX_OUTPUT_TOKENS=16384 and passes it with usage_context="qa_analysis".

^{Written for commit 2464dd2. Summary will update on new commits.}

Greptile Summary

QA grading now carries an explicit output-token limit through the LLM factory and uses thinking configuration only for supported Gemini model families. The previously reported custom-model failure was disproved: Google and Vertex requests for legacy or custom model IDs retain the output limit while omitting unsupported thinking settings.

Confidence Score: 5/5

No blocking failure remains.

Runtime checks exercised Google and Vertex factory behavior for supported, legacy, and custom model identifiers, confirming that incompatible thinking configuration is not attached.

T-Rex Logs

What T-Rex did

• Executed a controlled runtime harness against predecessor and current Google and Vertex factory branches using Gemini 2.5 and 3 identifiers, legacy Gemini 1.5, and the acme-gemini-2.5-wrapper.
• Validated the thinking configuration changes so that the custom-model request no longer receives an unsupported thinking configuration, with Gemini 2.5 applying thinking_budget and Gemini 3 applying thinking_level while legacy and custom identifiers omit thinking.
• Compared pre- and post-fix behavior and confirmed that after the prefix-match fix Gemini 2.5 uses thinking_budget=4096, Gemini 3 uses thinking_level=low, and Gemini 1.5 omits thinking, with the authored runtime script and paired log outputs documenting the results.

_{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "fix(qa): tighten Google model-family mat..." | Re-trigger Greptile}

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/services/workflow/qa/llm_config.py">

<violation number="1" location="api/services/workflow/qa/llm_config.py:54">
P2: max_tokens is threaded unconditionally from the QA path for every provider, but create_llm_service_from_provider only honors it in the GOOGLE branch (service_factory.py ~1005). The GOOGLE_VERTEX and Anthropic branches accept the new parameter and silently ignore it. The comment added next to QA_MAX_OUTPUT_TOKENS explicitly notes that Anthropic also defaults to 4096 output tokens, so an Anthropic (or Vertex) QA grader will still hit the same mid-object truncation that this PR is fixing, while the caller believes a generous ceiling was requested. Consider applying the output ceiling on the Anthropic branch (and Vertex if it shares the same cap) or scoping the QA flag to the provider actually covered, so the fix is not silently ineffective for other providers.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: max_tokens is threaded unconditionally from the QA path for every provider, but create_llm_service_from_provider only honors it in the GOOGLE branch (service_factory.py ~1005). The GOOGLE_VERTEX and Anthropic branches accept the new parameter and silently ignore it. The comment added next to QA_MAX_OUTPUT_TOKENS explicitly notes that Anthropic also defaults to 4096 output tokens, so an Anthropic (or Vertex) QA grader will still hit the same mid-object truncation that this PR is fixing, while the caller believes a generous ceiling was requested. Consider applying the output ceiling on the Anthropic branch (and Vertex if it shares the same cap) or scoping the QA flag to the provider actually covered, so the fix is not silently ineffective for other providers.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At api/services/workflow/qa/llm_config.py, line 54:

<comment>max_tokens is threaded unconditionally from the QA path for every provider, but create_llm_service_from_provider only honors it in the GOOGLE branch (service_factory.py ~1005). The GOOGLE_VERTEX and Anthropic branches accept the new parameter and silently ignore it. The comment added next to QA_MAX_OUTPUT_TOKENS explicitly notes that Anthropic also defaults to 4096 output tokens, so an Anthropic (or Vertex) QA grader will still hit the same mid-object truncation that this PR is fixing, while the caller believes a generous ceiling was requested. Consider applying the output ceiling on the Anthropic branch (and Vertex if it shares the same cap) or scoping the QA flag to the provider actually covered, so the fix is not silently ineffective for other providers.</comment>

<file context>
@@ -47,6 +51,7 @@ async def create_qa_llm_service(
             api_key,
             correlation_id=correlation_id,
             usage_context=QA_USAGE_CONTEXT,
+            max_tokens=QA_MAX_OUTPUT_TOKENS,
             **kwargs,
         )
</file context>

[cubic-dev-ai]

2 issues found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/services/pipecat/service_factory.py">

<violation number="1" location="api/services/pipecat/service_factory.py:945">
P1: QA grading still sends `thinking_level`/`thinking_budget` to custom or future model IDs containing these family substrings, so incompatible Google models can reject the request. Use an explicit capability allowlist (including supported revisions) and return `None` for every unrecognized ID.</violation>
</file>

<file name="api/tests/test_google_thinking_config.py">

<violation number="1" location="api/tests/test_google_thinking_config.py:15">
P2: These tests only call the isolated helper `_google_thinking_for_model`, so they never validate the actual QA wiring the PR is supposed to guard: that `max_tokens=16384` flows through `create_llm_service_from_provider` and attaches the thinking config to the `GoogleLLMSettings`/`GoogleVertexLLMSettings`, or that those settings serialize without the grader rejecting the request (the stated P1). The PR review-focus explicitly asked for a legacy/custom model at QA max_tokens=16384 and a Gemini 2.5 serialization test; consider adding one that constructs the settings with `max_tokens` + thinking and serializes them.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: QA grading still sends thinking_level/thinking_budget to custom or future model IDs containing these family substrings, so incompatible Google models can reject the request. Use an explicit capability allowlist (including supported revisions) and return None for every unrecognized ID.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At api/services/pipecat/service_factory.py, line 945:

<comment>QA grading still sends `thinking_level`/`thinking_budget` to custom or future model IDs containing these family substrings, so incompatible Google models can reject the request. Use an explicit capability allowlist (including supported revisions) and return `None` for every unrecognized ID.</comment>

<file context>
@@ -933,6 +933,22 @@ def _migrate_deprecated_google_model(model: str) -> str:
+    ``max_tokens`` ceiling alone protects the grader output from truncation.
+    """
+    ml = (model or "").lower()
+    if "gemini-2.5" in ml or "gemini-2-5" in ml:
+        return GoogleLLMService.ThinkingConfig(thinking_budget=4096)
+    if "gemini-3" in ml:
</file context>

[cubic-dev-ai]

P2: These tests only call the isolated helper _google_thinking_for_model, so they never validate the actual QA wiring the PR is supposed to guard: that max_tokens=16384 flows through create_llm_service_from_provider and attaches the thinking config to the GoogleLLMSettings/GoogleVertexLLMSettings, or that those settings serialize without the grader rejecting the request (the stated P1). The PR review-focus explicitly asked for a legacy/custom model at QA max_tokens=16384 and a Gemini 2.5 serialization test; consider adding one that constructs the settings with max_tokens + thinking and serializes them.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At api/tests/test_google_thinking_config.py, line 15:

<comment>These tests only call the isolated helper `_google_thinking_for_model`, so they never validate the actual QA wiring the PR is supposed to guard: that `max_tokens=16384` flows through `create_llm_service_from_provider` and attaches the thinking config to the `GoogleLLMSettings`/`GoogleVertexLLMSettings`, or that those settings serialize without the grader rejecting the request (the stated P1). The PR review-focus explicitly asked for a legacy/custom model at QA max_tokens=16384 and a Gemini 2.5 serialization test; consider adding one that constructs the settings with `max_tokens` + thinking and serializes them.</comment>

<file context>
@@ -0,0 +1,29 @@
+def test_gemini_25_uses_thinking_budget():
+    tc = _google_thinking_for_model("gemini-2.5-flash")
+    assert tc is not None
+    assert tc.thinking_budget == 4096
+    assert getattr(tc, "thinking_level", None) is None
+
</file context>

[skyfrostmage]

Addressed the review feedback: fence/truncation guard now model-family-aware + Vertex covered (2464dd2); whole-call routing test added (65d1821).