feat(qa): add whole-call grading toggle

[skyfrostmage]

Add qa_grade_whole_call on the QA node; when set, grade the whole transcript as one segment (a single verdict) instead of node-by-node. Needed when the scored content spans multiple nodes (e.g. gated questions in a later closing node), where per-node grading can't produce one authoritative result.

---

Summary by cubic

Adds a qa_grade_whole_call toggle to grade the entire transcript as one segment, returning a single verdict. Also adds regression tests that verify the toggle short-circuits to whole-call grading and skips node splitting.

• New Features
  • Added qa_grade_whole_call (default false) to the QA node spec; appears in the UI as “Grade Whole Call.”
  • When enabled, run_per_node_qa_analysis routes to _run_whole_call_qa_analysis to grade the full transcript once; existing per-node grading remains the default.

^{Written for commit 65d1821. Summary will update on new commits.}

Greptile Summary

Adds an opt-in QA-node setting for grading an entire transcript as one segment, while preserving node-by-node grading as the default. The enabled path forwards the expected inputs to whole-call grading and bypasses node splitting; the default path still reaches node splitting.

Confidence Score: 5/5

Safe to merge based on focused runtime coverage of both routing paths.

No blocking failure remains.

T-Rex Logs

What T-Rex did

• T-Rex ran the requested verification and reported that local artifact references were not uploaded.

_{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: "test(qa): cover whole-call grading toggl..." | Re-trigger Greptile}

[greptile-apps]

Whole-call toggle lacks orchestration coverage

The new qa_grade_whole_call early-return path has no test that invokes run_per_node_qa_analysis with nonempty feedback events and the toggle enabled. Existing coverage tests the regular per-node flow and the whole-call helper independently, so a future change that skips this branch, changes its arguments, or resumes node splitting would not be detected. Add an async orchestration test that mocks _run_whole_call_qa_analysis, enables the field, verifies its result is returned, and asserts node splitting is bypassed.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Focused whole-call QA routing regression harness

• An executable untracked harness that loads the changed analysis module, mocks the whole-call helper, and verifies both toggle states, with the takeaway that the enabled route is correctly selected.

Baseline toggle-off routing output

• The executed baseline harness output shows that toggle-off input retains the per-node path and does not await the whole-call helper, establishing the comparison condition.

Toggle-on whole-call routing output

• The executed regression harness output shows that toggle-on input returns the whole-call mock result and bypasses node splitting, proving the changed route works.

Existing test coverage search for whole-call QA routing

• The captured PR diff and complete matching test-suite references show that no existing test invokes the new toggle branch.

Attempted focused existing QA tests

• The captured pytest attempt stopped before collection because `api/conftest.py` requires an unavailable `dotenv` dependency, so the suite could not be run in this environment.

_{Ran code and verified through T-Rex}

[skyfrostmage]

Added test_qa_whole_call_routing.py covering this branch — it enables qa_grade_whole_call, mocks _run_whole_call_qa_analysis, and asserts its result is returned while split_events_by_node is never called. A companion test asserts the default path still reaches node splitting. (65d1821)

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/services/workflow/qa/analysis.py">

<violation number="1" location="api/services/workflow/qa/analysis.py:106">
P3: The new early-return branch for qa_grade_whole_call in run_per_node_qa_analysis isn't covered by an orchestration-level test. Existing tests only cover the regular per-node flow and the whole-call helper in isolation, so a future change that alters this branch's behavior or arguments, or accidentally falls through to node splitting, wouldn't be caught. Consider adding an async test that enables qa_grade_whole_call, mocks _run_whole_call_qa_analysis, and asserts its result is returned without splitting by node.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new early-return branch for qa_grade_whole_call in run_per_node_qa_analysis isn't covered by an orchestration-level test. Existing tests only cover the regular per-node flow and the whole-call helper in isolation, so a future change that alters this branch's behavior or arguments, or accidentally falls through to node splitting, wouldn't be caught. Consider adding an async test that enables qa_grade_whole_call, mocks _run_whole_call_qa_analysis, and asserts its result is returned without splitting by node.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At api/services/workflow/qa/analysis.py, line 106:

<comment>The new early-return branch for qa_grade_whole_call in run_per_node_qa_analysis isn't covered by an orchestration-level test. Existing tests only cover the regular per-node flow and the whole-call helper in isolation, so a future change that alters this branch's behavior or arguments, or accidentally falls through to node splitting, wouldn't be caught. Consider adding an async test that enables qa_grade_whole_call, mocks _run_whole_call_qa_analysis, and asserts its result is returned without splitting by node.</comment>

<file context>
@@ -99,6 +99,17 @@ async def run_per_node_qa_analysis(
+    # segment (one verdict) instead of node-by-node. Use when a workflow's scored
+    # content spans multiple nodes (e.g. gated questions in a later closing node),
+    # so per-node grading can't produce one authoritative result.
+    if getattr(qa_data, "qa_grade_whole_call", False):
+        logger.info(
+            f"Whole-call QA grading enabled for run {workflow_run_id}; "
</file context>

[skyfrostmage]

Added test_qa_whole_call_routing.py covering this branch — it enables qa_grade_whole_call, mocks _run_whole_call_qa_analysis, and asserts its result is returned while split_events_by_node is never called. A companion test asserts the default path still reaches node splitting. (65d1821)