Description
Follow-up from PR #6617 (fix for #6560). During that PR's review, impl-critic and the tester independently found that the dropped [stderr]-prefix test (formerly covered only via the removed legacy handle_tool_result harness) does not actually have an equivalent discriminating test on the production path.
The originally-proposed replacement, native_anomaly_stderr_output_records_error (R-AN-3, tafc_and_record_outcomes_tests.rs), asserts det.check().is_none() on 19 successes + 1 [stderr] call. AnomalyDetector's window/threshold setup (window=20, thresholds=0.5/0.7) means a single [stderr] call is a 1/20 = 0.05 ratio, which never crosses either threshold regardless of whether [stderr] classifies as AnomalyOutcome::Error or AnomalyOutcome::Success — the assertion passes either way. It cannot actually discriminate the branch under test.
The deleted legacy-harness test had the same problem for a different reason: it asserted on handle_tool_failure_outcomes's return value, which only checks for "[error]"/"[exit code" markers, never "[stderr]" — so it was also not actually testing the [stderr] branch it claimed to cover.
Impact
No regression from PR #6617 — this is a pre-existing gap that predates it (the "coverage" was illusory both before and after). But right now, no test in the codebase would catch a regression in the "[stderr]" branch of classify_tool_result (crates/zeph-core/src/agent/tool_execution/tool_result.rs:295).
Reproduction Steps
- Inspect
classify_tool_result at tool_result.rs:295 — confirm the "[stderr]" prefix branch exists and maps to AnomalyOutcome::Error.
- Inspect
native_anomaly_stderr_output_records_error (R-AN-3) in tafc_and_record_outcomes_tests.rs — confirm its AnomalyDetector window/threshold setup (window=20, thresholds=0.5/0.7) cannot distinguish a 1/20 [stderr] call's classification either way.
- Confirm no other test asserts directly on
classify_tool_result's return value for [stderr] input.
Expected Behavior
A test exists that directly asserts classify_tool_result("... [stderr] ...") (or an equivalent minimal-surface call) returns the expected classification (AnomalyOutcome::Error), independent of any downstream detector's window/threshold behavior.
Suggested Fix
Add a focused unit test calling classify_tool_result directly with [stderr]-prefixed and non-[stderr] inputs, asserting on the classification value itself rather than routing through AnomalyDetector's windowed aggregation (which is the wrong layer to assert this invariant at).
Environment
Description
Follow-up from PR #6617 (fix for #6560). During that PR's review, impl-critic and the tester independently found that the dropped
[stderr]-prefix test (formerly covered only via the removed legacyhandle_tool_resultharness) does not actually have an equivalent discriminating test on the production path.The originally-proposed replacement,
native_anomaly_stderr_output_records_error(R-AN-3,tafc_and_record_outcomes_tests.rs), assertsdet.check().is_none()on 19 successes + 1[stderr]call.AnomalyDetector's window/threshold setup (window=20, thresholds=0.5/0.7) means a single[stderr]call is a 1/20 = 0.05 ratio, which never crosses either threshold regardless of whether[stderr]classifies asAnomalyOutcome::ErrororAnomalyOutcome::Success— the assertion passes either way. It cannot actually discriminate the branch under test.The deleted legacy-harness test had the same problem for a different reason: it asserted on
handle_tool_failure_outcomes's return value, which only checks for"[error]"/"[exit code"markers, never"[stderr]"— so it was also not actually testing the[stderr]branch it claimed to cover.Impact
No regression from PR #6617 — this is a pre-existing gap that predates it (the "coverage" was illusory both before and after). But right now, no test in the codebase would catch a regression in the
"[stderr]"branch ofclassify_tool_result(crates/zeph-core/src/agent/tool_execution/tool_result.rs:295).Reproduction Steps
classify_tool_resultattool_result.rs:295— confirm the"[stderr]"prefix branch exists and maps toAnomalyOutcome::Error.native_anomaly_stderr_output_records_error(R-AN-3) intafc_and_record_outcomes_tests.rs— confirm itsAnomalyDetectorwindow/threshold setup (window=20, thresholds=0.5/0.7) cannot distinguish a 1/20[stderr]call's classification either way.classify_tool_result's return value for[stderr]input.Expected Behavior
A test exists that directly asserts
classify_tool_result("... [stderr] ...")(or an equivalent minimal-surface call) returns the expected classification (AnomalyOutcome::Error), independent of any downstream detector's window/threshold behavior.Suggested Fix
Add a focused unit test calling
classify_tool_resultdirectly with[stderr]-prefixed and non-[stderr]inputs, asserting on the classification value itself rather than routing throughAnomalyDetector's windowed aggregation (which is the wrong layer to assert this invariant at).Environment
crates/zeph-core/src/agent/tool_execution/tool_result.rs:295crates/zeph-core/src/agent/tool_execution/tests/tafc_and_record_outcomes_tests.rs(R-AN-3),crates/zeph-core/src/agent/tool_execution/tests/parallel_and_handle_tests.rs(comment documenting the gap)