[FEAT]: Add durable terminal evaluation contract - #146
Spencer Schoenberg (spencrr) wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
3cc7cc8 to
fb90c72
Compare
c3b5f80 to
9b7c6a6
Compare
9b7c6a6 to
ef98095
Compare
Spencer Schoenberg (spencrr)
changed the title
There was a problem hiding this comment.
🟡 Changes recommended
A huge terminal confidence value can raise an uncaught overflow and abort controller-side xdist merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds durable terminal-evaluation and population provenance across core results, reporting, and xdist transport.
Changes:
- Adds terminal evaluation, trace-end reason, evaluation-purpose, and resolver APIs.
- Centralizes population validation.
- Extends persistence, truncation handling, tests, and documentation.
File summaries
| File | Description |
|---|---|
rampart/core/_population.py |
Adds shared population validation. |
rampart/core/types.py |
Adds provenance enums and turn purpose. |
rampart/core/result.py |
Extends results and verdict resolution. |
rampart/core/execution.py |
Validates trial parameters early. |
rampart/core/__init__.py |
Exports new core APIs. |
rampart/pytest_plugin/_xdist.py |
Transports and validates provenance. |
rampart/reporting/json_file.py |
Serializes terminal metadata. |
rampart/probes/_single_turn.py |
Clarifies turn-limit behavior. |
rampart/probes/_factory.py |
Updates probe API documentation. |
rampart/attacks/_xpia.py |
Clarifies XPIA turn limits. |
rampart/attacks/_factory.py |
Updates XPIA API documentation. |
tests/unit/core/test_types.py |
Tests enums and turn validation. |
tests/unit/core/test_result.py |
Tests result contracts and resolvers. |
tests/unit/core/test_execution.py |
Tests early trial validation. |
tests/unit/pytest_plugin/test_xdist.py |
Tests transport and truncation. |
tests/unit/pytest_plugin/test_xdist_aggregation.py |
Tests worker/controller persistence. |
tests/unit/reporting/test_json_file.py |
Tests JSON provenance output. |
docs/api/core-types.md |
Documents new core APIs. |
docs/usage/results-and-reporting.md |
Explains result provenance. |
docs/usage/xdist.md |
Documents envelope behavior. |
docs/probes/behavioral.md |
Updates probe turn-limit semantics. |
docs/attacks/xpia.md |
Updates XPIA turn-limit semantics. |
Review details
- Files reviewed: 22/22 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ef98095 to
acbf019
Compare
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Extreme malformed integers can escape the xdist fail-closed error boundary.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
rampart/pytest_plugin/_xdist.py:998
- A sufficiently large integer overflows
float(), then its!rconversion can itself raiseValueErrorunder Python's integer-string digit limit. That exception escapes instead of becomingWorkerOutputError, so malformed worker data can abort the pytest hook rather than marking the run incomplete. Render the value through the existing safe string helper.
msg = f"Confidence could not be converted to float: {raw_confidence!r}."
- Files reviewed: 23/23 changed files
- Comments generated: 3
- Review effort level: Balanced
| try: | ||
| normalized_threshold = float(threshold) | ||
| except (OverflowError, ValueError) as exc: | ||
| msg = f"Expected finite number for population threshold, got {threshold!r}." |
|
|
||
| Raises: | ||
| TypeError: If ``value`` is not a string. | ||
| ValueError: If ``value`` is empty or exceeds the transport bound. |
| assert t.eval_result is er | ||
| assert t.eval_result is not None and t.eval_result.detected is True | ||
|
|
||
| def test_eval_purpose_round_trips(self): |
| Online evaluations attached to turns are available as | ||
| `result.turn_evaluations`. The older `result.eval_results` property remains a | ||
| compatibility view of the same turn-level list and intentionally excludes the | ||
| terminal evaluation. |
There was a problem hiding this comment.
Since we aren't following an explicit deprecation strategy right now - can we remove this eval_results property and go directly with turn_evaluations if it won't be used in the future?
| if not isinstance(value, str): | ||
| msg = "population id must be a string" | ||
| raise TypeError(msg) | ||
| if not value: |
There was a problem hiding this comment.
nit: Check for if not value before if not isinstance(value, str) since None will not be a string regardless :)
|
|
||
| Raises: | ||
| TypeError: If ``value`` is not a string. | ||
| ValueError: If ``value`` is empty or exceeds the transport bound. |
There was a problem hiding this comment.
nit: only returns ValueError is is empty - I don't see any logic around a transport bound?
| def eval_results(self) -> list[EvalResult]: | ||
| """Compatibility view of online evaluations attached to turns. | ||
|
|
||
| ``terminal_evaluation`` is intentionally excluded. New consumers |
There was a problem hiding this comment.
same note as L67 in results-and-reporting.md, we should just remove this now since we are ok with breaking changes & fix any of the call sites to call turn_evaluations directly :)
| ValueError: If ``evaluation.outcome`` is not a known EvalOutcome. | ||
| """ | ||
| outcome = _require_eval_outcome(evaluation=evaluation) | ||
| if outcome is EvalOutcome.DETECTED: |
There was a problem hiding this comment.
nit: stylistically i like not having many returns. i'd rewrite like
status = SafetyStatus.UNDETERMINED
if outcome is EvalOutcome.DETECTED:
status = SafetyStatus.UNSAFE
if outcome is EvalOutcome.NOT_DETECTED:
status = SafetyStatus.SAFE
return status
| Raises: | ||
| ValueError: If ``evaluation.outcome`` is not a known EvalOutcome. | ||
| """ | ||
| outcome = _require_eval_outcome(evaluation=evaluation) |
There was a problem hiding this comment.
same note about many returns here - see L352
| turn.turn_number # 0-indexed position | ||
| ``` | ||
|
|
||
| `terminal_evaluation` is the evaluator output for the terminal trace. It is an |
There was a problem hiding this comment.
Naming thoughts: Since this is the final-trace evaluator output vs turn-level evals...what do you think about naming this final_trace_evaluation? Terminal does not read to me as common of language as "trace" in our docs. It can even be final_evaluation or something without trace/terminal...but if we use trace then it would relate it to trace_end_reason while still adding clarity.
Description
Adds the per-execution provenance needed before final-trace verdict cadence changes.
Result.terminal_evaluationstores the evaluator output for the terminal trace,Result.trace_end_reasonrecords why trace production ended, andTurn.eval_purposeidentifies online stop checks.Result.turn_evaluationsmakes the online evidence boundary explicit whileeval_resultsremains a compatibility view.The direct attack and probe resolvers require one evaluation and reject unknown runtime outcomes instead of falling through. Population provenance now shares validation across
PopulationRef,PopulationResult, andexecute_trials_async, with invalid thresholds rejected before an execution factory runs.This PR also includes the persistence work previously split into #147 so the contract cannot land without transport support. The current xdist v2 envelope and JSON report carry terminal evaluation, trace-end reason, turn purpose, and population provenance together. Malformed worker data fails closed, including overflowing confidence values, and oversized results produce bounded incomplete markers while retaining population provenance when it fits.
Breaking changes
None for valid callers. Invalid population provenance and malformed evaluator outcomes now fail early instead of being accepted or falling through.
Checklist
pre-commit run --all-filespassesValidation: 1,122 unit tests pass. Strict documentation build and all pre-commit checks pass.