fix(rlm): fail closed on interpreter failures

[isaacbmiller]

1. Issue / repro

RLM currently treats a broken interpreter session as if the model merely wrote bad Python.

def execute(code, variables=None):
    raise CodeInterpreterError("protocol corrupt")

On main, RLM catches that failure, writes this into the trajectory, exhausts max_iters, and calls the extract LM:

[Error] protocol corrupt
RLM reached max iterations, using extract to get final output

The caller can receive an apparently valid answer from extraction even though the interpreter never produced a trustworthy result. This is the unsafe failure mode reported in stanfordnlp/dspy#9643; this PR does not claim to fix that issue's underlying response-ID mismatch.

2. Why this is the root cause

CodeInterpreterError currently represents two incompatible states:

• submitted Python failed, but the interpreter is healthy and the model can retry;
• the process or JSON-RPC protocol failed, so continuing the session is unsafe.

RLM catches the umbrella type:

except (CodeInterpreterError, SyntaxError) as error:
    return f"[Error] {error}"

That catch erases the only distinction that should decide whether the RLM loop may continue.

3. How we know the fix addresses the root cause

The new regression gives RLM a custom interpreter that raises CodeInterpreterError("protocol corrupt") for generated code and configures extraction to return "hallucinated".

On untouched main at 24ec85de4, the test fails with:

Failed: DID NOT RAISE <class 'CodeInterpreterError'>
RLM reached max iterations, using extract to get final output

This branch raises CodeInterpreterError immediately, so extraction is never reached.

The opposite boundary is tested through the real Deno/Pyodide interpreter: generated ZeroDivisionError, ValueError, and even a user-defined exception named CodeInterpreterError become CodeExecutionError. A subsequent 2 + 2 succeeds in the same session, proving that these failures remain recoverable.

4. Why this is the concise fix

The origin of the failure is already known at one boundary: PythonInterpreter.execute() receives either a correlated application-error response, or one of its existing process/protocol checks fails.

This PR expresses that fact as a subtype:

class CodeExecutionError(CodeInterpreterError):
    """Recoverable error raised by code running in a healthy interpreter."""

PythonInterpreter changes one generated-runtime-error raise site, and RLM narrows one catch:

except (CodeExecutionError, SyntaxError) as error:
    return f"[Error] {error}"

There is no message parsing, error-name heuristic, retry branch, runner rewrite, or replacement fallback.

5. Context needed to validate the change

PythonInterpreter already raises bare CodeInterpreterError before the generated-code error branch for:

• process startup/death and missing responses;
• health-check and setup RPC failures;
• response-ID mismatches;
• malformed or unexpected JSON-RPC frames.

Known application error codes returned for the matching execute request are submitted-code failures. SyntaxError remains its existing dedicated exception.

The max-iteration extraction fallback is still strategic and remains unchanged. It runs only after a healthy interpreter session produces recoverable iterations without a valid SUBMIT.

6. What the fix does in the code

• Adds and publicly exports CodeExecutionError.
• Makes it a subtype of CodeInterpreterError.
• Raises it for correlated Python/tool execution errors from PythonInterpreter.
• Keeps process/protocol errors as bare CodeInterpreterError.
• Makes sync and async RLM paths recover only from CodeExecutionError and SyntaxError.
• Documents that forward/aforward propagate interpreter failures.

Compatibility boundaries and downsides

• Intentional RLM behavior change: fatal interpreter failures now raise immediately instead of becoming trajectory text followed by extraction.
• Custom interpreters: implementations must raise CodeExecutionError for recoverable submitted-code failures. A bare CodeInterpreterError now tells RLM that the session is unsafe. This explicit contract replaces an ambiguity; there is no reliable generic fallback for old implementations.
• Existing broad handlers remain compatible: because CodeExecutionError subclasses CodeInterpreterError, existing except CodeInterpreterError code still catches both categories.
• More-specific type observation: direct PythonInterpreter callers may now observe CodeExecutionError where they previously observed the base class.
• No runner/protocol change: runner.js, JSON-RPC shapes, process restart behavior, tools, and provider adapters are untouched.
• No fallback removal: recoverable generated-code retries and healthy-session max-iteration extraction are unchanged.

Validation

• uv run --frozen pytest -q tests/predict/test_rlm.py --deno — 110 passed, 2 skipped
• uv run --frozen pytest -q tests/primitives/test_python_interpreter.py --deno — 54 passed
• focused boundary tests — 9 passed
• public import/subclass smoke test — passed
• repository pre-commit hooks on all six changed files — passed
• new fail-closed regression against untouched main — fails because no exception is raised; passes here