feat(dspy): add interpreter-level callbacks (on_interpreter_*)

[dbreunig]

📝 Changes Description

Adds first-class BaseCallback hooks for the CodeInterpreter layer so that dspy.RLM — and any other code-executing module — is observable the same way Module, LM, Tool, Adapter, and Evaluate already are.

Motivation. The callback system had zero hooks for the interpreter layer, leaving gaps for RLM: a cell's output was only visible one turn late (embedded in the next iteration's repl_history); terminal states (FinalOutput, CodeInterpreterError, SyntaxError) were flattened into "[Error] ..." strings, losing the real exception class and the SUBMIT payload; RLM's injected llm_query/llm_query_batched are plain closures, so the sandbox→host dispatch itself was invisible; and interpreter startup/shutdown (Deno process spawn) was unobservable.

Changes.

• dspy/utils/callback.py — Eight new no-op BaseCallback handlers: on_interpreter_execute_{start,end}, on_interpreter_tool_call_{start,end}, on_interpreter_startup_{start,end}, on_interpreter_shutdown_{start,end}. Routed via a structural isinstance(instance, dspy.CodeInterpreter) branch that dispatches by method name (execute / start / shutdown / _invoke_tool), following the Adapter precedent. The branch sits before the on_module_* fallback so interpreter events never masquerade as module events.
• dspy/primitives/python_interpreter.py — Decorate execute/start/shutdown with @with_callbacks; add an optional callbacks param for instance-level parity with LM/Module. Extract the core tool invocation into a new _invoke_tool seam and decorate that — _handle_tool_call keeps its try/except around it, so the real exception reaches on_interpreter_tool_call_end and is then converted to the same JSON-RPC error as before. Sandbox behavior is byte-for-byte unchanged.
• dspy/primitives/code_interpreter.py — Document that implementations should decorate these methods to participate (the Protocol can't enforce it).
• tests/mock_interpreter.py — Decorate the test double so it participates like a real interpreter.
• Docs — Add the handlers to the observability guide's BaseCallback table plus an RLM-flavored example (per-cell timing + sub-tool logging).

Test coverage. New tests/callback/test_interpreter_callback.py (recording-callback pattern) covers: execute start/end with code in inputs and result in outputs; the SUBMIT FinalOutput path; error surfacing (CodeInterpreterError in exception, still propagates); plain-closure tool dispatch with correct tool_name/kwargs and JSON-RPC error preservation for raising/unknown tools; call_id nesting of tool calls under the enclosing execute; lifecycle with idempotent shutdown; dispatch-routing integrity (no misrouting to on_module_*); the zero-callback fast path; and a full dspy.RLM run emitting one execute event per iteration. Deno-gated tests (@pytest.mark.deno, existing skip convention) exercise the real PythonInterpreter.

Local runs of tests/callback, tests/primitives, and tests/predict/test_rlm.py pass (Deno tests skip without --deno).

No behavior change without registered callbacks. with_callbacks returns immediately when the combined callback list is empty, so the empty-callback fast path is preserved and there is no behavior change when no callbacks are registered.

✅ Contributor Checklist

• Pre-Commit checks are passing locally (ruff check clean on all changed files)
• Title of the PR corresponds to the required {label}(dspy): {message} format
• Commit messages follow the required {label}(dspy): {message} format (commits use plain imperative summaries; can be reworded/squashed before any upstream submission)

⚠️ Warnings

• AI-assisted. Implemented with AI assistance (Claude Code); all changes reviewed and understood.
• This is an internal review PR on the fork. Before any submission to stanfordnlp/dspy, note their CONTRIBUTING policy: open an issue first for non-trivial features and disclose AI assistance (draft ISSUE.md/PR_DESCRIPTION.md prepared separately). The upstream issue/PR should be opened by a human in their own words.
• Deno integration tests can't run in every environment; they follow the existing @pytest.mark.deno skip convention and were not executed here (Deno unavailable).

🤖 Generated with Claude Code

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81e21f6f28

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Route lazy process creation through startup callbacks

When PythonInterpreter is used normally by RLM, start() is never called: execute() creates the Deno process directly through _ensure_deno_process(), and the default RLM context only calls shutdown(). Consequently, on_interpreter_startup_* is not emitted for the actual lazy process spawn, so these new lifecycle hooks only work for users who explicitly pre-warm the interpreter. Route lazy initialization through the decorated startup path or instrument the process-creation seam itself.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Avoid hard-coding the custom tool-dispatch seam name

For a custom CodeInterpreter that follows the new documentation and decorates a tool-dispatch seam named something conventional such as dispatch_tool, handler selection raises an unsupported-method ValueError, which with_callbacks catches and logs, so no tool-call events are emitted. Only the private PythonInterpreter-specific name _invoke_tool is recognized even though custom implementations are told they can decorate their own seam; provide a way to declare the event type or require and document a protocol-level method name.

Useful? React with 👍 / 👎.

[greptile-apps]

Greptile Summary

This PR adds interpreter-level callbacks for code execution observability. The main changes are:

• New BaseCallback hooks for interpreter execute, tool call, startup, and shutdown events.
• Callback routing for CodeInterpreter methods before the module fallback.
• PythonInterpreter support for instance callbacks and decorated execution, process spawn, shutdown, and tool-dispatch seams.
• Updated mock interpreter support for callback-aware tests.
• New callback tests covering execution results, errors, lifecycle events, tool dispatch, nesting, and RLM integration.
• Observability docs for the new hooks and an RLM logging example.

Confidence Score: 5/5

This PR is safe to merge with low risk.

The changes are additive and preserve the zero-callback fast path. Callback dispatch order keeps interpreter events from falling through to module handlers. No blocking correctness or security issues were found in the changed paths.

No files require special attention.

T-Rex Logs

What T-Rex did

• The interpreter-callback pytest run was performed to exercise the focused callback path and captured the command used, the current working directory, any warnings emitted, the pass/skip summary, and the exit code.
• The RLM pytest run was performed to validate the runtime library manager tests and captured the command, working directory, pass/skip summary, and exit code.
• The ruff changed-files check was run to verify code style on changed files and recorded the command, working directory, the success output, and the exit code.
• The Deno availability check was performed to determine how Deno-gated skips should be interpreted, and its unavailability status was captured.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
docs/docs/tutorials/observability/index.md	Documents the new interpreter callback hooks and adds an RLM-oriented observability example.
dspy/primitives/code_interpreter.py	Expands the CodeInterpreter protocol documentation with callback participation guidance for lifecycle, execution, and tool-dispatch methods.
dspy/primitives/python_interpreter.py	Adds instance callbacks and decorated seams for interpreter execution, process startup, shutdown, and sandbox tool dispatch without changing JSON-RPC behavior.
dspy/utils/callback.py	Adds BaseCallback interpreter handlers and routes structurally identified CodeInterpreter methods before the module fallback.
tests/callback/test_interpreter_callback.py	Adds broad unit and Deno-gated coverage for interpreter callback routing, nesting, lifecycle events, errors, and RLM integration.
tests/mock_interpreter.py	Updates the mock interpreter to support instance callbacks and emit the new interpreter lifecycle and execution callback events.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant User
participant RLM as dspy.RLM / caller
participant PI as PythonInterpreter
participant CB as BaseCallback handlers
participant Sandbox as Deno/Pyodide sandbox
participant Tool as Host-side tool

User->>RLM: forward(...)
RLM->>PI: execute(code, variables)
PI->>CB: on_interpreter_execute_start
PI->>PI: _ensure_deno_process()
PI->>CB: on_interpreter_startup_start/end (on spawn)
PI->>Sandbox: JSON-RPC execute
Sandbox-->>PI: tool_call(name, kwargs)
PI->>CB: on_interpreter_tool_call_start
PI->>Tool: invoke_tool(name, kwargs)
Tool-->>PI: result or exception
PI->>CB: on_interpreter_tool_call_end
PI-->>Sandbox: JSON-RPC result/error
Sandbox-->>PI: output / final / error
PI->>CB: on_interpreter_execute_end
PI-->>RLM: output or FinalOutput

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant User
participant RLM as dspy.RLM / caller
participant PI as PythonInterpreter
participant CB as BaseCallback handlers
participant Sandbox as Deno/Pyodide sandbox
participant Tool as Host-side tool

User->>RLM: forward(...)
RLM->>PI: execute(code, variables)
PI->>CB: on_interpreter_execute_start
PI->>PI: _ensure_deno_process()
PI->>CB: on_interpreter_startup_start/end (on spawn)
PI->>Sandbox: JSON-RPC execute
Sandbox-->>PI: tool_call(name, kwargs)
PI->>CB: on_interpreter_tool_call_start
PI->>Tool: invoke_tool(name, kwargs)
Tool-->>PI: result or exception
PI->>CB: on_interpreter_tool_call_end
PI-->>Sandbox: JSON-RPC result/error
Sandbox-->>PI: output / final / error
PI->>CB: on_interpreter_execute_end
PI-->>RLM: output or FinalOutput

Loading

_{Reviews (2): Last reviewed commit: "Address review: cover lazy startup and m..." | Re-trigger Greptile}

[dbreunig]

Addressed both automated-review findings in 2124394.

1. Lazy startup was invisible (Greptile P1 / Codex). Startup callbacks now fire from the actual process-spawn seam instead of the public start(). execute() spawns Deno lazily via _ensure_deno_process(), so decorating start() only covered explicit pre-warming — the default RLM flow never calls start(), so it never emitted a startup event. Startup emission moved to a decorated _spawn_process() and start() is left undecorated, so on_interpreter_startup_* now fires exactly once per real spawn on both the lazy and explicit paths, and not when the process is already running. Added non-Deno tests for both paths plus a Deno test asserting the lazy spawn nests under the enclosing execute().

2. Renamed the tool-dispatch seam _invoke_tool → invoke_tool (public). Rationale: callback routing dispatches by method name (mirroring the Adapter format/parse precedent), so a custom CodeInterpreter that wants tool-call events must decorate an identically named method. Since that name is effectively a documented participation contract for external implementers — not just an internal helper — asking them to adopt an underscore-private name (_invoke_tool) was inconsistent; the execute/start/shutdown names they must match are all public. Making it invoke_tool keeps the whole recognized-seam set public and coherent, with no new dispatch machinery (which DSPy has no precedent for). The rename is fully local — the method is new in this PR, isn't part of the CodeInterpreter Protocol, has no external callers, and no name collision. The CodeInterpreter docstring now spells out the recognized seam names (execute/start/shutdown/invoke_tool).

CI was green before these changes; re-running now.

---

Generated by Claude Code