feat(rlm): multimodal media, budget controls, multi-model routing, LocalInterpreter, depth>1, GEPA resilience

[rawwerks]

Making RLM production-ready: multimodal media, budget controls, multi-model routing, LocalInterpreter, depth>1, and GEPA compatibility

Closes #9289

Summary

Seven additive features that came from using RLM + GEPA on real multimodal tasks. Each addresses a failure mode hit in practice. All changes are backwards-compatible — default behavior is unchanged.

+3,115 lines across 10 files. 206 tests pass (0 failures, 38 deno-skips). Rebased on current main.

The seven changes

1. Multimodal media support via the dspy.Type sandbox protocol

Problem: RLM can only work with text. Audio/Image inputs are invisible to the agent.

Solution: Implements the to_sandbox() / rlm_preview() / sandbox_setup() / sandbox_assignment() protocol (from #9283) on Audio and Image. Any dspy.Type implementing this protocol is automatically:

• Detected as a multimodal input field (_detect_multimodal_fields)
• Injected into the sandbox via to_sandbox() (descriptor string for media)
• Previewed via rlm_preview() in the agent's variable context
• Available for llm_query_with_media(prompt, *media_var_names) which sends the actual content parts to the sub-LLM

Also adds _wrap_rlm_inputs() to auto-wrap raw values into their annotated dspy.Type, and _inject_pending_vars() for unified sandbox injection — both patterns from #9283.

Documents the protocol on the Type base class so future type authors know what to implement for RLM support.

class DescribeImage(dspy.Signature):
    image: dspy.Image = dspy.InputField()
    description: str = dspy.OutputField()

rlm = dspy.RLM(DescribeImage)
result = rlm(image=dspy.Image("photo.jpg"))
# Agent code: text = llm_query_with_media("Describe this", "image")

2. Multi-model sub-call routing via sub_lms

Problem: Some tasks benefit from cheap exploration + strong verification. RLM only supports a single sub_lm.

Solution: sub_lms={"flash": lm_fast, "pro": lm_pro} — sandbox code selects with llm_query(prompt, model="flash"). Falls back to sub_lm → dspy.settings.lm.

3. LocalInterpreter — unsandboxed host-process execution

Problem: Deno/Pyodide sandbox can't access host packages (PIL, numpy, soundfile, etc.).

Solution: New LocalInterpreter (175 lines) implements CodeInterpreter via exec(). State persists, tools injected, SUBMIT() identical. Intentionally unsandboxed for local experiments.

from dspy.primitives.local_interpreter import LocalInterpreter
rlm = dspy.RLM(MySignature, interpreter=LocalInterpreter())

4. Budget awareness — budget(), max_time, max_cost

Problem: Agents burn through iterations blindly. A single runaway example can blow your API budget in optimization loops.

Solution:

• max_time=120 — wall-clock limit, triggers extract fallback (not crash)
• max_cost=0.50 — dollar limit via litellm cost + BYOK upstream_inference_cost
• budget() — sandbox tool showing remaining resources with low-resource warnings

5. GEPA bootstrap_trace resilience

Problem: RLM timeouts/cost overruns raise exceptions that bootstrap_trace_data doesn't catch, losing all trace data.

Solution: Broad except Exception handler preserves partial trace as FailedPrediction (+7 lines in bootstrap_trace.py). KeyboardInterrupt is NOT caught.

6. Depth > 1 — recursive subcalls

Problem: llm_query() makes plain LM completions — no tools, no iteration. Sub-LLMs miss information in long contexts ("lost in the middle").

Solution: max_depth=2 makes llm_query() spawn a child RLM with its own REPL. Child can write code, call its own llm_query(), and iterate before SUBMITting.

rlm = dspy.RLM(
    MySignature,
    interpreter=LocalInterpreter(),
    max_depth=2,          # child RLMs get own REPLs
    max_iterations=15,    # per-level
    max_llm_calls=30,     # per-level (independent counters)
    max_time=300,         # remaining propagated to children
    max_cost=2.00,        # remaining propagated to children
)

Budget propagation: children get remaining time/cost, own iteration/call counters. Interpreter type matches parent (LocalInterpreter→LocalInterpreter, PythonInterpreter→PythonInterpreter). Tested E2E: 5/5 on LongMemEval_S with Gemini 3 Flash. Adapted from vanilla RLM depth>1.

7. Type protocol documentation on base class

Adds protocol comments to dspy.Type base class documenting the four methods for RLM sandbox opt-in, matching the pattern established by #9283.

File-by-file

File	Lines	What
dspy/adapters/types/audio.py	+21	rlm_preview, to_sandbox, sandbox_setup, sandbox_assignment
dspy/adapters/types/image.py	+25	Same protocol methods
dspy/adapters/types/base_type.py	+10	Protocol documentation
dspy/predict/rlm.py	+567	All features: media protocol, sub_lms, budget, cost, depth>1, _subcall, _wrap_rlm_inputs
dspy/primitives/local_interpreter.py	+175	New: unsandboxed exec()-based interpreter
dspy/primitives/python_interpreter.py	+76/-40	Generic _has_rlm_support + to_sandbox injection
dspy/teleprompt/bootstrap_trace.py	+7	GEPA exception handler
tests/predict/test_rlm.py	+1,792	All feature tests + depth>1 tests
tests/primitives/test_local_interpreter.py	+312	LocalInterpreter unit tests
tests/primitives/test_media_serialization.py	+170	Type protocol tests (Audio, Image, custom types)

How they compose

rlm = dspy.RLM(
    MyMultimodalSignature,
    sub_lms={"pro": lm_pro},        # multi-model routing
    max_time=120, max_cost=0.50,     # budget controls
    max_llm_calls=25,
    max_depth=2,                     # recursive subcalls
    interpreter=LocalInterpreter(),  # full Python access
)

# GEPA can optimize this without crashing on budget-exceeded examples
optimizer = dspy.GEPA(metric=my_metric, max_rounds=5)
optimized = optimizer.compile(rlm, trainset=examples)

Test summary

206 passed, 0 failures, 38 skipped (skips are pre-existing Deno-dependent tests)

Test Area	Count
Multimodal detection & registry (types protocol)	8
llm_query_with_media validation	5
Media instructions	2
Media content construction (LM calls)	4
Sub_lms routing	8
Budget/cost tracking	12
LocalInterpreter	33
Type protocol (Audio, Image, custom)	21
GEPA resilience	3
Depth > 1 (init, propagation, E2E)	30
_wrap_rlm_inputs / auto-wrapping	2

Relationship to #9283

The multimodal media support adopts the same dspy.Type sandbox protocol that #9283 introduces for DataFrame. We implemented to_sandbox() / rlm_preview() / sandbox_setup() / sandbox_assignment() on Audio and Image, and use the same generic _has_rlm_support() detection in PythonInterpreter. This means any future dspy.Type with the protocol works automatically with RLM — no RLM-specific code needed.

Implementation notes

• All changes additive — no breaking changes to existing behavior
• Default values preserve current behavior (max_time=None, max_cost=None, sub_lms={}, max_depth=1, default PythonInterpreter)
• Generic type detection: uses protocol checks (hasattr(annotation, 'to_sandbox')) not hardcoded type lists
• All files pass ruff check

[okhat]

The hero we needed, not the hero we deserved; thanks @rawwerks !!

[rawwerks]

@isaacbmiller - i think 1 & 7 probably go first (and together) since it looks like @kmad 's types are now in main. that seems like the next logical increment. do you agree?

[isaacbmiller]

Thanks for the PR! Going through these changes one by one:

 1. Multimodal media support via the dspy.Type sandbox protocol
Problem: RLM can only work with text. Audio/Image inputs are invisible to the agent.

Awesome!
I want to combine (1) and (7) on this PR with @kmad’s (Note #9283 is not yet merged), as these changes are connected. Will follow up on that PR.

2. Multi-model sub-call routing via sub_lm

I understand the desire to give the model flexibility in choosing a model. I don’t know if this change is currently worth the potential confusion that it will cause the model.

Do you have current usecases where inside of a single run you want to use multiple models?

3. LocalInterpreter — unsandboxed host-process execution

Great! Can we add an extra flag to verify that the user knows they are doing something dangerous (see BaseModule.load for how we force the user to pass an extra flag)

Separately, you can install packages from pip inside of pyodide. Is there a package you need that you are unable to install inside of pyodide?

4. Budget awareness — budget(), max_time, max_cost

These are great!
You could make the argument that max_time and budget should both be at a different level of abstraction in DPSy, but until we get there, this works.

If we do hit max_time or budget, should we throw or should we extract? I would lean towards throw, as if we had enough information to extract, the model would have hopefully called SUBMIT.

5. GEPA bootstrap_trace resilience
   Problem: RLM timeouts/cost overruns raise exceptions that bootstrap_trace_data doesn't catch, losing all trace data.

I also do not fully understand this problem. Is it just that after doing (4), that breaks GEPA somehow? Can you explain more?

6. Depth > 1 — recursive subcalls

I am very excited about this change too! I am hesitant on having the signature only be “prompt → response”?

IMO the stronger types via signatures is one of the main benefits of the DSPy version of RLMs.

7. Type protocol documentation on base class
   See (1)

TLDR:

• Will combine (1) and (7) in RLM Generic Type Support + Dataframe implementation #9283.
• 3 and 4 are great changes. Let’s break them out as separate PRs and make sure to add the flag for security on local interpreter.
• Need more information on (2). I am hesitant about adding this in currently for the sake of confusing the model with too much information.
• (5) I need more info
• (6) is very cool! lets discuss how to make it a fully fledged sub_rlm implementation with signatures.