feat(history): add typed history frames

[isaacbmiller]

Summary

• add typed HistoryFrame and Observation models while preserving legacy messages= construction
• add append helpers, open-episode detection, and pluggable compaction hooks
• export the new history frame types

Stack

• Base branch: isaac/react-v2-base-normalized-adapter (62adc7d34)
• Next PR: feat(adapter): render typed history frames #52

Validation

• uv run --extra dev pytest -q tests/adapters/test_history.py
• Full stack focused validation: uv run --extra dev pytest -q tests/adapters/test_history.py tests/adapters/test_history_formatting.py tests/adapters/test_history_lm_messages.py tests/adapters/test_tool.py tests/predict/test_reactv2.py tests/adapters/test_chat_adapter.py::test_chat_adapter_with_tool tests/adapters/test_chat_adapter.py::test_tool_call_with_null_content_does_not_raise

[greptile-apps]

Greptile Summary

This PR introduces a normalized LM type system (LMMessage, LMPart subtypes, LMRequest/LMResponse in core/types.py), a provider-translation layer (clients/openai_format.py), and a refactored adapter pipeline (plan_fields → render_request → call_lm → parse_response). It also replaces the flat History.messages list with typed HistoryFrame/Observation frames, adds append_inputs/append_outputs/append_observation helpers, and ships a pluggable compact_fn hook with a reference truncate_oldest_actions compaction strategy.

• Normalized adapter pipeline: base.py is refactored into four discrete stages; History, Image, Audio, and File fields are extracted in plan_fields before format() sees the signature, cleanly replacing the old _call_preprocess / _call_postprocess pattern.
• Typed history frames: HistoryFrame separates inputs, outputs, and observations into distinct frame objects; a messages property preserves backward compatibility with the old dict-based API; legacy constructors are accepted via a model_validator.
• Broad deprecation sweep: All legacy Type hooks (format, serialize_model, adapt_to_native_lm_feature, parse_lm_response, parse_stream_chunk) are marked deprecated with DeprecationWarning targeting removal in DSPy 3.5.

Confidence Score: 3/5

The new HistoryFrame-based append API and the _history_to_lm_messages renderer are structurally misaligned, causing incorrect message layout for any history built with the new helpers.

The core compatibility issue between HistoryFrame objects and the dict-based adapter rendering methods remains unresolved (noted in prior review threads). Additionally, _history_to_lm_messages generates a user+assistant pair for every frame, but append_inputs / append_outputs create separate single-purpose frames, so the renderer produces doubled, partially empty message pairs for any history built with the new API. Both defects affect the primary new feature surface. The large new files (core/types.py, clients/openai_format.py) and the adapter pipeline refactor are otherwise well-structured and appear safe.

dspy/adapters/base.py (_history_to_lm_messages) and dspy/adapters/types/history.py (frame-level rendering contract)

Important Files Changed

Filename	Overview
dspy/adapters/types/history.py	Introduces HistoryFrame/Observation/History, append helpers, and compaction hooks. The new frame-based API creates separate input and output frames that are structurally incompatible with how _history_to_lm_messages renders them (one user+assistant pair per frame regardless of frame type).
dspy/adapters/base.py	Major refactor to plan_fields / render_request / call_lm / parse_response pipeline. The new _history_to_lm_messages helper has a structural mismatch with the HistoryFrame API (per-frame user+assistant pairs instead of per-turn pairs).
dspy/clients/openai_format.py	New 791-line module that translates between DSPy normalized LM types and OpenAI-shaped JSON for chat, responses, and text-completion endpoints. Pure data mapping; no side effects.
dspy/core/types.py	New 1992-line normalized type system (LMMessage, LMPart subtypes, LMRequest/Response/Output, LMConfig) for provider-independent LM interaction. Types are well-structured with discriminated unions and validators.
dspy/adapters/json_adapter.py	JSONAdapter now calls plan_fields once to extract prompt_signature for structured-output schema building, then calls super().call which calls plan_fields again internally — double planning is safe but redundant.
dspy/adapters/types/base_type.py	Legacy Type.format/serialize_model/adapt_to_native_lm_feature/parse_lm_response/parse_stream_chunk all get DeprecationWarning wrappers targeting DSPy 3.5. Clean backward-compatible deprecation path.
tests/adapters/test_history.py	New focused test suite covering History construction, append helpers, has_open_episode, compaction, and round-trip model_validate. No tests exercise HistoryFrame rendering through an adapter.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Adapter
    participant PlanFields as plan_fields
    participant HistoryHelper as _history_to_lm_messages
    participant Format as format()
    participant ApplyPlan as _apply_planned_messages
    participant LM

    Caller->>Adapter: __call__(lm, lm_kwargs, signature, inputs)
    Adapter->>PlanFields: plan_fields(lm, lm_kwargs, signature, inputs)
    PlanFields->>PlanFields: detect History/Image/Audio/File fields
    PlanFields->>HistoryHelper: _history_to_lm_messages(signature, history)
    HistoryHelper-->>PlanFields: list[LMMessage] (history turns)
    PlanFields->>PlanFields: remove history/media from signature and inputs
    PlanFields-->>Adapter: plan dict (prompt_signature, messages, user_parts, tools, lm_kwargs)

    Adapter->>Format: format(prompt_signature, demos, plan_inputs)
    Format-->>Adapter: list[LMMessage] (system + demos + current user)

    Adapter->>ApplyPlan: _apply_planned_messages(messages, plan)
    ApplyPlan->>ApplyPlan: insert history messages before last user msg
    ApplyPlan->>ApplyPlan: extend last user msg parts with media parts
    ApplyPlan-->>Adapter: list[LMMessage] (complete context)

    Adapter->>LM: call_lm(lm, LMRequest)
    LM-->>Adapter: LMResponse

    Adapter->>Adapter: parse_response(plan, response, lm)
    Adapter-->>Caller: list[dict] (parsed output fields)

Loading

Comments Outside Diff (1)

1. dspy/adapters/base.py, line 210-217 (link)

   Per-frame user+assistant pairs produce malformed message structure for the new HistoryFrame API

   _history_to_lm_messages emits one user and one assistant message for every entry in history.frames. That pattern matches the legacy dict format, where each dict contained both input and output fields so a single entry mapped to a full exchange.

   The new append_inputs / append_outputs API creates separate frames — an input-only HistoryFrame followed by an output-only HistoryFrame. When iterated here each frame generates its own user+assistant pair: the input frame produces a valid user message but an empty assistant message; the output frame produces an empty user message but a valid assistant message. The net result is four messages — two of which are empty — instead of the correct two-message exchange.

   Even after fixing the HistoryFrame-vs-dict compatibility noted elsewhere, this structural mismatch would persist: the renderer needs to be frame-type-aware, emitting only a user message for input frames and only an assistant message for output frames, then pairing them into exchanges rather than treating each frame as a self-contained turn.

_{Reviews (4): Last reviewed commit: "feat(history): add typed history frames" | Re-trigger Greptile}