fix(dspy): preserve typed dict/map value schemas in enforce_required

[detail-app]

Warning

GitHub issue creation failed

Detail attempted to publish this bug to GitHub, but the issue could not be created. This fix PR was created without that issue, and missing tracker references are shown as Unknown issue.

You can review and merge this PR normally. Please review your tracker integration settings before the next publish run.

Detail bug report: View on Detail

📝 Changes Description

This MR/PR contains the following changes:

• Bug: On the OpenAI Structured Outputs ("strict") path, JSONAdapter._get_structured_outputs_response_format post-processes the pydantic JSON schema with an inner enforce_required normalizer (dspy/adapters/json_adapter.py). Its no-properties branch — written as a defensive fallback — unconditionally rewrote any type: object schema lacking a properties key to {"properties": {}, "required": [], "additionalProperties": false}. Pydantic emits typed dict/map fields (dict[str, int], dict[str, dict[str, int]], etc.) as exactly such an object — {"type": "object", "additionalProperties": <value-schema>} with no properties — so the typed additionalProperties was destroyed for any dict field that reached the strict path: nested in a container (list[dict[str, int]]) or as a sub-field of a pydantic BaseModel output (reached via $defs recursion). The top-level guard _has_open_ended_mapping only intercepts top-level dict[K, V], so nested typed dicts silently degraded on-wire to a closed empty object, and JSONAdapter.parse then accepted []/[{}, …] with the declared contents erased — silent data loss with no exception or warning. Introduced in b8d9092 (Fix JSON Adapter's first attempt, all Adapters for ReAct trajectories stanfordnlp/dspy#8051).
• Fix: Made enforce_required's no-properties branch mirror OpenAI's own strict-mode converter (openai/lib/_pydantic.py:49-51, which only sets additionalProperties: false when the key is absent):
  • preserve and recurse into a typed additionalProperties (the dict value schema),
  • collapse an open-ended additionalProperties: true (dict[str, Any]) to false,
  • only set additionalProperties: false when absent; add a vacuous required: [].
  • Open-ended nested dicts keep their prior closed-empty-object semantics (only the vacuous properties: {} key is dropped on the wire — semantically identical). Fixed-property objects and the top-level dict[K, V] routing guard are untouched.
• Closes Unknown issue

✅ Contributor Checklist

• Pre-Commit checks are passing (locally and remotely) — uv run ruff check clean on both changed files
• Title of your PR / MR corresponds to the required format — fix(dspy): preserve typed dict/map value schemas in enforce_required
• Commit message follows required format {label}(dspy): {message} — fix(dspy): preserve typed dict/map value schemas in enforce_required

⚠️ Warnings

Testing summary. Verified via unit tests, mocked end-to-end routing, the OpenAI strict-converter transform, and the parse path:

• New regression tests (tests/adapters/test_json_adapter.py) assert, at the schema level, that list[dict[str, int]] preserves items.additionalProperties: {"type": "integer"}; that dict[str, dict[str, int]] recursion preserves the inner typed map; that a dict[str, int] sub-field of a pydantic BaseModel output is preserved via $defs; and that open-ended list[dict[str, Any]] still collapses to the closed-empty-object (additionalProperties: false, no properties). Inverse regressions verify fixed-property objects (required = [all keys], additionalProperties: false), scalar arrays, and top-level dict[str, int] still routing to json_object.
• End-to-end mocked-LM tests confirm list[dict[str, int]] stays on the strict pydantic-model path (not the json_object fallback), and a populated emission {"metadata": [{"score": 3, "rank": 1}, {"score": 9}]} now parses back to the declared typed dicts (no silent data loss).
• On-wire test runs openai.lib._pydantic.to_strict_json_schema (the local transform litellm applies before sending to the OpenAI strict API) and asserts the typed additionalProperties survives — i.e. the API receives the typed map, not the closed empty object.
• Routine checks: ruff check passes on both files; pytest tests/adapters/ tests/predict/ → 540 passed, 63 skipped; the broader sweep including tests/evaluate/test_evaluate.py (which uses an entities: list[dict[str, str]] signature) → 552 passed, 88 skipped; CI-parity invocation with -n auto --dist worksteal -m 'not extra and not deno' → 540 passed, 2 skipped. Existing LMError-propagation tests still pass, confirming provider errors still re-raise without a JSON-mode retry.
• Not verified (no live key in this environment): a live OpenAI Chat Completions round-trip confirming the model now emits populated [{...}, {...}] entries against the corrected on-wire schema. Attempted the smoke with no key (LMServerError: Missing credentials) and a dummy key (LMAuthError: Incorrect API key); both blocked on credentials, not on the fix. Everything the live call would prove beyond the committed tests is the model's literal emission cardinality, which is the only predicted-not-observed gap.

AI disclosure. Authored by Detail (automatic bug-fix tool). The fix and tests were generated and verified locally; no live API key was available. A maintainer with an OpenAI key can run the smoke snippet from the issue to confirm the populated-emission consequence end-to-end.

---

Automatic Fixes PRs can be configured here.

[greptile-apps]

Greptile Summary

This PR corrects strict structured-output schema normalization so nested typed dictionaries retain their declared value schemas instead of becoming closed empty objects.

• Preserves and recursively normalizes object-valued additionalProperties.
• Continues closing open-ended mappings and fixed-property objects.
• Adds schema-level, routing, parsing, nested-model, and downstream-converter regression coverage.

Confidence Score: 5/5

The PR appears safe to merge, with the typed-map schema fix covered across normalization, routing, conversion, and parsing paths.

No actionable failures remain; the changed branch preserves typed map schemas while retaining the prior behavior for open-ended and fixed-property objects.

Important Files Changed

Filename	Overview
dspy/adapters/json_adapter.py	Preserves typed map value schemas during recursive strict-schema normalization while retaining existing closure behavior for open-ended objects.
tests/adapters/test_json_adapter.py	Adds comprehensive regression coverage for nested typed maps, open-ended maps, fixed objects, routing, parsing, and downstream strict conversion.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Pydantic output schema] --> B{Object has properties?}
    B -->|Yes| C[Require every property]
    C --> D[Set additionalProperties false]
    B -->|No| E{additionalProperties value}
    E -->|Typed schema| F[Preserve and recursively normalize value schema]
    E -->|true or absent| G[Set additionalProperties false]
    E -->|false| H[Keep closed map]
    F --> I[Add required empty list]
    G --> I
    H --> I
    I --> J[Strict structured-output response format]

Loading

_{Reviews (1): Last reviewed commit: "fix(dspy): preserve typed dict/map value..." | Re-trigger Greptile}