Skip to content

fix(mind): support json-object-only providers#128

Merged
keli-wen merged 6 commits into
masterfrom
codex/issue-126-json-object
Jul 22, 2026
Merged

fix(mind): support json-object-only providers#128
keli-wen merged 6 commits into
masterfrom
codex/issue-126-json-object

Conversation

@keli-wen

@keli-wen keli-wen commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Support json_object-only providers (DeepSeek, ...) for both structured-output stages — draft structuring (flows/paper/_structure.py) and agentic retrieval (mind/retrieval.py) — without degrading capable providers or adding a provider registry.
  • Add quantmind.utils.structured_output.run_structured: a strict-first fallback ladder shared by both call sites. It runs the SDK's native strict json_schema structured output and, only when a provider rejects that response_format at request time (a narrow openai.BadRequestError check), re-runs the same call once in JSON-object mode with local Pydantic validation. Callers pass only cfg.model; the fallback is invisible, and bad requests unrelated to response_format are re-raised, never swallowed.
  • The helper is layer-agnostic: each call site supplies its own build_agent (how the agent is constructed) and run (which runner executes it) closures, so flows keeps run_with_observability and mind keeps Runner.run — no mind -> flows import edge is introduced.

Root cause

output_type= always makes the Agents SDK send a strict response_format={"type": "json_schema"}. Native OpenAI routes accept it; some LiteLLM-routed providers (DeepSeek) reject that response_format outright at request time, before any output exists — so a parse-stage retry can't help; the request itself must change.

What changed since the first version of this PR

The first iteration detected incapability by a model.startswith("litellm/") prefix and forced every routed model into json_object mode. That downgraded capable providers (losing schema enforcement) and, because json_object gives no schema guarantee, required a growing salvage stack: a *_node_ids alias normalizer, a UUID regex scan over free text, and a second tool-less selector agent. This revision detects incapability by the provider's own rejection instead — restoring strict schema for every capable provider (including the GPT baseline, which now genuinely exercises the strict path) and deleting that salvage stack. Net -92 lines in mind/retrieval.py.

Trade-off

A json_object-only provider now pays one rejected first request per call (the strict attempt). No provider registry or capability table is introduced (see the design doc's Out-of-Scope). A per-process "this model rejected strict" memo could remove that cost later if it matters.

Validation

  • bash scripts/verify.sh — 394 passed, 85.34% coverage.
  • Offline (both stages): strict-is-the-default, strict→json_object fallback on a simulated response_format rejection, and an unrelated BadRequestError propagating unswallowed.
  • Live: scripts/verify_structure_e2e.py with OPENROUTER_API_KEY — OpenRouter DeepSeek V4 Flash exercises the JSON-object fallback, GPT-5.6 Luna is the strict-path baseline.

Closes #126

@keli-wen keli-wen self-assigned this Jul 21, 2026
keli-wen and others added 5 commits July 22, 2026 00:28
…object salvage

Replace the litellm-prefix heuristic that forced every routed model into
json_object mode with a strict-first fallback ladder. Run the SDK's native
json_schema structured output, and only when a provider rejects that
response_format at request time (a narrow openai.BadRequestError check) re-run
once in json_object mode with local Pydantic validation. Callers pass only
cfg.model; the fallback is invisible and unrelated bad requests are re-raised,
never swallowed.

- utils/structured_output.py: add run_structured(output_type, *, build_agent,
  run); drop requires_json_object_mode.
- flows/paper/_structure.py + mind/retrieval.py: route both structured-output
  call sites through run_structured via per-layer build_agent/run closures, so
  mind keeps Runner.run and flows keeps run_with_observability (no mind->flows
  edge introduced).
- mind/retrieval.py: delete the json_object salvage stack (the *_node_ids alias
  normalizer, the UUID regex scan, and the second tool-less selector agent) now
  that strict schema is restored for capable providers. Net -92 lines there.
- contexts/design/mind/retrieval.md: document the transparent fallback and its
  boundary/registry constraints.

verify.sh green: 394 passed, 85.34% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…standard

Addresses review of the json-object fallback.

- Move the run_structured design out of mind/retrieval.md — a helper used across
  layers should not be documented inside one consumer — into a concise standalone
  page contexts/design/utils/structured_output.md, listed in the design index.
  retrieval.md's Multi-Model Compatibility reverts to its high-level form: no
  embedded mechanism, no cross-reference.
- Add skills/quantmind-dev/references/tests.md (mirrored to .agents and .claude):
  the canonical "how to write tests" standard — offline-deterministic default
  suite, coverage scope, live behavior as a scripts/verify_<component>_e2e.py
  slice, and a change->test obligations table. develop-components.md and
  contexts/dev/README.md now route to it.
- Delete tests/test_verify_structure_e2e.py: an offline test that only restated a
  script's constants is a meta-test; an e2e script is validated by running it in
  the e2e workflow, not unit-tested. The new standard codifies this rule.

verify.sh green: 392 passed, 85.34% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed-output page

A flowchart of the strict-first ladder — strict json_schema, the response_format
rejection check, and json_object fallback vs re-raise — reads faster than the
prose alone in contexts/design/utils/structured_output.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g guideline

TD flowcharts stack tall because of Mermaid's large default node height, so on
GitHub a moderately complex one pushes the page down several screens. Flip the
structured-output ladder to LR with shorter node labels, and record a
compactness guideline in the design index: pick direction by the diagram's story
(LR for a linear pipeline/ladder, TD for layered structures), keep node text
short, and split past ~15-20 nodes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…guidance there

Add skills/quantmind-dev/references/write-contexts.md (mirrored to .agents and
.claude): the canonical standard for authoring pages under contexts/ — required
structure (Quick Summary/Contents, Contents-anchor match, index registration,
all enforced by tests/test_contexts.py), prose wrapping, and Mermaid diagram
guidance. Since Mermaid only shows up in contexts pages, the compactness /
orientation guidance moves here from the design index, which now links out to
it. Route it from SKILL.md Select Workflow and the contexts/dev Rule Index.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@keli-wen
keli-wen marked this pull request as ready for review July 22, 2026 04:49
@keli-wen
keli-wen merged commit 27a08e6 into master Jul 22, 2026
5 checks passed
@keli-wen
keli-wen deleted the codex/issue-126-json-object branch July 22, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(mind): structure build and retrieve fail on json_object-only providers

1 participant