fix(mind): support json-object-only providers#128
Merged
Conversation
…object salvage Replace the litellm-prefix heuristic that forced every routed model into json_object mode with a strict-first fallback ladder. Run the SDK's native json_schema structured output, and only when a provider rejects that response_format at request time (a narrow openai.BadRequestError check) re-run once in json_object mode with local Pydantic validation. Callers pass only cfg.model; the fallback is invisible and unrelated bad requests are re-raised, never swallowed. - utils/structured_output.py: add run_structured(output_type, *, build_agent, run); drop requires_json_object_mode. - flows/paper/_structure.py + mind/retrieval.py: route both structured-output call sites through run_structured via per-layer build_agent/run closures, so mind keeps Runner.run and flows keeps run_with_observability (no mind->flows edge introduced). - mind/retrieval.py: delete the json_object salvage stack (the *_node_ids alias normalizer, the UUID regex scan, and the second tool-less selector agent) now that strict schema is restored for capable providers. Net -92 lines there. - contexts/design/mind/retrieval.md: document the transparent fallback and its boundary/registry constraints. verify.sh green: 394 passed, 85.34% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…standard Addresses review of the json-object fallback. - Move the run_structured design out of mind/retrieval.md — a helper used across layers should not be documented inside one consumer — into a concise standalone page contexts/design/utils/structured_output.md, listed in the design index. retrieval.md's Multi-Model Compatibility reverts to its high-level form: no embedded mechanism, no cross-reference. - Add skills/quantmind-dev/references/tests.md (mirrored to .agents and .claude): the canonical "how to write tests" standard — offline-deterministic default suite, coverage scope, live behavior as a scripts/verify_<component>_e2e.py slice, and a change->test obligations table. develop-components.md and contexts/dev/README.md now route to it. - Delete tests/test_verify_structure_e2e.py: an offline test that only restated a script's constants is a meta-test; an e2e script is validated by running it in the e2e workflow, not unit-tested. The new standard codifies this rule. verify.sh green: 392 passed, 85.34% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed-output page A flowchart of the strict-first ladder — strict json_schema, the response_format rejection check, and json_object fallback vs re-raise — reads faster than the prose alone in contexts/design/utils/structured_output.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g guideline TD flowcharts stack tall because of Mermaid's large default node height, so on GitHub a moderately complex one pushes the page down several screens. Flip the structured-output ladder to LR with shorter node labels, and record a compactness guideline in the design index: pick direction by the diagram's story (LR for a linear pipeline/ladder, TD for layered structures), keep node text short, and split past ~15-20 nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…guidance there Add skills/quantmind-dev/references/write-contexts.md (mirrored to .agents and .claude): the canonical standard for authoring pages under contexts/ — required structure (Quick Summary/Contents, Contents-anchor match, index registration, all enforced by tests/test_contexts.py), prose wrapping, and Mermaid diagram guidance. Since Mermaid only shows up in contexts pages, the compactness / orientation guidance moves here from the design index, which now links out to it. Route it from SKILL.md Select Workflow and the contexts/dev Rule Index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
keli-wen
marked this pull request as ready for review
July 22, 2026 04:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
json_object-only providers (DeepSeek, ...) for both structured-output stages — draft structuring (flows/paper/_structure.py) and agentic retrieval (mind/retrieval.py) — without degrading capable providers or adding a provider registry.quantmind.utils.structured_output.run_structured: a strict-first fallback ladder shared by both call sites. It runs the SDK's native strictjson_schemastructured output and, only when a provider rejects thatresponse_formatat request time (a narrowopenai.BadRequestErrorcheck), re-runs the same call once in JSON-object mode with local Pydantic validation. Callers pass onlycfg.model; the fallback is invisible, and bad requests unrelated toresponse_formatare re-raised, never swallowed.build_agent(how the agent is constructed) andrun(which runner executes it) closures, soflowskeepsrun_with_observabilityandmindkeepsRunner.run— nomind -> flowsimport edge is introduced.Root cause
output_type=always makes the Agents SDK send a strictresponse_format={"type": "json_schema"}. Native OpenAI routes accept it; some LiteLLM-routed providers (DeepSeek) reject thatresponse_formatoutright at request time, before any output exists — so a parse-stage retry can't help; the request itself must change.What changed since the first version of this PR
The first iteration detected incapability by a
model.startswith("litellm/")prefix and forced every routed model intojson_objectmode. That downgraded capable providers (losing schema enforcement) and, becausejson_objectgives no schema guarantee, required a growing salvage stack: a*_node_idsalias normalizer, a UUID regex scan over free text, and a second tool-less selector agent. This revision detects incapability by the provider's own rejection instead — restoring strict schema for every capable provider (including the GPT baseline, which now genuinely exercises the strict path) and deleting that salvage stack. Net-92lines inmind/retrieval.py.Trade-off
A
json_object-only provider now pays one rejected first request per call (the strict attempt). No provider registry or capability table is introduced (see the design doc's Out-of-Scope). A per-process "this model rejected strict" memo could remove that cost later if it matters.Validation
bash scripts/verify.sh— 394 passed, 85.34% coverage.json_objectfallback on a simulatedresponse_formatrejection, and an unrelatedBadRequestErrorpropagating unswallowed.scripts/verify_structure_e2e.pywithOPENROUTER_API_KEY— OpenRouter DeepSeek V4 Flash exercises the JSON-object fallback, GPT-5.6 Luna is the strict-path baseline.Closes #126