-
Notifications
You must be signed in to change notification settings - Fork 286
feat(llm): declare reasoning levels per configured route #320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
c24ad4b
feat(llm): declare reasoning levels per configured route
furgalep 5b52990
test(llm): isolate SDK parameter dropping in effort probes
furgalep a1911a1
docs(llm): record capped Hub reasoning-level probes
furgalep 5fa6794
fix(llm): reserve request controls in reasoning declarations
furgalep c91c645
docs(llm): load example reasoning registry before using its alias
furgalep 8e84ddb
fix(llm): revalidate effort declarations before applying a level
furgalep 26658c2
docs(skills): teach agents to configure reasoning levels
furgalep 5a40b67
fix(llm): isolate reasoning declarations from route overrides
furgalep File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Reasoning levels | ||
|
|
||
| UnifiedLLM exposes the choices a configured route supports. Model names do not | ||
| determine those choices: registry YAML maps each label to exact request parameters. | ||
| Selecting a label applies `params.update(level_settings)` before dispatch. | ||
|
|
||
| ```python | ||
| from pathlib import Path | ||
|
|
||
| from nooa.unifiedllm import get_llm_client | ||
| from nooa.unifiedllm.registry import reload_registry | ||
|
|
||
| # From the repository root; this example registry is not loaded automatically. | ||
| reload_registry(Path("examples/reasoning_levels/llm_config.yaml")) | ||
| client = get_llm_client("gpt-5.6-sol") | ||
| print(client.reasoning_levels) # tuple of labels; None = unknown, () = unsupported | ||
| print(client.reasoning_default) # documented default, or None when unknown | ||
| response = await client.acall(messages, reasoning_level="high") | ||
| ``` | ||
|
|
||
| The constructor also accepts `reasoning_level` as a persistent selection. A | ||
| per-call selection overrides it; explicit `reasoning_level=None` uses the raw | ||
| base configuration for that call. `reasoning_default` is metadata only: declaring | ||
| it does not add parameters, change costs or override existing provider settings. | ||
|
|
||
| ## Registry declarations | ||
|
|
||
| ```yaml | ||
| models: | ||
| my-route: | ||
| model_name: openai/my-model | ||
| reasoning: {effort: medium, context: all_turns} | ||
| reasoning_default: medium | ||
| reasoning_levels: | ||
| low: {reasoning: {effort: low, context: all_turns}} | ||
| medium: {reasoning: {effort: medium, context: all_turns}} | ||
| high: {reasoning: {effort: high, context: all_turns}} | ||
| ``` | ||
|
|
||
| Write the complete nested block for each level. There is no deep merge or | ||
| inheritance: selecting a level replaces the base value at each key it sets. | ||
| The declarations are trusted configuration, just like the rest of the registry; | ||
| they are not restricted to a list of provider fields maintained by NOOA. | ||
| Client routing and framework controls are reserved: `model`, `api_base`, | ||
| `base_url`, `api_key`, `custom_llm_provider`, `client`, `messages`, `input`, `extra_body`, | ||
| and the three `reasoning_*` configuration fields. They cannot appear inside a | ||
| level's settings. A level changes effort, not the endpoint, credentials or history. | ||
|
|
||
| Omit `reasoning_levels` (or use null) when support is unknown. An empty mapping | ||
| explicitly declares selection unsupported. Unknown support, unsupported selection | ||
| and invalid labels produce distinct errors; an invalid label lists valid choices. | ||
| Without a managed selection, raw provider parameters continue working as before. | ||
|
|
||
| A selected level cannot be combined with a per-call setting of the same key, | ||
| including inside `extra_body`. Choose the label or the raw settings, not both. | ||
| Constructor defaults in `extra_body` are replaced by the selected settings, | ||
| just like top-level defaults. Unrelated defaults are preserved. | ||
| Declarations belong on the constructor, not per-call kwargs or `extra_body`. | ||
| Changing model or endpoint while using a managed level requires a new client: | ||
| one route's declared choices must not be applied to another route. | ||
| Supplying an SDK client per call is also rejected with a managed level because | ||
| that client can choose a different endpoint. When overriding a registry alias's | ||
| route or client type, inherited levels, default and selection are cleared; | ||
| declare replacement levels explicitly, or leave support unknown. | ||
|
|
||
| ## Where the data comes from | ||
|
|
||
| The [example registry](../examples/reasoning_levels/llm_config.yaml) illustrates | ||
| three request shapes, based on the [GPT-5.6 Sol model documentation](https://developers.openai.com/api/docs/models/gpt-5.6-sol), | ||
| [Claude effort documentation](https://platform.claude.com/docs/en/build-with-claude/effort) | ||
| and [Gemini's OpenAI-compatible API](https://ai.google.dev/gemini-api/docs/openai). | ||
| Its endpoints are placeholders. Replace them, the model IDs and the declared | ||
| choices with settings for your own route before making calls. It is not | ||
| auto-loaded. Load your configured file with `reload_registry(Path(...))`. | ||
| Private endpoint settings and credentials belong in private configuration, not | ||
| this public example. | ||
|
|
||
| Provider documentation describes provider APIs, not all gateway routes. Mocked | ||
| HTTP tests check that the example's settings survive the installed transport; | ||
| the opt-in live test checks your configured route's acceptance, not reasoning quality or every | ||
| level's behavior. Do not infer support merely from a successful HTTP response | ||
| if a gateway silently ignores parameters. | ||
|
|
||
| Do not enable LiteLLM's global `drop_params` when verifying a declaration: it can | ||
| discard fields for gateway IDs it does not recognize, even with per-call | ||
| `drop_params=False`. The HTTP tests explicitly disable that global flag and check | ||
| the serialized fields. NOOA does not change process-global SDK configuration. | ||
|
|
||
| LangChain/Pi data can inform maintenance, but neither is a runtime dependency or | ||
| an automatic build input. Updating a route means reviewing its small declaration | ||
| and request tests, rather than importing hundreds of profiles. | ||
|
|
||
| ## Scope and architecture | ||
|
|
||
| - `unifiedllm/reasoning.py` validates declarations and applies the chosen settings. | ||
| The same function serves synchronous and asynchronous Chat and Responses calls. | ||
| - Registry fields are passed into UnifiedLLM and consumed before provider dispatch. | ||
| Renderers, events and middleware do not translate reasoning levels. | ||
| - No selection changes existing behavior. Stored reasoning, compatibility gates, | ||
| session archives and replay remain unchanged. This is not a retention toggle. | ||
| - Effort labels are provider-local, not comparable units of intelligence or cost. | ||
| Changing effort can invalidate a provider's cached prefix. This PR does not add | ||
| cache-preserving mid-turn steering, TUI controls, or selection persistence. | ||
|
|
||
| ## Tests | ||
|
|
||
| Run `uv run pytest tests/unifiedllm/test_reasoning_levels.py tests/unifiedllm/test_reasoning_levels_wire.py`. | ||
| The tests check configuration ownership, invalid selections, route changes, | ||
| unchanged defaults and the serialized HTTP requests for the example routes. | ||
|
|
||
| For paid probes, configure registry aliases with a `low` level and credentials | ||
| through their normal `api_key_env` settings. Then opt in and name the aliases: | ||
|
|
||
| ```sh | ||
| NOOA_RUN_REASONING_LEVELS_LIVE=1 NOOA_REASONING_TEST_MODELS=my-route uv run pytest \ | ||
| tests/integration/test_reasoning_levels_live.py -m integration -q -s | ||
| ``` | ||
|
|
||
| The alias list is comma-separated. Missing aliases are skipped. Each configured | ||
| alias gets one request capped at 256 output tokens, with retries disabled. | ||
| The test checks outgoing settings and route acceptance, not that every effort | ||
| label changes model behavior. Provider-specific results belong with the | ||
| configuration used to run them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Illustrative request shapes, not a built-in or verified model catalog. | ||
| # Replace the placeholder endpoint, model IDs and levels with your route's settings. | ||
| models: | ||
| gpt-5.6-sol: | ||
| model_name: openai/gpt-5.6-sol | ||
| client_type: responses | ||
| api_base: https://gateway.example.com/v1 | ||
| api_key_env: MODEL_API_KEY | ||
| store: false | ||
| include: [reasoning.encrypted_content] | ||
| reasoning: {effort: medium} | ||
| # https://developers.openai.com/api/docs/models/gpt-5.6-sol | ||
| reasoning_default: medium | ||
| reasoning_levels: | ||
| none: {reasoning: {effort: none}} | ||
| low: {reasoning: {effort: low}} | ||
| medium: {reasoning: {effort: medium}} | ||
| high: {reasoning: {effort: high}} | ||
| xhigh: {reasoning: {effort: xhigh}} | ||
| max: {reasoning: {effort: max}} | ||
|
|
||
| claude-sonnet-5: | ||
| model_name: anthropic/claude-sonnet-5 | ||
| api_base: https://gateway.example.com | ||
| api_key_env: MODEL_API_KEY | ||
| # https://platform.claude.com/docs/en/build-with-claude/effort | ||
| reasoning_default: high | ||
| reasoning_levels: | ||
| low: {thinking: {type: adaptive}, output_config: {effort: low}} | ||
| medium: {thinking: {type: adaptive}, output_config: {effort: medium}} | ||
| high: {thinking: {type: adaptive}, output_config: {effort: high}} | ||
| max: {thinking: {type: adaptive}, output_config: {effort: max}} | ||
|
|
||
| gemini-3.1-pro-preview: | ||
| model_name: openai/gemini-3.1-pro-preview | ||
| api_base: https://gateway.example.com/v1 | ||
| api_key_env: MODEL_API_KEY | ||
| allowed_openai_params: [reasoning_effort] | ||
| # https://ai.google.dev/gemini-api/docs/openai#thinking | ||
| reasoning_default: high | ||
| reasoning_levels: | ||
| low: {reasoning_effort: low} | ||
| medium: {reasoning_effort: medium} | ||
| high: {reasoning_effort: high} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """Declared effort choices, independent of model names and provider discovery.""" | ||
|
|
||
| from collections.abc import Mapping | ||
| from copy import deepcopy | ||
| from typing import Any | ||
|
|
||
| from pydantic import BaseModel, ConfigDict, model_validator | ||
|
|
||
| _DECLARATIONS = {"reasoning_levels", "reasoning_default"} | ||
| # These select the client/request itself, not a provider's effort behavior. | ||
| _RESERVED = _DECLARATIONS | { | ||
| "reasoning_level", | ||
| "model", | ||
| "api_base", | ||
| "base_url", | ||
| "api_key", | ||
| "custom_llm_provider", | ||
| "messages", | ||
| "input", | ||
| "extra_body", | ||
| "client", | ||
| } | ||
|
|
||
|
|
||
| class ReasoningConfig(BaseModel): | ||
| """Map public level names to request settings for one configured route. | ||
|
|
||
| None means unknown support; an empty mapping means unsupported. The default | ||
| documents the route's default, not a request to send it on every call. | ||
| Declarations live in registry YAML, not a model-name table in this module. | ||
| """ | ||
|
|
||
| model_config = ConfigDict(extra="forbid", frozen=True) | ||
|
|
||
| levels: dict[str, dict[str, Any]] | None = None | ||
| default: str | None = None | ||
|
|
||
| @model_validator(mode="after") | ||
| def validate_declaration(self): | ||
| """Reject malformed choices and request-control fields at construction.""" | ||
| if self.default is not None and self.default not in (self.levels or {}): | ||
| raise ValueError("reasoning_default must name a declared reasoning level") | ||
| for level, settings in (self.levels or {}).items(): | ||
| if not level.strip() or not settings: | ||
| raise ValueError("reasoning_levels must have non-empty names and request settings") | ||
|
furgalep marked this conversation as resolved.
|
||
| if conflict := _RESERVED & settings.keys(): | ||
| raise ValueError( | ||
| f"reasoning level {level!r} contains reserved fields: {sorted(conflict)}" | ||
| ) | ||
| return self | ||
|
|
||
| def settings(self, level: str) -> dict[str, Any]: | ||
| """Validate a selection and detach its settings from the stored declaration.""" | ||
| if self.levels is None: | ||
| raise ValueError( | ||
| "Reasoning levels are unknown for this route; declare reasoning_levels" | ||
| ) | ||
| if not self.levels: | ||
| raise ValueError("Reasoning-level selection is not supported for this route") | ||
| if not isinstance(level, str) or level not in self.levels: | ||
| raise ValueError( | ||
| f"Invalid reasoning level {level!r}; allowed: {', '.join(self.levels)}" | ||
| ) | ||
| # Frozen Pydantic attributes do not freeze nested dictionaries. Reuse the | ||
| # declaration checks so later edits cannot introduce routing controls. | ||
| self.validate_declaration() | ||
| # Only the small chosen configuration is copied, never conversation data. | ||
| return deepcopy(self.levels[level]) | ||
|
|
||
|
|
||
| def apply_reasoning_level( | ||
| declaration: ReasoningConfig, | ||
| model: str, | ||
| defaults: dict[str, Any], | ||
| overrides: dict[str, Any], | ||
| default_selection: str | None, | ||
| ) -> dict[str, Any]: | ||
| """Resolve effort once before either client dispatches. | ||
|
|
||
| No selection leaves existing provider settings untouched. An explicit level | ||
| replaces constructor defaults; mixing it with per-call native controls is an | ||
| error. Route changes cannot inherit a declaration for a different endpoint. | ||
| This affects requested effort, never stored reasoning or replay compatibility. | ||
| """ | ||
| if _DECLARATIONS & overrides.keys(): | ||
| raise ValueError("reasoning_levels and reasoning_default belong on the client constructor") | ||
| params = {**defaults, **overrides} | ||
| extra = params.get("extra_body") | ||
| if isinstance(extra, Mapping) and (set(extra) & (_DECLARATIONS | {"reasoning_level"})): | ||
| raise ValueError("Reasoning configuration cannot be passed through extra_body") | ||
| level = params.pop("reasoning_level", default_selection) | ||
| if level is None: | ||
| return params | ||
| patch = declaration.settings(level) | ||
| if ( | ||
| any( | ||
| key in overrides and overrides[key] != defaults.get(key) | ||
| for key in ("api_base", "base_url", "custom_llm_provider") | ||
|
furgalep marked this conversation as resolved.
|
||
| ) | ||
| or overrides.get("model", model) != model | ||
| or overrides.get("client") is not None | ||
| ): | ||
| raise ValueError("Reasoning levels are route-specific; create a client for the new route") | ||
| explicit_extra = overrides.get("extra_body") | ||
| if conflict := patch.keys() & ( | ||
| overrides.keys() | (explicit_extra.keys() if isinstance(explicit_extra, Mapping) else set()) | ||
| ): | ||
| raise ValueError( | ||
| f"reasoning_level conflicts with explicit request field(s): {sorted(conflict)}" | ||
| ) | ||
| # Whole top-level values replace defaults. Authors write complete nested | ||
| # blocks in YAML; no provider-specific merge or inheritance rules live here. | ||
| # Remove replaced defaults from extra_body too: SDKs otherwise merge those | ||
| # back over the selected top-level values when assembling the HTTP body. | ||
| if isinstance(extra, Mapping) and patch.keys() & extra.keys(): | ||
| params["extra_body"] = {key: value for key, value in extra.items() if key not in patch} | ||
| params.update(patch) | ||
| return params | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.