feat(inference): Phase 3 P3-B — crate-native turn path + delete compatible*.rs [WIP] by senamakel · Pull Request #4784 · tinyhumansai/openhuman

senamakel · 2026-07-11T05:55:28Z

DRAFT — WIP. This is the single Phase 3 P3-B PR, built incrementally with CI
as the verification loop (the shared dev box OOMs on local builds). It is not
ready until the hot turn path is crate-native and compatible*.rs is deleted;
I'll mark it ready then. Reviewing now is welcome but the diff will grow.

Goal (issue #4249, Phase 3 P3-B)

Route the agent turn path off the host OpenAiCompatibleProvider
(compatible*.rs, a Box<dyn Provider> wire client) onto the vendored
tinyagents crate's native ChatModels, then delete compatible*.rs (~430KB).
create_chat_model_with_model_id already returns crate-native models (managed →
OpenHumanBackendModel, local/cloud → crate OpenAiModel); the turn path is the
last big consumer still on the Provider path.

Landed so far

Step 1 — managed charged-USD parity (billing bug fix). The crate-native
managed model left the openhuman.{billing,usage} envelope only on
ModelResponse.raw, so the seam's usage_info_from_response recovered $0
charged + dropped backend cached tokens — a latent underbilling bug already
live for one-shot managed callers (memory/chat.rs, meeting/checkpoint
summaries). OpenHumanBackendModel::project_managed_usage now re-projects
charged-USD + context window into the openhuman_usage_meta shape the cost
bridge reads, and backfills crate Usage.cache_read_tokens. Verified:
cargo check --lib clean; unit tests added.

Planned increments (each pushed for CI)

Crate-native TurnModelSource/build_turn_models (per-tier models, caps
derived without a Provider).
Flip producers lowest-blast-radius first (sub-agent extract → triage →
sub-agent runner → channels → root session builder).
Migrate remaining direct OpenAiCompatibleProvider::new sites + one-shot
create_chat_provider callers.
Delete compatible*.rs; mark ready.

Known parity notes (deferred to the dedicated live-test run)

Streaming billing: the crate SSE parser sets raw: None on the terminal
chunk, so streaming managed turns fall back to the catalog cost estimate
(token counts survive; authoritative charged-USD needs an upstream crate change
to preserve the final chunk raw). Non-streaming invoke is fully correct.
Error kind tags: crate-native models surface TinyAgentsError, so the
DomainEvent::AgentError kind tag softens for provider failures. Sentry
suppression is unaffected (both skips_sentry cases —
MaxIterationsExceeded, EmptyProviderResponse — are raised in the host turn
loop, not via the model). A lightweight ProviderError→AgentError classifier
can restore kind tags if wanted.

https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…e path (P3-B step 1) The crate-native managed model (OpenHumanBackendModel) delegates to a crate OpenAiModel whose parse_response leaves the `openhuman.{billing,usage}` envelope only on ModelResponse.raw — it has no field for charged USD — so the seam's usage_info_from_response (which reads the `openhuman_usage_meta` key ProviderModel writes) recovered $0 charged + dropped backend cached tokens. This is a latent underbilling bug already live for one-shot managed callers (memory/chat.rs, meeting/checkpoint summaries) and would extend to every interactive turn once the turn path goes crate-native. - model.rs: add merge_openhuman_usage_meta — the symmetric writer for usage_info_from_response, stashing charged-USD + context window under OPENHUMAN_USAGE_META_KEY on the response raw (no-op when both are zero). - openhuman_backend_model.rs: project_managed_usage reads the openhuman envelope off raw, re-projects charged-USD + context window into openhuman_usage_meta, and backfills crate Usage.cache_read_tokens from the (authoritative) envelope count when the standard parse produced none. Applied on the invoke path. - Streaming: documented parity gap — the crate SSE parser sets raw:None on the terminal Completed, so streaming managed turns fall back to the catalog cost estimate (token counts survive); restoring authoritative charged USD for streaming needs the crate to preserve the final chunk raw (tracked upstream). - Tests: project_managed_usage recovers charged/cached/window; no-op without envelope. Part of Phase 3 P3-B (turn-path crate-native cutover, tinyhumansai#4249). Local full builds OOM on the shared box; verification defers to CI. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

coderabbitai · 2026-07-11T05:55:36Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b74c9f5f-da05-41ee-b195-4bd4a4b8f9e0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

…native turn builder (P3-B step 2 foundation) The per-(role, explicit-model) analogue of create_chat_model_with_model_id, for the crate-native TurnModelSource to build the primary + each workload-tier route without a host Provider. Managed -> OpenHumanBackendModel pinned to the model (backend resolves the tier from request.model); local/cloud -> crate builders; bespoke (claude-code) -> ProviderModel over the resolved Provider. Respects the test-provider override. Additive (allow(dead_code)) — wired into TurnModelSource::build in the next increment. Part of Phase 3 P3-B (tinyhumansai#4249). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…e opt-in (P3-B step 2) TurnModelSource gains an optional crate_native (role, Arc<Config>): when set, build()/build_summarizer() construct the tiered TurnModels crate-natively via build_turn_models_crate (per-tier factory::create_turn_chat_model) instead of wrapping one host Provider in ProviderModels. The source still holds the resolved provider, used as (a) the escape-hatch for sub-agent inheritance / rhai and (b) a graceful fallback if crate-native construction fails — so producers can flip to new_crate_native one at a time without regressing behavior. build_turn_models_crate derives the harness-visible caps without a Provider: native_tools/supports_vision from the primary ModelProfile (managed backend, profile None, falls back to true / oh_tier_supports_vision(model)); provider_id from the resolved slug (managed for the backend); error_slot empty (crate-native surfaces TinyAgentsError directly — Sentry suppression is unaffected since both skips_sentry cases are host-loop-raised). Unbuildable tiers are skipped, not fatal. No producer flipped yet (new_crate_native is allow(dead_code)); the build/summarizer dispatch + fallback land first so the next increment can flip producers behind a verified path. cargo check --lib clean. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…3-B step 3, producer 1/7) First producer cutover: the sub-agent extract summarizer now builds its TurnModelSource via new_crate_native("summarization", cfg) when config loads, so build_summarizer constructs a crate-native ChatModel (managed -> OpenHumanBackendModel) instead of a ProviderModel. The resolved provider is retained for the escape-hatch + crate-native-failure fallback; a config-load failure degrades to the parent provider on the Provider path (unchanged). Lowest-blast-radius producer (internal extraction, no interactive Sentry path). new_crate_native is now live (allow(dead_code) removed). cargo check --lib clean. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…serving tinyhumansai#1257 (P3-B step 3, producer 2/7) The triage turn now builds crate-native models for the remote/managed arm via TurnModelSource::new_crate_native_from_string, preserving the tinyhumansai#1257 "triage never goes local" invariant: build_remote_provider's effective provider string (openhuman when force-managed) rides through a new primary override so build() resolves the managed backend rather than re-deriving from the role (which could go local). - factory::create_turn_chat_model_from_string(role, provider_string, config, model, temp): managed when the string is openhuman/empty/cloud (-> OpenHumanBackendModel pinned to model); else delegates to create_turn_chat_model. Respects test override. - CrateNativeSource gains primary_override: Option<String>; new_crate_native_from_string; build_turn_models_crate uses it for the primary + summarizer (routes unchanged). - ResolvedProvider gains crate_native: Option<(effective_string, Arc<Config>)> — Some for the remote/managed arm, None for the local arm (kept on the Provider path so it doesn't silently route to the managed backend and defeat the local fallback). - Evaluator flips on resolved.crate_native; resolved provider retained as escape-hatch + crate-native-failure fallback. cargo check --lib clean. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…sai#55 prompt-guided tools) Pulls in tinyagents#55 (prompt-guided/text-mode tool calling for non-native models) + the v1.9.0 release. Unblocks flipping LOCAL runtimes to crate-native on the turn path: they force native_tools=false and now get prompt-guided tools from the crate instead of losing tool calls. Host lib compiles clean against 1.9.0. Part of Phase 3 P3-B (tinyhumansai#4249). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…age e2e (P3-B fix) The triage flip added ResolvedProvider.crate_native; cargo check --lib doesn't compile tests/ integration targets, so the 4 mock ResolvedProvider constructors in tests/raw_coverage/inference_agent_raw_coverage_e2e.rs (globbed into the raw_coverage_all target) were missed and broke the full-suite CI on the pin-bump push. Added crate_native: None (mocks stay on the Provider path). Verified via cargo check --test raw_coverage_all. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…r (P3-B) The extract flip (producer 1/7) re-resolved "summarization" from config via create_turn_chat_model, bypassing the extract's already-resolved provider — incl. a test-injected mock the raw_coverage e2e observes (extraction_prompts()). Full suite went red: integrations_text_mode_handoffs_oversized_result_and_extracts_from_cache "got 0 prompts". Production stays managed either way, but a producer that injects/ resolves a SPECIFIC provider must keep it on the Provider path (or pass an effective string via new_crate_native_from_string, like triage) — plain new_crate_native role-resolves and drops the injected provider. Extract now stays TurnModelSource::new(extract_provider). new_crate_native (plain) is allow(dead_code) until the session-builder flip. Verified: the exact failing test now passes (cargo test --test raw_coverage_all --features e2e-test-support). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…P3-B, producer 3) The production main-turn agent now builds crate-native ChatModels from (provider_role, config) via create_turn_chat_model instead of a ProviderModel- wrapped provider (managed -> OpenHumanBackendModel, local -> crate OpenAiModel w/ prompt-guided tools, cloud -> crate OpenAiModel native). - setters.rs: AgentBuilder::crate_native_provider(provider, role, Arc<Config>) — new_crate_native source; the plain .provider() setter (Provider path) stays for tests that inject a mock they observe. - factory.rs:1144: .provider(provider) -> .crate_native_provider(provider, provider_role, Arc::new(config.clone())). - EXACT metadata parity: build_turn_models_crate now takes provider_id/native_tools/ supports_vision as params, derived in TurnModelSource::build from the RETAINED provider (self.provider.telemetry_provider_id/supports_native_tools/supports_vision) — the SAME source build_turn_models uses — so these fields are byte-identical to the Provider path (fixes the supports_vision divergence); only the MODELS go crate-native. Sub-agents inherit the retained provider via the escape-hatch (Provider path, unaffected). Crate-native-build failure falls back to the Provider path. Safe for tests: e2e/raw_coverage tests build agents via Agent::builder().provider(mock) directly (Provider path) or TurnModelSource::new(provider); NONE drive the production factory, so the flip is not reachable from them. Live per-provider validation deferred to the dedicated tinyagents run. Compiles clean (lib + raw_coverage_all). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…rness_e2e (P3-B) The session-builder flip (2528fa0) broke ~14 tests/agent_harness_e2e.rs turn tests: they drive the production factory (factory.rs:1144) via the harness/SSE flow and inject a ScriptedProvider NOT through test_provider_override, so create_turn_chat_model's override guard doesn't catch it — the crate-native re-resolution builds a real managed model (no backend) -> no SSE events -> timeout. Reverted factory.rs:1144 to .provider(provider). Kept (allow(dead_code)): the metadata-parity refactor, crate_native_provider setter, new_crate_native. Restores tinyhumansai#4784 to green (triage flip + infrastructure intact). Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…2e mock (P3-B, producer 3) Adds SSE streaming to the agent_harness_e2e scripted upstream mock (serves text/event-stream data: choices[].delta chunks + [DONE] when stream:true, matching the crate OpenAiModel::stream parser; both old + crate paths parse real SSE), which was the blocker: the crate streaming path needs real SSE but the mock only returned JSON. Re-flips factory.rs:1144 to .crate_native_provider so the main interactive turn builds crate-native models (managed->OpenHumanBackendModel etc.) with metadata parity from the retained provider. Verifying at the end per the full-cutover plan. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

ChannelRuntimeContext gains config: Option<Arc<Config>> (Some in production via start_channels; None in tests keeps the Provider path). The channel turn builds crate-native models via new_crate_native_from_string("chat", route.provider, config) when config is present; the cached provider is retained for the escape-hatch + fallback + metadata parity. cargo check --lib clean (all 24 ChannelRuntimeContext sites carry the new field). Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

Pins tinyagents#56 (OpenAiModel::stream falls back to parsing a non-SSE JSON body as one completion) so the crate-native turn cutover works against OpenHuman's JSON-only e2e mocks. Pinned to the branch commit (the vendored [patch] path build resolves it directly; no crates.io release needed). Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…roducer 5) Both sub-agent graph paths (default + custom) build a crate-native subagent_source: a workload Hint -> that role; any other spec (Inherit/Exact/override) reuses the parent's role via TurnModelSource::crate_native_role() when the parent is crate-native. Resolved provider retained for fallback + metadata parity; falls back to the Provider path when no role is derivable / config load fails. Completes the turn-producer cutover (session, channels, triage, sub-agent all crate-native). cargo check --lib clean. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…e3b-turnpath-cutover

…e3b-turnpath-cutover # Conflicts: # app/src-tauri/Cargo.lock # tests/raw_coverage/memory_threads_raw_coverage_e2e.rs

senamakel added 28 commits July 11, 2026 05:59

fix(inference): preserve resolved subagent provider

44b1e29

refactor(inference): port one-shot consumers to chat models

eb43642

refactor(inference): build explicit providers as chat models

c085273

refactor(memory-tree): use crate-native local summarizer

105a1fc

refactor(agent): drop provider retention from native turns

0e21a53

refactor(agent): resolve subagents through model sources

fe4330c

refactor(channels): stop caching production providers

cfe1788

refactor(triage): carry crate-native model sources

70c3091

refactor(inference): port remaining one-shot provider calls

faab86d

test(memory-tree): remove stale provider test

2dc28ab

chore(tauri): refresh tinyagents lock entry

edd79b8

refactor(agent): keep integrations text mode crate-native

e761dc0

refactor(inference): remove legacy provider adapters from live calls

6ff9dbd

refactor(inference): delete host compatible wire client

7291301

fix(inference): preserve crate adapter parity edges

3d7618d

senamakel added 30 commits July 12, 2026 03:44

test(inference): serve native SSE content type

4e47f28

test(memory): disambiguate host tool memory store

2fb2026

Merge remote-tracking branch 'upstream/main' into feat/inference-phas…

a798cb0

…e3b-turnpath-cutover

test(memory): construct crate store through host bridge

50827e4

fix(agent): preserve injected text-mode providers

ba6c99a

Merge remote-tracking branch 'upstream/main' into feat/inference-phas…

7162a22

…e3b-turnpath-cutover # Conflicts: # app/src-tauri/Cargo.lock # tests/raw_coverage/memory_threads_raw_coverage_e2e.rs

test(inference): preserve streamed content whitespace

25b5079

test(inference): preserve streamed reasoning whitespace

2c70040

test(inference): expect crate-native stream usage

bd801f8

test(inference): retain embedded tool json as text

96e039c

test(inference): preserve compatible response content

970913a

test(inference): expect generic reasoning projection

33ddcc1

test(inference): drop legacy function usage projection

0088241

test(inference): assert crate normalized 404

35ec213

test(inference): retain crate-native tool specs

9cea5b4

test(inference): drop legacy tool synthesis expectations

d0e1e2f

test(inference): retain standard compatible usage

805698f

test(inference): assert attributable tool delta

96dd012

test(inference): preserve crate request transcript

c8bffe3

test(inference): omit generic reasoning extension

f532231

test(inference): accept crate missing-key wording

c538d8c

test(inference): expect unauthenticated dispatch error

9cccdc0

test(inference): assert streamed provider failure

11315b2

test(inference): expect history stream fallback chunk

02b651e

test(inference): configure crate provider base URL

79a0665

chore(tinyagents): update to latest main

c6ad431

test(tinyagents): initialize tool validity metadata

fcd1f12

test(inference): expect latest reasoning extraction

526bceb

test(inference): match legacy reasoning projection

47ad1ec

test(inference): retain legacy usage boundary

b05e49a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(inference): Phase 3 P3-B — crate-native turn path + delete compatible*.rs [WIP]#4784

feat(inference): Phase 3 P3-B — crate-native turn path + delete compatible*.rs [WIP]#4784
senamakel wants to merge 61 commits into
tinyhumansai:mainfrom
senamakel:feat/inference-phase3b-turnpath-cutover

senamakel commented Jul 11, 2026

Uh oh!

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

senamakel commented Jul 11, 2026

Goal (issue #4249, Phase 3 P3-B)

Landed so far

Planned increments (each pushed for CI)

Known parity notes (deferred to the dedicated live-test run)

Uh oh!

coderabbitai Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading