Perf/hot path overhaul - #114
Merged
Merged
Conversation
… paths - cache ~/.timbal file config reads; skip re-resolving inherited platform config on session-chained runs (was re-reading disk every run) - gate per-event logging on effective log level so suppressed logs no longer pay model_dump() - run param/when lambdas and hooks inline instead of a threadpool round-trip - run plain sync handlers inline on the event loop; new Runnable.offload_blocking opt-in restores the executor for blocking handlers - convert events, Span and RunStatus from pydantic models to plain __slots__ classes (new timbal/_slots.py base) with model_dump()/model_dump_json() compat and a validate_event() wire-rehydration helper - store hot Runnable/RunContext runtime state as plain instance attributes instead of PrivateAttr (~20x faster reads); declare workflow wiring attrs as real fields instead of extra="allow" - fix latent bug: hook is_coroutine flags were written to the class by the validator, letting instances with different sync/async hooks clobber each other (now per instance, with regression test) - misc: uuid7(as_type="hex"), skip empty-session dump per span save Workflow bench vs LangGraph bare: sequential p50 1.13ms -> 0.59ms, burst-500 p50 415ms -> 106ms, throughput c=200 841/s -> 3747/s. Agent loop p50 -40%, memory per run -78%. All results in benchmarks/langchain/results/ refreshed. Co-authored-by: Cursor <cursoragent@cursor.com>
… single-tool dispatch - _execute_simple: plain sync/coroutine handlers bypass the async-generator (event, output, collector) tuple protocol entirely; _execute_handler delegates to it and Tool overrides it for the credential-to-proxy fallback - skip approval policy resolution when requires_approval is False (default); no ApprovalPolicyDecision construction per call - agent: single tool call iterates the tool directly (like Workflow steps) instead of spinning up a queue + task; shared dispatch-failure helpers - workflow: put_nowait on the unbounded event queue; dependency waits without a gather() task per dependency - misc: cache TimbalCollector class after first import; skip get_session() coroutine for nested calls Workflow bench p50 vs previous commit: sequential 671 to 484 us, diamond 539 to 474 us; burst-500 p50 116 to 95 ms. Fan-out overhead now 49 us/branch vs LangGraph bare at 93. Agent burst wall 23.6 to 16.4 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
Golden tests for event counts, type order, path nesting, and DeltaEvent forwarding — including parallel tool calls, parallel workflow steps, agent-as-tool grandchildren, and workflow-in-workflow. Guards upcoming event-plumbing refactors against dropped, duplicated, or reordered events. Co-authored-by: Cursor <cursoragent@cursor.com>
…the API boundary Runnable.__call__ is now a thin wrapper returning TimbalCollector(self._stream(**kwargs)); the former decorated generator body lives in _stream(). The collector — whose job is .collect() and pending approval/interaction enrichment — only wraps the stream the caller sees. Framework internals (workflow steps, agent tool dispatch, the agent's LLM call, command-path tools) iterate _stream() directly, removing one collector layer (one __anext__ frame + isinstance dispatch) per event per nesting level. Public API is unchanged. Guarded by the golden stream-shape tests added in the previous commit (exact event counts/order/paths incl. grandchildren and delta forwarding). Workflow probe: 646k -> 618k function calls per 300 runs, best-run p50 597 -> ~516 us. Burst-500 p50 ~95 -> ~90 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
Extract three self-contained blocks from the ~550-line _stream body, with no behavior change: - _apply_approval_gate(): the whole human-approval gate (redaction, resume claim, cancel/deny/pending/approve outcomes, edit-on-approve). Returns (proceed, approval_event, validated_input); all span mutations happen inside. The caller just yields the pending event and returns when gated. - _finalize_suspend(): records a suspend() pause on the span and builds the InteractionEvent. - _spawn_background_task(): background-mode task spawn + registration on the parent runnable. Drops the vestigial `nonlocal output, collector` from the bg closure: `output` was provably dead (no await between task spawn and the outer read) and `collector` only mattered in an unreachable cancellation race. _stream now reads as the actual lifecycle: context setup -> START -> resolve/ validate input -> approval gate -> pre_hook -> execute (simple | streaming | background) -> suspend/finalize -> exception ladder -> OUTPUT. Co-authored-by: Cursor <cursoragent@cursor.com>
Three scenarios with faked LLMs (no tools): N-message history injected stateless (N up to 200), 20-message history with message size up to 20KB, and a stateful 20-turn session (Timbal parent_id chaining vs LangGraph MemorySaver checkpointer). Timbal traces on, LangGraph bare. Full-mode results: Timbal wins every configuration — 5.4x at 10-message histories narrowing to 1.34x at 200 (validation+dump scale per message: 12.2 vs 11.2 us/msg marginal), 2.3-3.9x on long messages (33 vs 47 us/KB), and 5.4-12.4x per turn on stateful sessions (6.2 vs 41.7 ms full-session wall). The N=200 convergence quantifies the payoff for caching message dumps/conversions - the next optimization candidate. Co-authored-by: Cursor <cursoragent@cursor.com>
…histories Profiling a 200-message turn showed 79% of wall time in dump(): the same history is re-serialized ~3x per turn (span input dump, memory dump, LLM span input dump), and every pydantic content item paid two exception-raising __getattr__ marker probes. - Message grows a _cached_dump slot (validated against len(content), the only sanctioned in-place mutation being content appends). dump() serves the cache on both sync and async paths; hooks may mutate messages in place, so post_hook now explicitly invalidates output caches via new utils.invalidate_message_dump_caches(). - serialization: BaseModel branch (with an isinstance Runnable check) moved before the getattr marker probes, so content models no longer raise through pydantic __getattr__ per item; SlotModel marker check runs after. - TestModel token estimate uses len(text) for TextContent instead of stringifying whole pydantic models (str() kept for non-text blocks — compaction tests rely on large tool results driving usage). 200-message turn probe: 2413 -> 463 us (5.2x). Long-conversation bench vs LangGraph bare: history scaling 1.34x-5.4x -> 8.3x-14.7x (marginal cost 12.2 -> 1.5 us/message vs LG 7.5), message-size scaling 2.3x-3.9x -> 9.6x-16.5x (0.23 vs 51 us/KB), 20-turn session wall 6.2 -> 2.8 ms (LG 35.7), session memory 164 -> 45 KB. Agent bench p50 drops to 270-330 us (13-20x vs LG bare) since dumps also repeat within single runs. Co-authored-by: Cursor <cursoragent@cursor.com>
Workflow step execution is now a single _run_step async generator (dep wait, when-guard, param resolution, event streaming, outcome classification on StepStatus.signal) shared by two consumers: - linear chains (each step depends exactly on its predecessor — the common sequential pipeline) iterate it directly: no task per step, no queue, no fan-in wakeups. Insertion order is topological since links can only point at already-registered steps. - true DAGs keep the task-per-step + queue multiplexer via a thin _enqueue_step_events wrapper (sentinel = the step's own StepStatus). Linear-mode parity details: BaseException from user when/resolver callables is contained like task isolation did (surfaced via WorkflowStepError), and a step swallowing the task's CancelledError (recording 'interrupted') is detected via current_task.cancelling() and re-raised before forwarding post-cancel events, preserving partial-output salvage semantics. Also: cache imports in _emit_default_tool_usage (ran two imports per successful tool call) and drop the dead _is_timbal_runnable marker (its only reader was removed with the serialization branch reorder). Sequential workflow: p50 569 -> 329 us (LG bare 1.42 ms), burst-500 p50 90 -> 66 ms (LG 329 ms), 618k -> 498k function calls per 300 probe runs. Fan-out/diamond unchanged (task path). Co-authored-by: Cursor <cursoragent@cursor.com>
Sync handlers run inline since the executor removal; a blocking one stalls every concurrent run on the worker with no visible signal. Handlers running past TIMBAL_BLOCKING_WARN_MS (default 100ms) now log a one-time actionable warning (make it async, or set offload_blocking=True). Costs one perf_counter pair per sync call; probe unchanged (386 us/run). Co-authored-by: Cursor <cursoragent@cursor.com>
A top-level agent awaited inside a plain tool handler swaps in a fresh RunContext (concurrent-sibling logic) via set_run_context, which the _stream finally never restored. Historically masked because agent tools and workflow steps ran in tasks with copied contexts; the direct in-task iteration fast paths removed that isolation, so the swap leaked into the caller's run and its next LLM call failed against the wrong trace. Nested invocations (entry call id set) now restore the caller's run context on exit; top-level invocations keep leaving theirs set for implicit same-task session chaining. _restore_context also compares run-context identity, not just call id. Caught by the delegation correctness gates in the openai_agents/google_adk benchmarks; regression test added. Co-authored-by: Cursor <cursoragent@cursor.com>
Full-mode reruns of all 20 benchmarks across crewai, agno, pydantic, openai_agents, and google_adk against the optimized framework. Timbal agent p50 drops from ~570-980us to ~220-310us across every suite; burst p50 drops from 9-17ms to ~230-240us (runs complete with almost no forced suspensions). Headline agent-loop p50 vs competitors (same run, same machine): - CrewAI 1.13: 309us vs 8.7ms (was 978us vs 3.2ms) - Agno (current): 277us vs 656us (agno itself improved since last run) - PydanticAI: 279us vs 3.1ms - OpenAI Agents: 287us vs 1.6ms - Google ADK: 223us vs 2.4ms Notes on reproduction environments: - crewai runs on isolated Python 3.12 (its pinned instructor -> pydantic-core 2.33.2 has no cp314 wheel); Timbal+CrewAI numbers in that file share the interpreter, so the comparison stays internally consistent. - pydantic graph benches run with pydantic-ai<1 (the scripts target the pre-1.0 Graph API; porting them to the new API is future work). - other frameworks run at current releases, so their own numbers moved too. The delegation correctness gates in openai_agents/google_adk caught a real run-context leak fixed in the previous commit. Co-authored-by: Cursor <cursoragent@cursor.com>
Bugbot finding: _latency accumulated in-memory traces across timed iterations (unlike _memory), skewing later samples against Timbal since LangGraph accumulates nothing. Clear outside the timed window per iteration; results refreshed (Timbal p50s improve ~5-12%). Co-authored-by: Cursor <cursoragent@cursor.com>
Cancelling an agent mid-tool ended the run as 'success': the tool's _stream swallows the task's CancelledError (recording its own span interrupted), and since the single-tool fast path iterates the tool on the agent's task, the agent never saw the cancellation and kept looping. Same fix as the linear workflow path: detect current_task.cancelling() and re-raise before forwarding post-cancel events (preserves LLM-output salvage). Also applied to the command tool path, where direct iteration predates this branch. Caught by the integration suite (test_key_agent_interruptions, deselected from default runs); offline twin regression tests added so the default suite covers it. Also force-adds the missing test.jsonl fixture — the blanket *.jsonl gitignore rule silently excluded it, so test_files_integration::test_jsonl always failed on its local variant (pre-existing on main). Co-authored-by: Cursor <cursoragent@cursor.com>
warmup_voice_stack ran at boot for every Agent app whenever the timbal[voice] extra was installed (e.g. platform images built from timbal[all]), downloading and loading Smart Turn + Namo + Silero ONNX models for deployments that never touch voice. New voice_warmup_intended() gate: warm up only when the runnable declares voice_config, a TIMBAL_VOICE_* / ELEVENLABS_VOICE_ID env is set, or TIMBAL_VOICE_WARMUP is explicitly truthy (the playground launcher sets it for its child servers so first-Start Smart Turn stays instant; it can also force-disable warmup for voice apps). Unit tests cover all gate paths. Co-authored-by: Cursor <cursoragent@cursor.com>
PR SummaryMedium Risk Overview LLM dispatch is reorganized from a single Docs and benchmarks: Reviewed by Cursor Bugbot for commit 38af6be. Bugbot is set up for automated code reviews on this repo. Configure here. |
…28s -> 0.39s openai + anthropic were imported at llm_router/fallback_model module level (~460ms, a third of the agent import) just for 2 client classes and 8 exception types. Clients now import inside _resolve_client (sys.modules hit after first call); error-classification tuples build lazily in the new core/provider_errors.py — both consumers only classify on LLM error paths, where the raising SDK is already imported. from timbal import Agent: 1,277ms -> 389ms, peak RSS 72 -> 55MB, modules loaded 1,641 -> 536. Neither SDK imports until the first LLM call. Verified against the real Anthropic API (lazy client resolution end-to-end). Co-authored-by: Cursor <cursoragent@cursor.com>
…port openai Two remaining spots loaded both SDKs regardless of provider: - provider_error_classes() imported openai+anthropic to build classification tuples. An exception can only be an instance of a class from an SDK that is already imported (it raised it), so tuples now build from sys.modules presence only — possibly empty for TestModel-only processes. Rebuilt per call (attribute lookups, error paths only) so mixed fallback chains stay correct when a second provider joins later. - the collector registry imported all impls on first use; the anthropic and openai collector modules import their SDK types at module level. The registry now loads SDK-free collectors (string, timbal) on first use and provider collectors via a lazy_loader hook keyed on the chunk's module prefix — the chunk's class lives in its SDK's module tree, so provider detection needs no import. Subprocess-isolated tests assert: Agent import loads neither SDK, a full TestModel agent run loads neither, anthropic client resolution loads only anthropic (and vice versa), and error classification works with one or zero SDKs present. Verified with a real anthropic/claude-haiku run: openai never imports. Co-authored-by: Cursor <cursoragent@cursor.com>
The sonnet-4-6 runs flaked intermittently on content assertions (answers like 'What's Alice's score?' phrased without the expected literal). All 18 sonnet-5 variants pass. Offline sonnet-4-6 references (dispatch/collector fixtures) untouched — they never hit the API. Co-authored-by: Cursor <cursoragent@cursor.com>
Transformer subcommands registered their argparse parsers by importing every transformer module, each of which imports libcst at module level (~160ms). The guard was argv string-sniffing against a lightweight-ops set, so --help (and any op not in that set) paid the full cost. Now a static op table (cli-name -> module, help line) provides stub parsers for help listings, and only the operation actually invoked imports its module and registers the full parser. A pre-parser that knows the global flags extracts the requested op, so '--path some-dir add-mcp' can't be misread. The fragile argv sniffing is gone. Tests pin: the table stays in sync with modules on disk, top-level --help lists every op without importing libcst, and 'add-mcp --help' still renders the full parser. Co-authored-by: Cursor <cursoragent@cursor.com>
sync_to_async_gen used None as its end-of-stream sentinel, so a sync generator handler yielding None mid-stream silently dropped every chunk after it. Replaced with a unique sentinel object (and hoisted the per-iteration closure allocations). The final output now retains all chunks including the None; the None chunk still isn't surfaced as a streaming delta (collector protocol treats process()->None as skip — a cosmetic quirk, documented in the regression test). Also removes the one dead import a vulture scan found (ast.literal_eval in tool_use.py); the scan was otherwise clean across core/state/types/ collectors. Co-authored-by: Cursor <cursoragent@cursor.com>
…ation counter The command fast path's enumerate() reused 'i', shadowing the agent-loop iteration counter in the enclosing scope. Harmless today only because the command branch always returns; any future fall-through would silently corrupt max_iter accounting. Renamed to arg_idx with a guard comment. Also drops the unknown pytest.mark.timeout from the lazy-SDK tests (the plugin isn't installed, so the mark was inert and produced the suite's one warning; subprocess.run timeouts already cover it). Co-authored-by: Cursor <cursoragent@cursor.com>
…ones — net -187 lines Nine codegen transformers carried private copies of the same machinery (import insertion x6, insert-before-assignment x5, step-call matching x5, the add-tool/add-mcp validation preamble x2). Extracted into cst_utils: is_step_call, step_matches_target, parse_call_statement, parse_function_def, insert_imports, insert_before_assignments, validate_tools_target, merge_config_kwargs, assignment_resolves_to, and a StepCallRewriter base class owning leave_Expr for add-edge/remove-edge/set-param. evals: seq!/parallel! now share span-timing helpers (_spans.py); email!/json! share the bool-expectation scaffold on BaseValidator. Error messages are byte-identical (label param covers the wording difference). Zero behavior change: 510 codegen + 36 evals tests pass; codegen and evals are now clone-free at the 8-line threshold (pylint duplicate-code). Co-authored-by: Cursor <cursoragent@cursor.com>
…er concern, zero perf cost llm_router.py was a 645-line module with F-grade complexity mixing the provider registry, client cache/resolution, retry loop, and three inlined API branches. Now core/llm/ has one module per concern: router.py _llm_router dispatch (F -> D complexity) registry.py _ProviderConfig + _PROVIDERS (add a provider here) clients.py client cache, key/platform-proxy resolution, warmup retry.py transient-failure retry loop messages.py Anthropic Messages API responses.py OpenAI Responses API (openai, xai) chat_completions.py Chat Completions + compatible providers API modules build request kwargs once per request and return the stream factory consumed by the single shared retry loop, so the streaming path keeps the exact same generator nesting: 33-35us/request before and after, import time unchanged (~170ms), provider SDKs still lazy. Callers (age, memory_compaction, voice, audit script) now import from timbal.core.llm; test monkeypatch targets moved to the defining modules (llm.clients._get_client, llm.router.TIMBAL_OPENAI_API, llm.retry.random.uniform). Full suite: 3040 passed, unchanged.
Conflicts were codegen-only: main's transformer hardening (PR #113 — annotated assignments, aliased/module-qualified entry points, loud no-op failures) landed on the same files our dedupe refactor consolidated. Resolution keeps the shared-helper structure and re-expresses main's behavior through it: - insert_before_assignments grew AnnAssign anchors and an optional step_calls_of anchor — one change instead of main's five copies - assignment_resolves_to accepts AnnAssign; StepCallRewriter sets matched - allow_noop / matched / leave_AnnAssign / has_step_expr validation applied per transformer exactly as on main All 505 codegen tests (incl. main's new coverage and the full suite (3080) pass. EOF ) Co-authored-by: Cursor <cursoragent@cursor.com>
…ests out of tests/core
…che reads cost 0.1x
Send top-level cache_control on every Messages request (opt out with
model_params={"cache_control": None}). Count cache_creation/cache_read
input tokens toward compaction utilization, which otherwise sees a
nearly-full cached context as ~5% used and never compacts.
…e provider switches Merge provider_params["tools"] with generated client tools instead of clobbering them in all three API adapters. Skip provider-internal server-tool blocks (server_tool_use, web_search_tool_result) when replaying Anthropic memory to OpenAI APIs instead of raising.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7102569. Configure here.
An assistant message whose content is only server-side tool blocks (or
thinking with reasoning_as="omit") serialized to a bare {"role": "assistant"}
dict, which OpenAI-shaped APIs reject. Return None and skip the turn in the
chat completions adapter. Also mark the platform proxy tests as integration
so they do not join the offline suite once the env file exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
pedrolivaresanchez
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.