feat: agent mode tool-calling via Vercel AI SDK#451
Merged
Conversation
Transform Agent Mode from text-only chat into a full agentic experience with native tool/function calling. The agent can now search notes, copy to clipboard, search the web (via cloud API), and check calendar events. - Tool registry with OpenAI-compatible function calling format - ReAct execution loop with parallel read-only tool execution - SSE streaming with incremental tool call argument accumulation - Inline tool execution UI (compact pills with status animations) - Text input field alongside voice input for tool-heavy workflows - Dynamic system prompt with tool usage instructions - IPC handler for web search via OpenWhispr cloud API - Database migration for tool message metadata - i18n strings for all 10 supported locales
Replace manual SSE parsing and AgentLoop with AI SDK streamText + stepCountIs for tool-calling agent mode. Add unified provider factory supporting OpenAI, Groq, Anthropic, Gemini, and custom endpoints. - Add ai, @ai-sdk/openai, @ai-sdk/groq, @ai-sdk/anthropic, @ai-sdk/google - Add src/services/ai/providers.ts with getAIModel factory - Add ToolRegistry.toAISDKFormat() using jsonSchema wrapper - Add ReasoningService.processTextStreamingAI() with full tool support - Remove AgentLoop.ts (replaced by stepCountIs) - Remove dead toOpenAIFormat/OpenAIFunctionTool from ToolRegistry - Simplify AgentOverlay to 3 streaming paths (tools/cloud/BYOK)
…isableThinking Uses AI SDK's providerOptions API to send reasoning_effort: "none" to Groq for models flagged with disableThinking in the model registry.
- Add mounted ref to guard state updates after overlay unmount - Add AudioManager.cleanup() call on unmount - Handle tool-result stream chunks to show actual results in UI - Set tool status to "executing" until result arrives (fix state thrashing) - Add try/catch in ToolRegistry.toAISDKFormat() execute wrapper - Extend AgentStreamChunk type with tool_result variant - Add success field to IPC web-search response for contract consistency
- Add get_note tool to fetch full note content by ID - Add create_note tool with folder resolution and cloud sync - Add update_note tool for title, content, and folder changes - Add shared resolveFolderId utility for folder name lookup - Include note ID in search_notes results for cross-tool reference - Register tools and add system prompt instructions - Add translation keys for all 10 locales
- Enable tool calling for cloud agent mode (clipboard, notes, web search, calendar) - Implement NDJSON streaming via IPC batch approach for reliable event delivery - Add multi-step tool-calling loop with AI SDK v6 message format - Fix mountedRef StrictMode bug causing empty renders - Return actual tool result data to LLM instead of just display text - Extract MAX_TOOL_STEPS constant, remove redundant comments
When cloud backup is enabled and user is signed in, the search_notes agent tool now uses the cloud hybrid search (pgvector + FTS) instead of local SQLite FTS5 keyword search. Falls back to local search transparently on cloud failure.
Replace the buffered IPC middleman with direct fetch from the renderer to the API. Text now streams token-by-token instead of arriving all at once after the full response completes. Adds AbortController support for instant cancellation on unmount or user stop.
The renderer's fetch() can't access auth cookies stored on the Neon Auth domain. Add a get-session-cookies IPC handler to retrieve them from the main process cookie jar and forward as an explicit Cookie header.
Browser fetch() forbids setting the Cookie header, so direct renderer-to- API streaming can't authenticate. Switch to event-based IPC: main process reads the API stream and forwards each chunk via webContents.send() as it arrives. This matches the pattern used for AssemblyAI/Deepgram streaming.
Replace basic tool pills with step-based visualization showing the full tool lifecycle: shimmer accent while executing, checkmark pop on completion, expandable detail, and contextual clipboard confirmation. - ToolCallStep: left-border accent, per-tool icons, shimmer animation - Clipboard: inline "Copied to your clipboard" with green check - Input bar: thinking shimmer bar, tool icon in executing state - Empty state: mic icon with dual-line CTA - Title bar: shows agent name, softer shadow - Tighter spacing throughout (gaps, heights, bubble widths) - New CSS: tool-step-shimmer, tool-check-pop, tool-status-sweep - i18n: copiedToClipboard + orType in all 10 locales
…ults
The executeToolCall callback was returning raw data (full JSON) which was
displayed in the tool step UI. Now returns { data, displayText } so the
LLM gets structured data while the UI shows a human-readable summary.
Also truncates Exa web search article text to 500 chars per result to
prevent massive payloads.
- Increase MAX_TOOL_STEPS from 5 to 20 to prevent agent from getting stuck on multi-step workflows - Add metadata field to ToolCallInfo so tool results can carry structured data (e.g. note ID) to the UI without mixing it into displayText - Created/updated/fetched notes show as clickable steps with a primary accent — clicking opens the note in the control panel - New agent-open-note IPC handler navigates the control panel to the note
- Note cards: when create/update/get_note completes, render a compact clickable card at the bottom of the message bubble with title, icon, and "Open note" label. Clicking opens the note in the control panel. - Input bar: replace generic loading dots with dictation-panel-inspired indicators — pulsing blue circle for listening, accent wave bars for transcribing, shimmer sweep for thinking. - Fix duplicate copiedToClipboard keys in all locale files.
…mode The agent-open-note IPC was reusing navigate-to-meeting-note which activates meeting recording mode. Add a dedicated navigate-to-note event that only sets the active note and view without starting the recorder.
Resolved conflict in preload.js — kept both onNavigateToNote (feature branch) and update notification APIs (main).
# Conflicts: # src/types/electron.ts
- Add AbortController with 60s timeout to streaming fetch in ReasoningService (prevents hanging on network failure) - Wire cancelActiveStream() to abort the streaming controller - Add isProcessing guard to processWithAnthropic and processWithLocal to prevent concurrent request overlap - Wrap JSON.parse(argsJson) in try/catch in agent tool executor to gracefully handle malformed LLM tool arguments - Fix calendar "tomorrow" window: compute minutes until end-of-tomorrow instead of hardcoded 2880 (48h) which over-fetched into the day after - Replace hardcoded English strings in agent chat with i18n t() calls: "New conversation", "Unknown tool:", "Error:" prefix - Add unknownTool and invalidArgs translation keys to all 10 locales - Translate pricing feature arrays in all 9 non-English locales - Translate enterprise subtitle and CTA in all 9 non-English locales - Fix dark mode: replace raw bg-emerald-100/text-emerald-600 with opacity-based bg-green-500/10 and text-green-500 in OnboardingFlow
- Remove duplicate ToolCallEntry from AgentOverlay, import Message/ToolCallInfo from AgentChat - Thread metadata through addAgentMessage (database → IPC → preload → types → call sites) - Add StreamMessage type for tool role + array content at API boundary - Memoize tool registry via ref, invalidate on settings change or new chat
…ission checks IntegrationsView and UpcomingMeetings were checking systemAudio.status === "denied" which missed "not-determined" and "unknown" states — causing the permission gate to be skipped when it shouldn't be. Now uses mode === "native" && !granted, matching the pattern already used in SettingsPage and PermissionsSection. Also changed both flows to call systemAudio.request() (native consent dialog) instead of systemAudio.openSettings (jumps straight to System Settings).
…apid delete/switch Move next-note selection into removeNote (uses fresh getState), flush pending debounced saves on note switch, use refs instead of stale closures in title/content change handlers, and clear local editor state on note deletion.
Adds offline semantic search using a local Qdrant vector DB sidecar and MiniLM ONNX embeddings (384-dim). Notes are indexed on create/update/delete and searched via Reciprocal Rank Fusion (FTS5 + vector). Gracefully degrades to FTS5-only when Qdrant is unavailable. - Qdrant binary management: download script + process lifecycle manager - Local embedding pipeline: WordPiece tokenizer + ONNX inference - Vector index: Qdrant collection CRUD + batch reindex - Hybrid search: RRF merge with 0.3 cosine threshold filter - Search tool: three-tier fallback (cloud → local semantic → FTS5) - Settings toggle in Privacy & Data with i18n (10 languages) - .env persistence for LOCAL_SEMANTIC_SEARCH across restarts
Add architecture docs, dev setup instructions, troubleshooting guide, settings reference, and testing checklist for the Qdrant sidecar and MiniLM embedding pipeline.
Semantic search is no longer optional — Qdrant starts unconditionally on app launch if the binary is available. Embedding model auto-downloads on first run. Qdrant binary downloads automatically in predev/prestart. Removes: settings toggle, localSemanticSearchEnabled setting, i18n keys, enable/disable IPC handlers, LOCAL_SEMANTIC_SEARCH env var.
- Add qdrant-* to extraResources filter so binary is bundled in release - Add onnxruntime-node to asarUnpack (native module needs unpacking) - Fix localEmbeddings.downloadModel() to use runtime downloadUtils.js instead of build-time scripts/lib/download-utils (not packaged in ASAR)
Add download-qdrant.js step to linux, windows, and macOS jobs in both build-and-notarize.yml and release.yml. Passes --platform/--arch for macOS cross-compilation (same pattern as whisper-cpp and sherpa-onnx).
processTranscription() was missing the skipReasoning check, so BYOK transcription paths (local whisper, parakeet, OpenAI API, Mistral) still ran LLM cleanup before sending text to the agent chat. Only the OpenWhispr Cloud and streaming paths had the guard. Add early return in processTranscription() so agent mode always gets raw transcription.
…chips Enable browser-level echo cancellation (AEC) on the meeting microphone stream so macOS CoreAudio subtracts speaker output from mic input, eliminating duplicate transcription when not using headphones. Add metadata chips to the note editor showing date, linked calendar event, folder, and meeting type. Add gcalGetEvent IPC to fetch calendar event details for display. Update CLAUDE.md and SECURITY.md to reflect current API key storage (plaintext, not yet encrypted — tracked in #532).
Make the folder metadata chip clickable with a dropdown picker matching the sidebar's move-to-folder UX. Remove parchment background from light mode inputs, align folder picker input styles across both dropdowns, and bump export button contrast.
Add generation counter to initializeNotes so stale IPC responses from earlier folder clicks are discarded. Clear active note immediately on folder switch and double-guard post-load note selection. Fix title flicker by using note's own data on the first render before local state syncs. Await save flush when switching notes to prevent data races. Fix stale closure in handleDeleteFolder by reading fresh folder data.
…ments Guard against empty OpenAI Realtime deltas firing onPartialTranscript repeatedly, and skip onFinalTranscript when the completed transcript is empty to avoid re-sending the previous segment.
Skip re-filtering the note list by folder during meeting mode so the active note stays visible and the recording continues uninterrupted.
Make search input border transparent with stronger outline, and bump metadata button/text opacity for better readability in both themes.
…rding (#530) Meeting transcription always used the cloud OpenAI Realtime pipeline, ignoring the user's local transcription setting. When local mode was selected (Parakeet or Whisper), the app tried to authenticate with the cloud and failed with "No session cookies available." Route meeting transcription through local engines when useLocalWhisper is enabled. Buffer 5-second audio chunks, downsample from 24kHz to 16kHz for local engine compatibility, and gate on RMS energy to prevent Whisper hallucinations on silent chunks.
Use kAudioObjectPropertyScopeGlobal instead of kAudioObjectPropertyScopeInput for CoreAudio property listeners in macos-mic-listener. The input scope was not receiving property change notifications when browsers like Chrome activate the mic for Google Meet. Add a 5s heartbeat poll as safety net. Reset hasPrompted on the audio activity detector when exiting meeting mode so future detections are not silently blocked. Add diagnostic logging to suppression paths in _onMicStateChanged.
…alog Redesign ActionManagerDialog with master-detail two-panel layout for better prompt editing UX. Add Actions button to notes sidebar for quick access. Add i18n keys across all 10 locales.
electron-builder v26 requires publisherName in azureSignOptions. Added as env var to config and both CI workflows.
Standard platform convention: ⌘, on macOS (Apple HIG), Ctrl+, on Windows/Linux. Adds "Settings..." menu item with accelerator to app menu (macOS) and File menu (Windows/Linux), IPC bridge to renderer, and fallback keydown listener alongside existing ⌘K handler.
…r download The PR build step was failing because azureSignOptions in electron-builder.json triggers Azure Trusted Signing even without auth env vars. Merged the two conditional build steps into one that always passes signing credentials. Also adds the missing windows-mic-listener download step to both CI workflows (was in prebuild:win but missing from CI), and fixes inconsistent GITHUB_TOKEN fallback on windows-fast-paste and windows-key-listener download steps.
Remove the PermissionsGate that blocked the entire UI when microphone
permission wasn't granted. Users can now browse notes, settings, and
use system-audio-only recording without mic access. Mic permission is
still requested at point-of-use (recording, device enumeration) with
clear error toasts when denied.
Fix system audio permission detection on macOS by replacing the broken
systemPreferences.getMediaAccessStatus("screen") check (which queried
screen-capture TCC, not system-audio TCC) with a cached status from
AudioTapManager.checkAccess(). The status resolves after the user
clicks "Grant Access" or completes a recording session, without
triggering unexpected system dialogs on passive checks.
One-way write-through from SQLite to .md files on disk, organized by folder. Includes settings toggle, custom save location picker, rebuild button, and "Show in Finder/Explorer/Files" context menu items on notes and folders. Off by default; first enable triggers full rebuild.
electron-builder's ${env.} interpolation is not supported for
azureSignOptions fields, causing literal strings to be passed to
Invoke-TrustedSigning instead of resolved values. Hardcode the
non-secret resource identifiers directly in the config and remove
the now-unnecessary env var passthrough from workflows.
Add responsive layout to the settings dialog that adapts when the window is narrow (< 800px), such as during meeting recorder mode at 1/3 screen width. - SidebarModal: collapse sidebar from 208px to 48px icon-only rail using ResizeObserver with callback ref (for Radix Portal compat), propagate compact state via SettingsLayoutContext - SettingsRow: stack vertically in compact mode (label above control) - SettingsPanelRow: reduce padding in compact mode - Plans grid: switch from 4 to 2 columns in compact mode - DeveloperSection: switch grid from 2 to 1 column in compact mode - ReasoningModelSelector: add break-all on URL code blocks - NoteBottomBar: use primary color tokens for recorder controls
Probe the native audio tap binary once during deferred init to cache
the real TCC status for the session. Uses getMediaAccessStatus("screen")
as a guard — only probes when the user has been asked before (not
"not-determined"), so first-time users never see an unexpected consent
dialog. Fixes the card showing "Grant Access" after restart even when
permission was already granted.
Remove the getMediaAccessStatus("screen") guard from resolvePermission
— it checks kTCCServiceScreenCapture (wrong TCC entry), not
kTCCServiceAudioCapture which process taps actually use. The macOS
consent dialog only appears on the very first call to
AudioHardwareCreateProcessTap; subsequent calls are always silent.
Trim excess comments from IPC handler.
The marker file + 3 helper methods protected against a non-problem: new users seeing the macOS consent dialog slightly before onboarding. This is standard macOS behavior (Zoom, Discord, Chrome all do it). Revert to a simple probe on startup.
… probe Replace the startup probe (which could trigger the macOS consent dialog without user action) with file-based persistence in userData. The permission status is written to .system-audio-permission when the user explicitly clicks "Grant Access" or when a recording resolves the status. On next launch, the constructor reads the cached value — no probe, no dialog, correct UI state immediately. Follows the same persistence pattern as mic permission (renderer uses localStorage, main process uses userData file).
Remove PermissionsGate.tsx — no longer imported after the gate was removed from main.jsx. Simplify requestAccess catch handler.
|
Size alert: due to this MR, the 1.6.7 Linux tarball nearly doubles (185MB → 426MB). Major additions are bundled CUDA ONNX Runtime libs (libonnxruntime_providers_cuda.so ~316MB, libonnxruntime.so.1 35MB) plus qdrant-linux-x64 96MB. Was this really necessary ? |
charian47
pushed a commit
to charian47/openwhispr
that referenced
this pull request
Apr 25, 2026
…ling feat: agent mode tool-calling via Vercel AI SDK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full agent mode implementation with Vercel AI SDK tool-calling, embedded chat, local semantic search, and comprehensive UX polish.
Agent & Chat System
Local Semantic Search
Notes & Meeting Improvements
Linux & Platform Fixes
UI Polish