Skip to content

Whisper / alternative STT models for much better speech recognition #4

Description

@grajat90

Summary

Add higher-accuracy speech-to-text backends behind the existing core.Transcriber interface (internal/core/interfaces.go), so users can opt into much better recognition than the default.

Context

  • STT today is native-first: SpeechAnalyzer/SpeechTranscriber in the Swift helper (macos-helper/Sources/Helper/Capture/LiveSpeechPipeline.swift), adapted to Go via internal/stt.
  • DESIGN.md/CLAUDE.md already anticipate whisper.cpp as an optional post-v1 fallback behind the same interface.

Scope

  • Integrate whisper.cpp (and leave room for other models) as an alternative transcriber, selectable via internal/config like the distiller/answerer providers.
  • Keep the hard invariant: audio never crosses the IPC boundary; any whisper backend that needs audio must run inside the helper (macos-helper/), emitting only Utterance text to the core.
  • Model management: download/select model size, document accuracy vs latency/resource tradeoffs.
  • Benchmark accuracy against the native transcriber on a fixed fixture (test/fixtures/speech.wav) so the win is measurable.

Open questions

  • whisper.cpp in-helper (Swift bridging) vs a separate on-device process; both must keep audio local.
  • Streaming vs chunked transcription and its effect on segmentation latency.

Related: multilingual support (separate issue) largely rides on model choice.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions