# this can be use to run on local model from ollama
APEX_FORCE_OFFLINE=1 swift run
A macOS native voice agent built in Swift. Local LLM inference. Online/offline fallback. 100% terminal-built. One week, hardcore mode.
Demo video: Watch the demo

Phase 1 & 2: Command router + local LLM (Ollama/qwen3:8b) understanding natural language.
ApexAgent is a personal AI assistant for macOS — think Jarvis, but running entirely on your machine. It listens to your voice, understands natural language, controls your Mac, and speaks back. No cloud dependency required.
Built from scratch in Swift. No Xcode. No tutorials. Just vim + swift build + swift run.
┌─────────────────────────────────────────────────────────┐
│ ApexAgent │
│ │
│ Voice Input (STT) │
│ ├─ Online: Deepgram Flux (streaming WebSocket) │
│ └─ Offline: Apple Speech framework (on-device) │
│ ↓ │
│ Intent Understanding (LLM) │
│ ├─ Online: GLM API (cloud) │
│ └─ Offline: Ollama + Qwen3 8B (local) │
│ ↓ │
│ Tool Execution │
│ ├─ Process (shell commands: open, quit, volume) │
│ └─ Accessibility API (click UI elements in apps) │
│ ↓ │
│ Voice Output (TTS) │
│ ├─ Online: Deepgram Aura-2 (natural voice) │
│ └─ Offline: AVSpeechSynthesizer (built-in) │
└─────────────────────────────────────────────────────────┘
The agent auto-detects network status. Online → cloud APIs (smarter, faster). Offline → local models (private, no internet needed). The user never notices the switch.
| Day | Phase | Goal | Done When |
|---|---|---|---|
| 1 (Thu) | Command Router | Terminal REPL with hardcoded commands | open spotify opens Spotify |
| 2 (Fri) | Local LLM | Ollama integration, natural language routing | "crank up the volume" → volume 80 |
| 3 (Sat) | Online/Offline Fallback | GLM + Ollama auto-switch | Kill wifi → still works |
| 4 (Sun) | Voice Input (STT) | Apple Speech + Deepgram | Speak "open spotify" → it opens |
| 5 (Mon) | Voice Output (TTS) | AVSpeech + Deepgram Aura-2 | Agent speaks back |
| 6 (Tue) | macOS UI | SwiftUI menu bar chat app | Not a terminal anymore |
| 7 (Wed) | Polish + Demo | Bug fix, record demo video | Portfolio-ready |
- Optionals (
?!if let) — the core Swift concept, no Python equivalent - Structs vs Classes — value types vs reference types, when to use each
- Enums with associated values — Swift's superpower, Python has nothing like this
- Protocols — like Python ABCs but enforced at compile time
- Closures — Swift's lambdas, trailing closure syntax
- Error handling (
try/catch/throw) — different from Python's try/except - Async/await — concurrency, continuations, actors
- String interpolation (
\(variable)) — like f-strings but different syntax
- Arrays —
map,filter,forEach,contains - Dictionaries —
Dictionary(uniqueKeysWithValues:), optional value access - Sets — fast O(1) lookup
- Tuples — fixed-size, named elements
- Custom structs — building your own data containers
- Enums as state machines — modeling intent with associated values
- Process — run shell commands from Swift (
open,osascript,pmset) - SwiftUI — declarative UI:
View,@State,@Binding,VStack,HStack - AppKit integration — menu bar, window lifecycle,
MenuBarExtra - AVFoundation —
AVAudioEnginefor mic capture,AVAudioPlayerfor playback - AVSpeechSynthesizer — built-in text-to-speech, no API key needed
- Speech framework —
SFSpeechRecognizer, on-device streaming STT - Network framework —
NWPathMonitorfor online/offline detection - Accessibility API — read and click UI elements in other apps
- URLSession — HTTP POST/GET to Ollama and GLM APIs
- JSONDecoder/Encoder — parse LLM JSON responses
- WebSocket — streaming audio to Deepgram Flux
- Async networking —
async/awaitwithURLSession.data - Protocol-oriented design —
LLMProviderprotocol, swap implementations at runtime
- Local LLM inference — Ollama, model quantization, memory budget on M1 Pro
- Prompt engineering — system prompts for JSON intent extraction
- Tool calling — LLM picks a tool from a registry, fills arguments, executes
- Two-mode architecture — online (cloud) vs offline (local), graceful degradation
- Voice pipeline — STT → LLM → tool → TTS, the full loop
- Streaming responses — SSE-style partial results from both STT and LLM
- Swift Package Manager —
Package.swift, dependencies, targets - Terminal-first workflow —
vim+swift build+swift run, no IDE - Git discipline — daily commits, one feature per commit
- Protocol-based architecture —
LLMProvider,STTProvider,TTSProvider - Error handling at boundaries — user input, network, audio, shell commands
- TCC permissions — microphone, accessibility, automation consent
| Component | Technology | Cost |
|---|---|---|
| Language | Swift 6 | Free |
| Build | Swift Package Manager | Free |
| Editor | Vim | Free |
| Local LLM | Ollama + Qwen3 8B | Free |
| Cloud LLM | GLM API (DeepSeek-compatible) | Cheap |
| Online STT | Deepgram Flux | Free $200 credit |
| Offline STT | Apple Speech framework | Free |
| Online TTS | Deepgram Aura-2 | Free $200 credit |
| Offline TTS | AVSpeechSynthesizer | Free |
| Vector store (future) | ChromaDB / MLX | Free |
- Machine: MacBook Pro M1 Pro (32GB unified memory, 16 cores)
- Memory bandwidth: 200 GB/s
- Local LLM budget: ~24GB usable for model + context
- Metal support: Metal 4
ApexAgent/
├── Package.swift # SPM manifest
├── Sources/
│ └── ApexAgent/
│ ├── ApexAgent.swift # entry point
│ ├── (future files...)
│ │ ├── CommandRouter.swift # Phase 1 — parse + route commands
│ │ ├── ShellRunner.swift # Phase 1 — Process() wrapper
│ │ ├── OllamaProvider.swift # Phase 2 — local LLM
│ │ ├── GLMProvider.swift # Phase 3 — cloud LLM
│ │ ├── LLMProvider.swift # Phase 3 — protocol
│ │ ├── NetworkMonitor.swift # Phase 3 — online/offline
│ │ ├── AudioRecorder.swift # Phase 4 — mic capture
│ │ ├── SpeechProvider.swift # Phase 4 — Apple STT
│ │ ├── DeepgramSTT.swift # Phase 4 — Deepgram STT
│ │ ├── SpeechSynthesizer.swift # Phase 5 — AVSpeech TTS
│ │ ├── DeepgramTTS.swift # Phase 5 — Deepgram TTS
│ │ └── ContentView.swift # Phase 6 — SwiftUI
├── Tests/
│ └── ApexAgentTests/
└── README.md
- Computah (
~/degree/PersonalAgentic/computah/) — Swift macOS notch app with Deepgram + Jev + Accessibility API. Used as reference for voice, LLM calls, and app structure. Not forked — patterns are studied and reimplemented.
No tutorials. No courses. No "let me watch one more video."
Write code → build → fix errors → repeat.
The compiler is your teacher.
The terminal is your classroom.
The deadline is your motivator.
Nik (@nickaqmal02) — Final-year CS student, aspiring AI Engineer. Building this to learn native macOS development and local LLM integration in one focused week.
MIT