You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Friday is not a chatbot. It is a fully autonomous desktop AI agent that lives on your Windows machine, watches your screen, listens to your voice, understands your goals, and takes action — [...]
Think of it as having a real-world version of Tony Stark's FRIDAY running on your PC. It monitors what you're doing, proactively comments and helps, controls your entire desktop via voice, manages[...]
What makes Friday different from other agents: she learns who you are. Import your conversation history from Claude, ChatGPT, or Gemini and Friday extracts your preferences, communication styl[...]
Friday is open source, Windows-native, self-hosted, and built to eventually compete with commercial agents like Devin, Cline, and Claude Code.
Feature Overview
🎙️ Voice & Audio
Feature
Status
Details
Gemini Live native audio
✅ Working
Primary voice interface via friday_live.py
Groq Whisper STT
✅ Working
Fallback speech-to-text
Porcupine wake word
✅ Working
"Hey Friday" detection via voice_wake.py
Multi-TTS engine support
🔧 In Progress
ElevenLabs, Google TTS, pyttsx3
Typing effect output
✅ Working
Terminal typing effect for responses
Audio crackling fix
✅ Fixed
Jitter buffer increased to 12 chunks (~2.4s), smoother drain, no discontinuity
👁️ Vision & Screen Awareness
Feature
Status
Details
Live API native video feed
✅ Working
Automatic ~1 FPS, 720p streaming — model sees screen without calling any tool
Active window detection
✅ Working
screen_watcher.py via pywinctl
Gemini Vision analysis
✅ Working
Screen-to-description pipeline
Vision-grounded clicking
✅ Working
vision_click() — finds elements by description
Background monitor with proactivity
✅ Working
Unsolicited observations via Gemini proactivity
Error detection on screen
🔧 In Progress
Detects visible Python/JS errors
Visual search
✅ Working
"Friday find the word X on screen"
🖥️ Desktop Automation & Control
Feature
Status
Details
Mouse & keyboard control
✅ Working
pyautogui-based via friday/tools.py
App launching & closing
✅ Working
open_app(), close_app() — system discovery chain
Spotify control
✅ Working (Web API + keyboard fallback)
Full Spotify API — search, play, queue, volume
Netflix/streaming control
🔧 In Progress
Vision-based navigation
File system access
✅ Working
With run_cmd + system access
Roblox launcher
✅ Working
Web search place ID → roblox:// URI
Microsoft Store launcher
✅ Working
ms-windows-store:// URI
Windows Clock (alarms, timers, stopwatch, reminders, focus)
The installer checks prerequisites (Python, Git, Node.js), creates a .venv, installs dependencies, runs an interactive onboarding wizard for all API keys (OpenCode Zen, NVIDIA, Google, Groq, GitHub, Telegram, Discord, OSINT, Social Media), creates friday_memory/, adds friday to PATH, and verifies all modules.
After install:
friday
That's it. FRIDAY starts the dashboard, API, sidecar heartbeat, memory checks, and live voice engine with structured terminal output.
First-time setup: The onboarding wizard walks you through getting your OPENCODE_ZEN_API_KEY (free at opencode.ai), which is the primary LLM brain. All other keys are optional and can be added later via the dashboard or by editing .env.
Usage
friday # Full supervisor: dashboard + voice engine
friday dashboard start # Start dashboard services (background)
friday dashboard stop # Stop dashboard services
friday dashboard status # Dashboard health + URLs + latency
friday dashboard open # Open dashboard in browser
friday status # System health: dashboard, sidecar, memory
friday doctor # Diagnostics: modules, ports, configs
friday doctor --verbose # Full diagnostic report
friday fix # Auto-repair common issues
friday memory-tree read people # Read from memory tree
friday sidecar status # Sidecar network status
Environment Variables
Variable
Required
Purpose
GOOGLE_API_KEY
Yes
Gemini Live (brain + vision)
OPENCODE_ZEN_API_KEY
Yes
OpenCode Zen API — primary LLM (free, unlimited tokens, big-pickle model)
PICOVOICE_ACCESS_KEY
No
Wake word detection
SPOTIFY_CLIENT_ID
No
Spotify control
SPOTIFY_CLIENT_SECRET
No
Spotify auth
GROQ_API_KEY
No
Fast inference / Whisper STT
NVIDIA_NIM_API_KEY
No
NVIDIA NIM fallback models (Qwen 3 Coder 480B, Nemotron Omni 30B)
NVIDIA_VISION_API_KEY
No
NVIDIA NIM vision models (camera/screen analysis)
GITHUB_TOKEN
No
GitHub API for OSINT features
SHODAN_API_KEY
No
Shodan device search
VIRUSTOTAL_API_KEY
No
VirusTotal file/URL reputation
TELEGRAM_BOT_TOKEN
No
Telegram bot for remote commands
DISCORD_BOT_TOKEN
No
Discord bot integration
INSTAGRAM_USER/_PASS
No
Instagram automation
REDDIT_CLIENT_ID/_SECRET
No
Reddit API access
Create .env from .env.example:
copy .env.example .env
Add to Windows Startup
FRIDAY can auto-start with Windows:
python -m friday.cli config
# Or ask: "Friday, add yourself to Windows startup"
Voice Commands
Friday responds to natural language. No rigid syntax required.
Screen & Vision
"Friday, what do you see on my screen?"
"Friday, is there an error on my screen?"
"Friday, find the word msconfig on my screen"
"Friday, click the play button"
Apps & Desktop
"Friday, open Spotify"
"Friday, play Naruto OST on Spotify"
"Friday, open VS Code"
"Friday, close Chrome"
"Friday, play Bloxfruits on Roblox"
Browser & History
"Friday, open the Jarvis repo I was loOKing at — the one by vierisid"
"Friday, what was I reading about LangGraph earlier?"
"Friday, open my IITM course"
Messaging & Communication
"Friday, message Mangesh on Instagram — tell him I'll be back soon"
"Friday, send an email to dev@example.com with a job offer letter from NexSemble"
"Friday, read my latest emails"
Goals & Productivity
"Friday, I have an IITM 8-week course ending May 31st at iitm.course.url — track it"
"Friday, what are my goals today?"
"Friday, how long have I been on YouTube today?"
"Friday, what's my screen time?"
Smart Home
"Friday, tell Alexa to switch off the lights"
"Friday, switch off my bedroom lights"
LLM & System
"Friday, switch to Claude"
"Friday, what LLM are you using?"
"Friday, do a deep research on NVIDIA's new GPU architecture and generate a report"
"Friday, generate a Python Flask server file"
Memory & Learning
"Friday, I've imported my Claude chat history — learn from it"
"Friday, what do you know about me?"
"Friday, update your understanding of my preferences"
"Friday, forget what I told you about X"
Module Reference
File
Purpose
friday.py
Root entry point. Gemini Live session launch with 5x auto-restart
friday/live.py
Main engine. 150+ tool declarations, TOOL_MAP, audio playback, system prompt, Phase 14-16 integration
Veronica report fix — 3-tier fallback (LLM synthesis → markdown parsing → structured sections), no more raw data dumps
Camera auto-switching — FRIDAY can switch/list/cycle cameras via cv_tool
Native Windows app (PyQt6 or Tauri)
Settings panel with key vault
Plugin system
Packaged installer (.exe)
Contributing
Friday is an open source project built solely by Arnav Phulari, shared publicly through NexSemble — a peer-learning and collaborative tech community in Pune, India[...]
Pull requests are welcome. For major changes, open an issue first.
# Fork and clone
git clone https://github.com/your-username/friday.git
# Create feature branch
git checkout -b feature/your-feature-name
# Make changes, then push
git push origin feature/your-feature-name
# Open a Pull Request on GitHub
Guidelines
Keep Windows-native compatibility (no Docker, no mandatory WSL2)
Never hardcode localhost — use OLLAMA_BASE_URL from env
Every new tool must be registered in friday_live.py's execute_tool()
Test on Windows 10 and 11
Known Issues
Issue
Status
Workaround
Gemini GOAWAY timeout on long idle
Fixed
keepalive_task() pings every 45s
Gemini 1008 GoAway on long tool sequences
Mitigated
Tool-ceiling removed; user accepts risk
Chrome SQLite locked when browser open
Fixed
Copy to temp before reading
pyautogui coordinates break on resolution change
In Progress
Use vision_click() instead
OpenCLI browser open <url> hangs on Windows
Fixed
Now opens via webbrowser + bind instead
Spotify "No active device" error
In Progress
Open Spotify app first, then retry
See_screen Gemini API 429 (rate limit)
Workaround
Falls back to Gemini 1.5 Flash automatically
SDK bug: send_realtime_input raises ValueError with >1 arg
CV engine MobileNet-SSD model download on first use
First-run delay
~7MB download, cached after that
Sidecar UDP discovery limited to LAN
By design
Use --server URL flag for WAN connections
JWT tokens stored in plaintext on disk
Known
Encrypted storage planned for v2.1
NVIDIA NIM intermittent HTTP 500 errors
Mitigated
Old key had ~30% error rate; key separated into research vs vision, new key resolves most issues
Deep research PDF report synthesis
Fixed
LLM was generating 1000+ JSON sections with 128K max_tokens → truncated output → raw data dump. Fixed: reduced to 80+ sections, 32K max_tokens, 3-tier fallback with LLM re-synthesis
deepseek-ai/deepseek-v4-flash times out
Known
Unreliable on NIM; replaced with qwen/qwen3-coder-480b-a35b-instruct as primary
Playwright + UIA COM access violation
Mitigated
Playwright browser tests corrupt UIA COM tree. Desktop functions auto-fallback to win32 backend
Friday is solely built by Arnav (@hackers-reality)
Co-leader of NexSemble — a peer-learning and collaborative tech community based in Pune, Maharashtra, India.
"This is not a chatbot . This is Friday — the real one."
Community — Join NexSemble
Friday is built in public and shared through NexSemble — a community of builders, hackers, and learners who ship real projects together. Come talk Friday, AI agents, and whatever you're bui[...]
-success?style=flat-square){kind=icon}
