alpha.8+: swap llama-server backend for genie-ai-runtime (opt-in then default)

[ai-hpc]

Goal

Swap GenieClaw's LLM backend from llama-server (llama.cpp) to jllm-server (https://github.com/GeniePod/genie-ai-runtime) — first as opt-in, then as default — once the runtime ships v1.0 (or earlier if parity comes sooner).

genie-ai-runtime is an Orin-tuned C++17 + CUDA inference runtime specifically built to coexist on the 7.6 GB iGPU alongside whisper-server, Piper, DFN, and Home Assistant. It's currently v0.1.0-alpha.1 — code-complete, pending hardware validation.

Tasks

Opt-in (target: GenieClaw alpha.8)

• New config field: [core] llm_backend = "llama-server" | "genie-ai-runtime", default "llama-server".
• When llm_backend = "genie-ai-runtime", setup-jetson.sh audits that /opt/jllm/bin/jllm-server exists and the jllm-server.service unit is enabled.
• LlmClient HTTP shape: confirm /v1/chat/completions (jllm) and /completion (llama) can both be served behind the same Rust client, or add a thin shim.
• Document the swap procedure in doc/llm-backend-swap.md.
• Test cycle on the Jetson: run a week of voice cycles against jllm with the latency banner (Print first-voice-reply latency banner at startup #19) recording first-reply times for A/B against llama.

Default flip (target: genie-ai-runtime v1.0)

• Flip the llm_backend default to "genie-ai-runtime" in deploy/config/geniepod.toml.
• setup-jetson.sh's [5/6] Checking llama.cpp block becomes a fallback check; primary is jllm.
• Update README.md alpha.X verification block with the new backend's measured numbers.
• CHANGELOG entry describing the swap and how to opt back into llama-server.

Acceptance

• Either backend can be selected via config alone — no rebuild of genie-core required.
• First-reply latency banner numbers on jllm are ≤ llama on the same model + prompt set.
• Memory budget audit (per doc/workflow/prompt.md §7) shows the swap freed iGPU vs llama (this is the whole reason genie-ai-runtime exists).

Related

• GeniePod/genie-ai-runtime — the runtime repo. Current alpha milestones live there.
• alpha.6: STT latency degrades from ~270 ms to ~3.6 s after LLM activity (iGPU contention) #2 — iGPU contention work that motivated a custom runtime instead of just tuning llama.cpp further.
• CPU pinning / core isolation for voice subsystem latency stability #25 — CPU pinning, complementary lever.
• Real streaming TTS: speak first sentence during LLM generation, not after #26 — real streaming TTS, also complementary on first-reply latency.

[ai-hpc]

Closing as done. The runtime swap that this umbrella issue tracked landed in a different shape than the original opt-in-then-default plan: when genie-ai-runtime v1.0.0 dropped before the alpha.8 GenieClaw cut, we skipped the opt-in phase and went straight to default-with-fallback.

Acceptance criteria status

• Either backend selectable via config alone, no rebuild. [services.llm].backend = "llama_cpp" | "genie_ai_runtime" is wired end-to-end. Backend identity flows through LlmClient::backend_name() into /api/health, startup logs, and genie-ctl status. Landed in llm: introduce backend facade #35, voice: report configured llm backend #38, core: expose configured llm backend #39, runtime: resolve llm service from config #40, llm: select backend from config #43.
• Memory budget audit shows the swap freed iGPU vs llama. Issue Evaluate raising genie-ai-runtime context default to 8192 #75's Jetson evidence in PR setup: start genie-ai-runtime before memory-heavy services #76 confirms genie-ai-runtime + Qwen3-4B Q4_K_M + INT8 KV fits an 8192-token context with ~1.9 GB runtime budget free on Orin Nano 8 GB — comfortably more headroom than the llama.cpp + Phi-4-mini baseline. genie-model-cache-status.sh (PR setup: preserve warm genie-ai-runtime model cache #70) gives operators a tool for ongoing audit.
• First-reply latency A/B on the same prompt set. Not formally recorded — the runtime is now the default, the latency banner from Print first-voice-reply latency banner at startup #19 is in place, but no head-to-head comparison was published. Effective verification: PR setup: preserve warm genie-ai-runtime model cache #70's page-cache fix makes restart-after-load ~1.3 s warm vs ~30 s cold (matches the runtime's ROADMAP perf table), and the user has been running on this stack day-to-day.

Where each task ended up

#27 task	Status	Resolved by
[core] llm_backend config field	✓ (shape became [services.llm].backend)	#43, #55
setup-jetson.sh audits genie-ai-runtime binary	✓	#56 (--runtime genie-ai-runtime), #63 (auto-fallback when missing)
LlmClient HTTP shape unified	✓	#35 (backend facade), #74 (request profile per backend)
Swap procedure documented	✓	#55 (geniepod.toml comments + ARCHITECTURE.md / README)
Default flip in deploy/config/geniepod.toml	✓	#55 (closes #52, the v1.0.0-grounded replacement for stale #33)
setup-jetson.sh step 5 becomes fallback check	✓	#55 (renamed to "Check LLM runtimes")
README updated with new backend numbers	✓	#55 + alpha.9 release notes
CHANGELOG entry	✓	#55

Related issues / PRs filed against the same backend-swap workstream

• Make llama.cpp an optional LLM backend (introduce LlmBackend abstraction) #32 — LLM backend abstraction (closed by the llm: introduce backend facade #35–llm: select backend from config #43 series)
• Default to genie-ai-runtime (v0.1.0-alpha.8) once #32 lands #33 — closed as stale (pinned to alpha.8 timing); replaced by Default to genie-ai-runtime v1.0.0 (replaces stale #33) #52
• Default to genie-ai-runtime v1.0.0 (replaces stale #33) #52 — default flip to v1.0.0 (closed by Fix: Default to Genie-ai-runtime #55)
• Build + install pipeline for genie-ai-runtime v1.0.0 (ship genie-ai-runtime.service alongside genie-llm.service) #54 — runtime build/install pipeline (closed by feat(deploy): genie-ai-runtime v1.0.0 build+install pipeline (issue #54) #56)
• setup-jetson.sh enables genie-ai-runtime.service without verifying jetson-llm-server is installed #60 — setup-jetson auto-fallback when binary missing (closed by fix(setup-jetson): auto-fallback to llama.cpp when jetson-llm-server is missing (#60) #63)
• setup-jetson.sh patch_services_llm_backend fails with 'Permission denied' — auto-fallback prints NOTE but never patches geniepod.toml #65 — awk perm bug in PR fix(setup-jetson): auto-fallback to llama.cpp when jetson-llm-server is missing (#60) #63's fallback patcher (still open at time of alpha.8+: swap llama-server backend for genie-ai-runtime (opt-in then default) #27 closure)
• Investigate slow Qwen3 model loading and warm weight behavior #69, Evaluate raising genie-ai-runtime context default to 8192 #75 — runtime tuning that came after the swap landed (closed by setup: preserve warm genie-ai-runtime model cache #70, setup: start genie-ai-runtime before memory-heavy services #76)
• genie-core chat fails through genie-ai-runtime while health reports connected #73 — chat API integration after the default flip (closed by core: shape genie-ai-runtime chat requests #74)
• Chat UI writing indicator does not animate on :3000 #79, Dashboard services table only shows core and llm #80 — UI / dashboard fixes once the swap was visible to users (closed by ui: animate chat writing indicator #81, dashboard: show full systemd service status #84)

If anyone wants the formal A/B latency record, that's a clean targeted follow-up — most of the instrumentation is already in place (banner from #19, per-backend logging from #38/#39).

PR #47 was the original opt-in attempt against this issue; closing it separately as superseded by the work above.