deploy: pin genie-* services to dedicated CPU cores for voice latency…

[enjoyandlove]

Summary

Closes #25

• Pins each genie-* systemd unit to a dedicated CPU bucket on the Jetson Orin Nano (6 cores) so the voice path (wake → STT → LLM → TTS → playback) no longer competes with itself for scheduler time. Closes CPU pinning / core isolation for voice subsystem latency stability #25.
• Bucket layout: 0–1 system + ALSA + MQTT + genie-api/genie-governor/genie-health/genie-wakeword, 2–3 whisper-server (STT decode, 2 threads), 4 llama-server (GPU-bound, hosts CUDA dispatch + sampler), 5 genie-core and every audio child it spawns (piper, sox, deep-filter, arecord, aplay).
• Moves genie-wakeword off cores 4–5 (now reserved for llama / core) onto the system bucket 0–1; keeps SCHED_FIFO priority 50 so the continuous audio loop is not preempted by best-effort work sharing those cores.
• Documents the bucket map, verification commands (taskset -pc, ps -L -o psr, tegrastats), and the acceptance signal in doc/deployment-and-ops.md — including why we stopped short of kernel-level isolcpus= (requires a Jetson reflash / extlinux edit, out of scope for this PR).

Real Behavior Proof

• I have built and run the affected code locally.
• I have verified the equivalent path with the targeted CI-size-budget test.

Test plan

• sudo systemctl daemon-reload && sudo systemctl restart genie-*.service on the Orin Nano and confirm every unit reaches active (running).
• For each service, run taskset -pc "$(systemctl show -p MainPID --value <svc>.service)" and confirm the affinity matches the bucket map above.
• ps -L -o pid,tid,psr,comm -p "$(pidof whisper-server)" shows decode threads spread across cores 2–3 only; same check for llama-server shows all threads on core 4.
• Verify genie-core's spawned children (piper, sox, deep-filter, arecord, aplay) inherit core 5 via for p in $(pgrep -P "$(systemctl show -p MainPID --value genie-core.service)"); do taskset -pc "$p"; done.
• Run 10 consecutive voice cycles end-to-end and capture STT latency; median ± 100 ms is the acceptance signal called out in CPU pinning / core isolation for voice subsystem latency stability #25. Watch sudo tegrastats --interval 250 during the run for unexpected core saturation.
• Sanity-check wake-word responsiveness on the new 0–1 affinity (SCHED_FIFO 50) — confirm no missed wakes under simultaneous LLM load.

[ai-hpc]

(duplicate review — see the full review just below.)

[ai-hpc]

Design is sound and the doc page is exactly the right shape — bucket map, verification commands, acceptance signal, and an honest "we stopped short of isolcpus= because that needs a Jetson reflash" note. The keep-SCHED_FIFO-but-move-affinity-to-0-1 treatment of genie-wakeword is also the right call (the previous 4 5 clashes with this PR's own llama / core assignment). I'd be happy to land this in spirit.

Three things blocking merge as-is, in rough order of severity:

1. The pinning targets genie-llm.service, but the default LLM unit on main is now genie-ai-runtime.service

Between when this PR was opened (2026-05-17) and today, the default LLM backend got flipped:

• #55 — flipped [services.llm].backend default to genie_ai_runtime, made genie-ai-runtime.service the default unit.
• #56 — landed the genie-ai-runtime v1.0 build/install pipeline.
• #70 — removed ExecStartPre=drop_caches from genie-ai-runtime.service to preserve warm Qwen3-4B page-cache across restart.
• #76 — added Before=genie-whisper.service genie-whisper-warmup.service homeassistant.service genie-core.service to genie-ai-runtime.service, plus --int8-kv -c 8192 and Environment=GENIEPOD_AI_RUNTIME_CONTEXT=8192.

This PR pins genie-llm.service to CPUAffinity=4, which is correct for an operator on the llama.cpp fallback path. But on the default alpha.9 stack, genie-llm.service is masked / not running — the active LLM unit is genie-ai-runtime.service, which this PR leaves unpinned. So the central acceptance criterion of #25 (LLM on dedicated core 4, no contention with whisper / core) doesn't actually hold on the default deploy.

Fix: add a CPUAffinity=4 line to deploy/systemd/genie-ai-runtime.service with the same justification comment, parallel to the one this PR adds to genie-llm.service. The two units already Conflicts= each other so only one will ever be active at a time; the affinity setting on the inactive one is a no-op.

2. PR body contains the Claude Code AI-attribution trailer

The trailer at the bottom of the PR body — the robot-emoji line that names the tool — is exactly what #91 just added a Contribution / PR body checklist CI rule against. Same project norm that CONTRIBUTING.md "Commit hygiene" already calls out for Co-Authored-By: Claude / Copilot / other AI-assistant trailers. Using AI tooling to help draft the PR is welcome (and the writeup here is genuinely good); attribution in the PR description and git log stays with the human contributor so credit is unambiguous.

The check didn't run on this PR's existing rollup because the workflow uses pull_request_target and the PR hasn't been touched since the rule landed — but the next edit / push will fire it as a failure. Fix is one line: strip the footer from the PR body before re-pushing.

3. CI hasn't run on this PR

statusCheckRollup is empty. The PR pre-dates the CI suite landing on main (PR #37 / #49 / #50 / #51 / #62), and the standard pull_request workflows source their definitions from the PR's head ref — which doesn't carry them. After #89 switched the contribution-checklist workflow to pull_request_target, that one will fire on the next push, but cargo fmt / clippy / test and aarch64 cross-compile need first-time-contributor approval for fork PRs.

This isn't your problem to fix — but it means the PR can't be verified to compile / clippy-clean / cross-compile until either (a) a maintainer approves the CI run, which we can do once you push the rebased version, or (b) you git rebase main to bring the workflow files onto your head branch.

Two smaller things, neither blocking

• genie-core.service CPUAffinity=5 interaction with PR #87. PR #87 changed genie-core to use tokio::task::LocalSet::spawn_local so HTTP requests multiplex on a single OS thread. Pinning genie-core to one core is consistent with that single-thread design — the chat-vs-health-poll concurrency PR #87 unlocked stays intact. But it's worth a sentence in the comment acknowledging the interaction so a future reader doesn't reflexively bump the thread count without understanding the trade-off.
• --threads=4 on llama-server with CPUAffinity=4 (one core). Your comment acknowledges this and suggests revisiting --threads 1 or 2 if prompt-eval shows scheduler churn. That's the right framing — leave the threads at 4 for now and put the measurement burden on whoever sees the issue.

Suggested path forward

Pick one:

(a) Refresh and re-push. Rebase onto main, add CPUAffinity=4 to deploy/systemd/genie-ai-runtime.service, strip the 🤖 Generated with [...] trailer from the PR body. We approve CI, check the 7 jobs, and merge. The bulk of the work (doc page, layout, justification comments) is reusable as-is.

(b) Close as superseded if you don't have the bandwidth, and we either re-open with the rebase or someone else picks up #25. Pattern matches what we did with #47 a few days ago — the design here would feed into the follow-up cleanly.

Either is fine. Let me know which way you want to take it, and we'll move from there.

[enjoyandlove]

Hi,@ai-hpc
I have just fix this PR. Could you please review it?
Thank you for your time.

[ai-hpc]

Thanks for the revision — PR body now has the proof checkboxes ticked and the AI-attribution trailer is gone (point #2 from the earlier review). The added pinning on genie-api / genie-governor / genie-health / genie-mqtt / genie-wakeword / genie-whisper / genie-core is all good and lands real value.

One blocker, plus a new issue this revision introduced. Both small.

Blocker — deploy/systemd/genie-ai-runtime.service still has no CPUAffinity=

This was the substantive concern from the first review and it's still open. Confirmed against 43fa029 on this PR's head:

$ gh api 'repos/GeniePod/genie-claw/contents/deploy/systemd/genie-ai-runtime.service?ref=43fa0299' \
    --jq .content | base64 -d | grep -nE '^CPUAffinity'
# (no output)

On the alpha.9 default deploy, the active LLM unit is genie-ai-runtime.service, not genie-llm.service. As this PR stands, the central acceptance criterion from #25 ("LLM on dedicated core 4") only fires when an operator has explicitly flipped [services.llm].backend = "llama_cpp". The default deploy gets no LLM pinning at all — Qwen3 ends up on whichever core the kernel picks, contending with whisper / core / system.

Fix is one stanza on deploy/systemd/genie-ai-runtime.service, mirroring what's already on genie-llm.service:

# CPU pinning for voice latency stability (issue #25). Same core 4 as the
# llama.cpp fallback (genie-llm.service), since genie-ai-runtime and llama-
# server are mutually exclusive via Conflicts= and only one is ever active.
# Inference is GPU-bound, so a single dedicated core hosts CUDA dispatch +
# sampler without contending with whisper (2-3) or genie-core (5).
CPUAffinity=4

New issue this revision introduced — doc documents pinning that doesn't exist

The updated doc/deployment-and-ops.md now says core 4 is llama-server / jllm-server (GPU-bound; one core hosts CUDA dispatch + sampler — whichever LLM backend is active). That's factually wrong on the merged state of this PR: an operator following the doc and running

pid=$(systemctl show -p MainPID --value genie-ai-runtime.service)
taskset -pc "$pid"

…would see the runtime is NOT on core 4, despite the doc claiming it. Same fix as the blocker — once the unit has CPUAffinity=4, the doc becomes accurate.

CI status

I approved the two action_required runs (CI and Cross-compile (aarch64 / Jetson)) for this PR's head, so the next cycle should give us the full check matrix. Currently only the PR body checklist has run on the new commits; the cargo / cross-compile / no-default-features signal needs the queued runs to land before this can merge.

Suggested next step

One follow-up commit: add the CPUAffinity=4 stanza to deploy/systemd/genie-ai-runtime.service. The rest of this PR is good as-is. Once that lands and CI clears, I'll re-review and merge.

[enjoyandlove]

Hi, @ai-hpc
Could you please review this PR again?

[ai-hpc]

Both blockers from the previous review are resolved. Diffing deploy/systemd/genie-ai-runtime.service at the new head 7ccf14d:

+# CPU pinning for voice latency stability (issue #25). Inference is GPU-bound
+# (jetson-llm-server offloads to Jetson iGPU), so one dedicated core is enough
+# to host CUDA dispatch + sampler + tokenizer without contending with whisper
+# (2-3) or genie-core (5). Mirrors genie-llm.service; only one of the two units
+# runs at a time (Conflicts=genie-llm.service above), so both can claim core 4.
+CPUAffinity=4

That's exactly the right shape — including the Conflicts=genie-llm.service callout that explains why both LLM units can safely declare CPUAffinity=4 without stepping on each other. Future readers won't have to puzzle out why two units claim the same core.

With this commit, the doc page's "core 4 — llama-server / jllm-server (whichever LLM backend is active)" line is now accurate on both backend paths. The doc-vs-code mismatch I flagged in the v2 review is closed.

Summary of where the PR sits versus my earlier review points:

1. PR body trailer / proof checkboxes (v1 review): fixed in the earlier revision.
2. genie-ai-runtime.service missing CPUAffinity= (v1+v2 review blocker): fixed in commit 7ccf14d. Doc and code now agree.
3. CI (v1+v2 review): I approved the new CI and Cross-compile runs that were queued on this push. Waiting on them to land green before final merge — the substantive review is done.

Once CI clears on 7ccf14d, this is ready to land. Approving on the code; the maintainer can squash-merge as soon as the remaining checks turn green.

[enjoyandlove]

Hi, @ai-hpc
I have just addressed requested update.
could you please review this PR again?

[ai-hpc]

Merged at 7268bbba418b6134b52c80a40b1263733d5e50ac.
Thanks to @enjoyandlove!

[enjoyandlove]

@ai-hpc
Thank you for merging this PR.