desktoprobo — voice control with Claude (feature/claude-agent)

Speak to the robot and it moves, verifies its own motion, and talks back. This branch turns the desktop robot into a language-only teleoperated agent: a ReSpeaker mic feeds always-on offline speech recognition, Claude parses each utterance into a typed command frame, a planner grounds it against session memory, a closed-loop executor drives the wheels over /cmd_vel, an observer checks the odometry against what was commanded, and the robot speaks the outcome through Piper TTS — with a direct stop path that reacts to the partial hypothesis of "stop" without waiting on any network.

Everything shared with the rest of the project is documented elsewhere — this README covers only the Claude agent:

For	See
Hardware, Ubuntu 24.04 + ROS 2 Jazzy install, motor & ReSpeaker setup, PS4 teleop	README.md on master
Study design & experimental-manipulation rationale (DESIGN_DECISIONS.md)	the full-system / baseline-feature-free branches
The architecture spec this branch implements	agent_architecture.html
Package usage, offline runner, study event-log schema	src/desktorobo/desktorobo_agent/README.md

---

Quick start

cd ~/desktoprobo && source /opt/ros/jazzy/setup.bash && source install/setup.bash

# Full voice session on the robot (motor node + agent, Vosk mic, Piper voice):
ros2 launch desktorobo_agent agent_launch.py

# Typed commands instead of the mic — two terminals; `ros2 launch` does not
# forward the terminal's stdin to nodes, so run the agent directly:
ros2 run mobile_robot_control mobile_robot_control_node
ros2 run desktorobo_agent agent_node --ros-args -p speech:=text \
  -p piper_model:=/home/pi/voice_models/piper/en_US-amy-low.onnx

# No robot needed — same pipeline against a kinematic mock, on any machine:
cd src/desktorobo/desktorobo_agent && python3 -m desktorobo_agent.run_offline

Say things like "move forward twenty centimeters", "turn left a little bit", "keep going until I say stop", "one step is ten centimeters", "do you have a camera?" — and "stop" at any time.

---

How the Claude agent works — one utterance, end to end

What happens between "move forward five centimeters" and the robot's spoken report, in order:

1. Capture. arecord streams all 6 raw channels from the ReSpeaker (plughw:CARD=ArrayUAC10,DEV=0, S16_LE @ 16 kHz); channel 0 — the board's own AEC + beamformed mix — is extracted and fed to Vosk (asr.py). Because the AEC reference cancels the robot's own speech out of channel 0, the mic can stay hot while the robot talks and it cannot obey itself.
2. Recognition. Vosk emits partial hypotheses while you are still speaking and one final text when you pause. Partials go only to the direct stop path (step 3); finals are stamped with an arrival time and enqueued for the utterance worker thread — the recognizer thread never blocks on Claude or on speech output.
3. Direct stop path. If a partial hypothesis starts with stop / halt / freeze (within its first three words), the planner preempts the executor immediately — no queue, no interpreter, no LLM, no network. The running action dies at its next 50 ms control tick. This is why saying "sto—" mid-motion halts the robot before the utterance even finishes.
4. Triage. The Interpreter facade decides who parses the final text (interpreter.py). Bare stop phrases take a rule fast path (~1 ms measured). In the default auto backend the rule parser runs first on everything: a confident parse (any primitives, a question, a stop) is used as-is, and non-actionable fragments of ≤ 2 words are dismissed as ASR noise — so on a Pi whose Claude round-trip is seconds, routine template commands and mic noise never build a backlog behind the LLM. Only utterances the rules can make nothing of go to Claude; utterances that waited > 10 s in the queue lose LLM eligibility entirely (they describe a robot state that no longer exists).
5. Claude. ClaudeInterpreter sends the system prompt (the robot's capabilities, the JSON schema, 15 parsing rules) plus 6 few-shot examples — all verbatim Study 1 utterances — and the live text, with the session's grounding context (user-defined units, calibrated qualitative terms, last action) serialized into the prompt. Two transports are tried per utterance: the Anthropic SDK (needs ANTHROPIC_API_KEY; 6 s timeout, ~1 s typical), then the claude -p headless CLI using the machine's Claude Code login (15 s timeout, 5–12 s measured on this Pi). The reply's JSON is extracted (raw, fenced, or outermost-braces), validated through the enum constructors, and becomes a CommandFrame; any failure at any stage falls back to the rule parser — the pipeline never blocks on, and never dies from, the network.
6. Grounding. The planner resolves each primitive's magnitude to meters or radians (magnitude.py): quantitative units convert directly, qualitative words ("a little bit") and invented units ("one step") resolve through the session's negotiated values or the defaults table, relative ops ("the same amount", "twice that") read the last-action register — which holds the observed magnitude of the previous action, not the commanded one.
7. Policy. ACT / PROBE / CLARIFY (policy.py): probes execute a diagnostic wiggle; an undefined unit or an unclarified "a little bit" triggers one spoken question per term per session (ask-once), and an unanswered question never blocks — the next utterance either answers it or drops it. Everything else acts immediately.
8. Execution. One action at a time, closed-loop on wheel odometry (execution.py): a 20 Hz control loop integrates per-tick pose deltas and commands velocity with a P-style approach profile until the remaining distance is within 5 mm (or 2° for turns). Preemption is checked every tick; a stall watchdog aborts if the pose stops changing for 1 s while velocity is commanded, and bounded actions carry a hard deadline of 3 × (expected duration + 1 s), floored at 5 s — stale odometry or a blocked drivetrain can never become a runaway.
9. Motion. Velocities go out as a /cmd_vel Twist; the motor node clamps them (0.15 m/s, 1.0 rad/s), runs the differential-drive kinematics, and writes goal velocities to the two XL330s. Wheel positions are read back at 10 Hz and integrated into /odom.
10. Verification. The monitor compares actual vs expected (monitor.py): within 12 % (floors: 1.5 cm / 5°) is "as commanded"; undershoot is attributed to slip/resistance, overshoot to coasting; and an "actual" implying more than twice the motor node's speed caps is flagged as an odometry artifact — reported honestly as sensing glitch, and barred from session calibration.
11. Feedback. Acknowledgments echo the grounded parse ("Got it — forward 5 centimeters."), completion reports speak only when something deserves attention (R4.2), questions get answers from proprioception. Piper synthesizes at the voice's own sample rate and aplay is pinned to the ReSpeaker's 3.5 mm jack — a dedicated speaker thread keeps a talking robot fully interruptible.

---

Architecture

The runtime implements agent_architecture.html: a linear main pipeline with support modules feeding inward, a monitor/feedback loop closing back to the human, and a privileged stop path.

flowchart LR
    H(["Human"]) --> ASR["ASR<br/>Vosk, always-on"]
    ASR -- "finals" --> INT["Interpreter<br/>Claude + rules"]
    INT --> PLAN["Planner / Manager"]
    PLAN --> EXE["Execution<br/>closed-loop, 20 Hz"]
    EXE -- "/cmd_vel" --> ROB["Robot<br/>motor node, XL330s"]
    ROB -- "/odom" --> MON["Observer / Monitor"]
    GND["Grounding &amp; Calibration"] <--> INT
    GND <--> PLAN
    POL["Policy<br/>ACT · PROBE · CLARIFY"] --> PLAN
    MON --> FB["Feedback<br/>Piper TTS"]
    FB -- "speech" --> H
    ASR -. "partial 'stop'" .-> EXE
    MON -- "observed actuals" --> GND

Loading

Architecture box → module map

agent_architecture.html box	Module	What it does
Interpreter (R1·R2·R3·R6)	interpreter.py	Rule parser + Claude backend behind a triage facade; typed CommandFrame out
Grounding & calibration	grounding.py	Unit contracts, qualitative recalibration, frame convention, ask-once registries, last-action registers, taught routes, disclosure scripts
Policy (ACT/PROBE/CLARIFY)	policy.py	Exactly one decision per utterance; at most one clarification, never re-asked
Planner / Manager (R2–R5)	planner.py	Batching, preemption, repair routing (repeat/reset/probe/route), clarification Q&A, question answering, sanity notes
Primitive action library	command_frame.py	PrimitiveKind: move, rotate, continue, continue-until-stop, stop, probe, repeat-last, reset-orientation, navigate-goal
Execution (R1·R2·R3·R5)	execution.py	Serial interruptible runtime, closed-loop on odometry, stall/deadline watchdogs
Robot	robot.py, ros_nodes.py	ROS-free RobotInterface; RosRobot adapter (/cmd_vel out, /odom in) or MockRobot (kinematic sim, optional noise)
Observer / Monitor	monitor.py	Expected vs actual, cause attribution, implausible-measurement guard
Feedback	feedback.py	Ack/report/answer/clarify phrasing; non-blocking Piper/say/print TTS chain
Direct stop / override path	app.handle_partial()	Fires on partial ASR hypotheses; bypasses queue, interpreter, and LLM
— (wiring)	app.py	Assembles everything; utterance queue + worker thread; staleness gate; shutdown ordering
— (I/O)	asr.py, eventlog.py	Speech sources (PyAudio / arecord / stdin); append-only ndjson study log

All agent logic is plain ROS-free Python; ros_nodes.py is a thin adapter. That is why the identical pipeline runs on a laptop against MockRobot (run_offline.py) and why the test suite runs without ROS.

The Interpreter: how Claude parses commands

The centerpiece of the branch. Three layers, tried in this order for every utterance:

1. Stop fast path. An anchored stop-phrase match ("stop", "halt", "freeze", "wait", "hold on", "that's enough", …) returns a STOP frame from the rules immediately — stop never waits on a network round-trip (measured 1 ms).

2. Rule triage (auto backend). The rule parser is a full implementation of the Study 1 command taxonomy: number-word normalization ("twenty" → 20), batch splitting on "and then", per-segment primitive extraction (move / rotate / continue-until-stop / repeat / reset), magnitude classification (quantitative / qualitative / invented-unit / relative), and ambiguity flagging (frame reference, referential-without-vision, unspecified magnitude). If it produces primitives, a question, or a stop, its frame is used and Claude is never called. Non-actionable utterances of ≤ 2 words are classified as ASR noise — unless they look like a fragment of a real command (a bare unit or direction), which still goes to Claude.

3. Claude (ClaudeInterpreter). For utterances the rules can't parse — compound phrasings, indirect requests, questions in free form. Transport chain, re-tried per utterance so a transient failure never disables Claude for the session:

Priority	Transport	Requires	Latency	Timeout
1	Anthropic SDK (messages.create, max_tokens=700, max_retries=0)	ANTHROPIC_API_KEY in env or workspace .env	~1 s	6 s
2	claude -p "<prompt>" --model <model> headless CLI, stdin=DEVNULL	Claude Code login on the machine	5–12 s measured	15 s
3	Rule parser	nothing	~1 ms	—

The model defaults to claude-haiku-4-5 (chosen for latency; override with DESKTOROBO_LLM_MODEL). A dependency-free .env loader reads $CWD/.env then ~/desktoprobo/.env, with real environment variables always winning (see .env.example). The prompt teaches the robot's reality — sensor-less, two wheels, desk scale — and demands JSON only; the 6 few-shot examples are verbatim participant utterances covering batching, continue-until-stop, qualitative magnitudes, ungroundable object references, questions, and invented units. Replies are parsed defensively (raw JSON, fenced block, or outermost braces), validated field-by-field through the enum constructors, and any failure — timeout, bad JSON, invalid enum — degrades to the rule parse of the same utterance.

Backlog protection (added after live testing, see git history): utterances are processed serially, so with a multi-second transport every needless LLM call delays everything behind it. Besides the triage above, the utterance worker refuses to spend an LLM round-trip on any utterance that already waited more than 10 s in the queue — it may still act via the rules, but it must not push the conversation further into the past. A shutdown flag also keeps backlogged commands from firing into a tearing-down system after Ctrl-C.

Threading model

Thread	Owner	Runs	Must never
ASR recognizer	asr.py	Vosk over the audio stream; partial/final callbacks	block — the partial-stop check is its only synchronous work
Utterance worker	app.py	interpret → plan, one utterance at a time	die — every exception is caught and logged
Executor worker	execution.py	the 20 Hz control loop, one action at a time	play audio or call the LLM
TTS speaker	feedback.py	piper→aplay playback, serialized	be waited on by anyone (speak() only enqueues)
ROS executor	ros_nodes.py	/odom callback, session-start timer	—
Motor node timers	odrive_command.py	10 Hz odometry, 10 Hz watchdog, 3 s reconnect — all serial I/O on one thread, every path exception-guarded	crash on a dead port

The completion callback (action_complete) fires on the executor thread while it is still marked busy; it only enqueues speech and updates state, so the executor can never be wedged by feedback.

Stop, safety, and the things that must not fail

Stop is layered so that no single failure leaves the robot moving:

1. Partial-hypothesis stop — preempts within one 50 ms control tick, measured 0.12 ms for the preempt call itself; no network involved.
2. Final-utterance stop — rule fast path, flushes the action queue.
3. Executor watchdogs — stall (pose frozen 1 s under commanded velocity) and deadline (3 × (expected duration + 1 s), floored at 5 s) abort the action and say why.
4. Motor-node /cmd_vel watchdog — 0.5 s without a Twist zeroes both motors, and retries every 100 ms until the zero write is confirmed on the wire; after ~0.3 s of unconfirmed writes it forces an immediate port recovery instead of waiting for the 3 s reconnect tick.
5. Servo-side dead-man — the XL330 bus watchdog (20 × 20 ms) makes the motors stop themselves 400 ms after instruction packets cease, covering even a SIGKILLed motor node.
6. Velocity clamps — 0.15 m/s / 1.0 rad/s at the motor node, upstream of the kinematics; the agent's own cruise speeds (0.10 m/s, 1.2 rad/s) sit at or below desk scale.
7. Sanity bounds, disagree-and-comply (R5.3) — commands beyond desk scale (> 1.5 m, > 2 turns) execute, but with a voiced concern: "Going ahead anyway; say stop if I should stop."

The PS4 teleop stack is intentionally not launched with the agent — both publish /cmd_vel. Keep it in a second terminal as an emergency stop-and-abort (see the agent_launch.py docstring).

The robot boundary: motor node and odometry

The agent talks to the same /cmd_vel ↔ /odom seam the joystick teleop used; odrive_command.py (the motor node) was hardened on this branch:

• Port resolution by stable ID — /dev/serial/by-id/usb-FTDI… instead of /dev/ttyUSB0, because USB hiccups re-enumerate the device and a node holding the old fd writes into the void.
• Recovery ladder — a 3 s tick re-pings motors, re-enables torque that silently dropped after a brownout (the motor still answers pings!), and on the second consecutive dead tick reopens the port entirely. All three failure modes occurred live during testing.
• Odometry integrity — the XL330 position register is a continuous ±2³¹ accumulator, so the first read after a (re)start is arbitrary: the baseline is seeded from the first real read (None-seeded, re-armed on every motor re-init) and any per-tick delta above half a wheel revolution (2048 counts — ~5 rev/s, five times the commanded maximum) is rebased instead of integrated. Upstream, the monitor's plausibility guard (> 0.30 m/s or > 2.0 rad/s implied) catches whatever slips through and keeps it out of session calibration.
• Clean shutdown — zero velocity + torque off + port close for both motor IDs on exit, tolerant of a second Ctrl-C mid-teardown.
• Startup ordering — the agent node waits for the first /odom message (20 s deadline) before speaking the onboarding script and running the demo wiggle; otherwise the demo would drive against a frozen pose until the stall watchdog fired, physically over-rotating the robot.

Key numbers

Quantity	Value	Where
Control loop	20 Hz (50 ms tick)	defaults.CONTROL_RATE_HZ
Goal tolerance	5 mm / ~2°	GOAL_TOL_M, GOAL_TOL_RAD
Cruise / floor speeds	0.10 m/s, 1.2 rad/s / 0.05 m/s, 0.5 rad/s	defaults.py
Motor clamps	0.15 m/s, 1.0 rad/s	odrive_command.py
Bare "move forward" / "turn left"	10 cm / 30°	DEFAULT_MOVE_M, DEFAULT_TURN_RAD
"a little bit" (move / turn)	5 cm / 15°	qualitative buckets, recalibratable per session
Undefined "step" / "unit"	10 cm	until the user declares otherwise
"one wheel rotation"	14.45 cm (2π × 0.023 m)	physically true
"N seconds"	N × cruise speed	time-as-magnitude proxy
Monitor tolerance	12 % relative, floors 1.5 cm / 5°	monitor.py
Implausible-actual guard	> 0.30 m/s or > 2.0 rad/s implied	2× the motor clamps
Stop-path preemption	≤ 1 control tick	execution.py
LLM staleness gate	10 s queue age	app.STALE_LLM_S
Claude timeouts	SDK 6 s, CLI 15 s	defaults.py
Wheel radius / track	0.023 m / 0.135 m	odrive_command.py

---

What it took to make this work on the Pi

The agent package was built and validated against MockRobot; this branch is the record of making it real on this specific Raspberry Pi 5 and robot.

Claude without an API key. This Pi has no ANTHROPIC_API_KEY, so the SDK transport could never activate and "Claude" would silently have been rules-only. The claude -p CLI transport (commit f996f95) reuses the machine's Claude Code login: system prompt + few-shots folded into one -p prompt, --model passed through, stdin=DEVNULL so the child can never steal the node's stdin, output fence-stripped before JSON parsing. A dependency-free .env loader was added for the day a key exists.

Audio without PyAudio. The stock Pi image has no portaudio, so the PyAudio-based VoskASR could not start. ArecordVoskASR captures via an arecord subprocess (alsa-utils is always present), auto-detects the ReSpeaker from arecord -l, reads all 6 channels raw, and feeds channel 0 to Vosk — same partial/final callback contract, so the direct-stop path was untouched. It auto-restarts arecord if USB resets kill the stream. make_source("voice") does a real import pyaudio probe (not find_spec — an installed-but-broken pyaudio would crash later) and falls back cleanly.

TTS that sounds right and comes out the right hole. Piper's playback rate is read from the voice's own .json — amy-low is 16 kHz, and the previous hardcoded 22050 played 37 % too fast. aplay is pinned to the ReSpeaker jack by ALSA name because the Pi 5 has no analog jack and PulseAudio silently routes to HDMI. Piper/aplay children are killed and reaped on every failure path so zombies can't accumulate over a session.

Motor node hardening (commit d9bdfe4) and pipeline hardening (commit 17c9253): the watchdogs, recovery ladder, odometry integrity, LLM backlog triage, and startup/shutdown races described above. Three of those fixes came directly from failures observed live:

1. Mid-session the U2D2 re-enumerated ttyUSB0 → ttyUSB1; the node kept writing into the dead fd and the robot silently stopped responding — while the executor's stall watchdog aborted the action and the robot said it was blocked. The architecture's monitor/feedback loop caught its own outage. Fixed with by-id resolution + port reopen + watchdog escalation.
2. A session log showed a queued "move forward" starting after Ctrl-C — the backlog fired into teardown. Fixed with the shutdown flag + staleness gate.
3. A phantom odometry jump (position-accumulator baseline) made the monitor "attribute" motion the body never performed. Fixed at both ends: baseline seeding in the motor node, plausibility guard + calibration skip in the agent.

Machine setup done for this branch

pip install --user --break-system-packages anthropic vosk   # anthropic 0.117.0, vosk 0.3.45
~/voice_models/vosk/vosk-model-small-en-us-0.15              # Vosk small English model
~/voice_models/piper/en_US-amy-low.onnx (+ .json)            # Piper voice, 16 kHz

piper-tts, numpy, dynamixel_sdk, and ROS 2 Jazzy were already present. Optional: cp .env.example .env and add an ANTHROPIC_API_KEY for the ~1 s SDK transport; without it the CLI transport is used, and with neither the agent runs rules-only.

---

Test record

Offline (this Pi):

• pytest test/ — 69/69 pass (the acceptance suite, utterances verbatim from Study 1 transcripts; re-run 2026-07-27 with all branch changes).
• Claude parsing via the CLI transport, no API key on the machine: "move forward five centimeters" → move/forward/quantitative 5 cm (7.2 s); "turn to my left a little bit and then keep going until I say stop" → rotate(left, qualitative little) + continue-until-stop with frame_reference and qualitative_magnitude flagged (11.9 s); "do you have a camera?" → question (5.5 s); "stop" → rule fast path (1 ms).
• Direct stop during continue-until-stop: preempt call 0.12 ms; monitor attributed "you stopped me"; robot spoke "Stopped — I'd gone 10 centimeters."

On the robot (2026-07-17; tiny motions only — the robot was cable-tethered):

Command (parsed by Claude)	Expected	Actual (odometry)	Monitor
move forward three centimeters	0.030 m	0.0317 m	ok — as commanded
turn left fifteen degrees	15.0°	16.1°	ok — as commanded
move backward two centimeters	0.020 m	0.0228 m	ok — as commanded
turn right ten degrees	10.0°	10.15°	ok — as commanded

Acknowledgments and reports were spoken audibly through the ReSpeaker jack during these runs. The robot transcribes its own speech poorly — expected and desirable: the AEC exists precisely so it cannot obey itself.

Known limitations / next steps

• CLI-transport latency is 5–12 s per utterance that actually needs Claude (SDK path with a key: ~1 s). Triage keeps template commands off the LLM, but a complex utterance still gets its acknowledgment only after the parse; a pre-parse "mm-hm" earcon would mask the gap.
• Continue-until-stop and the onboarding demo wiggle were exercised on the mock robot but deliberately not on the tethered robot (continuous motion vs cables). Untether before trying them.
• Human-voice ASR accuracy, hot-mic behavior in a noisy room, and speaker volume still need a human in the room.
• The PS4 teleop stack must not run alongside the agent — both publish /cmd_vel.
• Pi-specific defaults (Piper voice path, tts_engine:=piper) live in agent_launch.py; a bare ros2 run desktorobo_agent agent_node uses the node's generic defaults (tts_engine:=auto, printed speech unless piper_model is passed).

---

License & Attribution

Built on the TiltyBot platform: https://github.com/imandel/tiltybot