SPOT Unity VR Interface

This Unity project provides the VR interface for SPOT: Spatial Perception-Oriented Long-Horizon Humanoid Teleoperation (paper, arXiv:2609.07933). It displays side-by-side binocular fisheye video on a virtual dome, supports IMU-based visual stabilization, and captures headset and controller poses for streaming or recording.

Training and teleoperation

The training and robot teleoperation processes are provided in Genesis-Humanoid. Follow that repository's instructions for training, deployment, and robot teleoperation. This Unity project supplies the operator's VR view and headset/controller input stream.

Requirements

• Unity 6000.3.10f1 (Unity 6).
• An OpenXR-compatible VR headset and controllers.
• For dome viewing: a camera device available to Unity, providing side-by-side binocular fisheye video. The default device name is OBS Virtual Camera.
• For stabilized viewing: a camera-mounted serial IMU. The project supports WT61C IMU data at 115200 baud.
• Optional: a body IMU, used to add body orientation to streamed or recorded data.

A body IMU is not required to use this Unity interface. You can stream headset/controller poses and record sequences without one. On the setup screen, turn Body IMU off and leave its port field unchanged; the port is ignored while the option is off. Camera IMU stabilization is configured separately through Dome (Cam + IMU).

Configure the setup screen

1. Connect your headset and controllers, make your camera feed available, and connect any IMUs you plan to use.
2. Open Assets/Scenes/Main.unity in Unity and enter Play mode.
3. Set the fields below to match your devices and network.
4. Select an operation to begin.

The prefilled values are examples for the included scene. Replace them with your own device names, serial ports, and destination IP address.

Setting	Default	What to enter or select
Dome (Cam + IMU)	On	Enable to display camera video with Start Teleop or Mock TWIST. Disable for pose streaming without the dome.
Camera name under Dome	OBS Virtual Camera	Enter the camera device name recognized by Unity. Use a side-by-side fisheye video feed.
Camera IMU port under Dome	COM8	Enter the camera IMU's serial port for stabilized Start Teleop mode. This port is not used by Mock TWIST.
Body IMU (optional)	On	Turn off if you do not have a body IMU. Enable only to include body orientation during streaming or recording.
Body IMU port	COM3	Required only when Body IMU is enabled. Otherwise, this field is ignored. IMU components use 115200 baud.
Target IP	192.168.1.2	Enter the numeric IP address that should receive pose packets. This field applies to streaming, not local recording.

Complete the settings before starting: the setup screen closes when you select an operation. Leaving a text field blank uses its default value. Settings are not saved between sessions. To change them, restart Play mode or the application.

Choose an operation

• Start Teleop: starts XR pose streaming. With Dome enabled, also starts camera video and camera IMU stabilization. Body IMU is controlled separately by its toggle.
• Mock TWIST: starts XR pose streaming and enables the restricted-view mask. Enable Dome to view video. Both eyes see the left camera image through a 50° horizontal, 16:9 viewing window. The simulated viewing delay is 0 seconds.
• Mock SONIC: starts XR pose streaming without dome video, regardless of the Dome toggle. Body IMU still follows its toggle.
• Start Recording: prepares local headset and controller pose recording, with optional body IMU input. This button does not start video or pose streaming. Follow the recording steps below to capture a sequence.

Record a sequence

1. Select Start Recording on the setup screen.
2. Squeeze the right controller trigger at least 80% to begin capture.
3. Release the trigger, then squeeze it again to stop and save.
4. Repeat to record additional sequences.

Stop and save before ending the session; an active recording is not saved automatically on exit.

Recordings are NumPy (.npy) files named sequence_<timestamp>.npy, saved in the PoseRecordings folder under Unity's Application.persistentDataPath. Check the Unity Console for the full save path. The target recording rate is 120 Hz.

Video and streaming

The camera capture requests 3840 × 1920 at 30 FPS; the actual resolution and frame rate depend on your camera device.

Streaming sends headset and controller poses, controller inputs, and optional body IMU angles to Target IP over UDP port 5005, at a target rate of 120 Hz. Keep the headset and both controllers tracked so pose packets can be sent.

Receive poses in Python

The included Receiver/XRClient.py receives Unity's UDP pose stream. Its coordinate-conversion helpers are included in Receiver/utils, so the receiver can be used from this project folder alone.

Run the receiver

Use Python 3.10 or newer. From this Unity project folder, install the Python dependency and start the receiver on the computer that will receive the poses:

python -m pip install -r Receiver/requirements.txt
python Receiver/XRClient.py

The receiver listens on UDP port 5005 on all local network interfaces. In Unity's setup screen, set Target IP to this computer's IP address (or 127.0.0.1 if Unity and the receiver run on the same computer). Select Start Teleop, Mock TWIST, or Mock SONIC. The terminal prints the latest received frame periodically. Start Recording saves locally and does not send packets to this receiver.

For a different listen address or port:

python Receiver/XRClient.py --host 0.0.0.0 --port 5005

--host is the receiver computer's local bind address. If you change --port, also change the Unity scene's PoseUdpSender target port in the Inspector. Allow incoming UDP traffic on that port. Stop the receiver with Ctrl+C; stop this preview receiver before another application binds the same port.

Use poses in Python code

With this project folder on your Python import path (for example, running a script from this folder):

from Receiver.XRClient import XRClient

client = XRClient(udp_port=5005, device="cpu")
try:
    frame = client.get_frame()  # Waits for the first packet, then returns the latest frame.
    print(frame["link_pos"])
    print(frame["link_quat"])
    print(frame["button_states"])
finally:
    client.shutdown()

Frames contain frame_id, recv_time, link_pos, link_quat, button_states, and body_imu. Pose tensors are ordered headset, left controller, right controller. Positions use Genesis axes (x forward, y left, z up), and quaternions use w, x, y, z order. Body IMU values are roll, pitch, and yaw in degrees; when Body IMU is disabled in a fresh Unity session, these fields are zero. No body IMU hardware is required.

The receiver exposes pose data for integration; running it alone does not command a robot. Use Genesis-Humanoid for the robot teleoperation workflow.

Troubleshooting

• No dome video: enable Dome (Cam + IMU), check the camera device name and active video feed, and use Start Teleop or Mock TWIST.
• IMU connection fails: check the selected serial port and device connection. Disable Body IMU if you are not using one.
• No pose packets: verify Target IP, UDP port 5005, and tracking for the headset and both controllers. Check the Unity Console for connection errors.
• No recording file: selecting Start Recording only prepares capture. Use the right trigger to start, then release and press again to stop and save.