[Issue] Intermittent LowState Missing Causing Damping Mode During Workstation Deployment

[Ticotico410]

Hi,

I am encountering an intermittent LowState message loss issue when running SONIC's workstation-side C++ deployment (g1_deploy_onnx_ref). When the issue occurs, the safety logic is triggered and the robot switches to damping mode.

Setup

The deployment runs on an external workstation connected to the robot LAN. The DDS communication uses the robot network interface (e.g., enp4s0).
The tested input modes include:

• ZMQ manager input (planner mode)
• PICO-based whole-body teleoperation
  The image and hand services are running on the robot PC2 side. The overall system layout is similar to our previous teleoperation stack.

Observed Behavior
The issue happens intermittently and is not tied to a specific input mode. I have observed the same behavior in both planner mode (through ZMQ manager) and PICO whole-body teleoperation. This issue appears as long as the workstation-side C++ deployment process (g1_deploy_onnx_ref) is running. When the issue occurs, the deployment reports that LowState is missing or stale. The LowState age exceeds the configured timeout threshold, which triggers the safety handling path in g1_deploy_onnx_ref.cpp, causing the robot to switch into damping mode.

Additional Observations
The issue does not appear to be directly correlated with a specific teleoperation mode switch. A separate onboard FSM-based teleoperation and recording pipeline, using the same robot LAN and similar image/hand services, does not reproduce this LowState-loss and damping behavior. The main difference is that the workstation-side deployment runs an external high-frequency (~500 Hz) LowCmd publishing loop.

Questions
Has anyone encountered intermittent LowState dropout when running an offboard deployment setup (workstation to robot PC1 communication with rt/lowstate and rt/lowcmd at around 500 Hz)?

Are there any recommended checks or configurations for this issue, such as DDS NIC/IRQ affinity, CycloneDDS configuration, subscriber queue depth, CPU isolation or priority settings for the command writer thread, or possible interference from other DDS clients (e.g., audio/hand services) sharing the same network interface?

Are there any recommended timeout adjustments or mitigation strategies before moving the control loop fully onboard?

I can provide additional logs if needed, including LowState dropout timestamps, NIC statistics (ifconfig output), DDS configuration, and CPU/thread scheduling information.

Thanks!

[MazlumOzkul]

Hi,

We are experiencing the same issue in our setup. We run the SONIC controller from an external PC connected to the robot via Ethernet LAN. In some sessions, LowState intermittently becomes stale/missing, causing the deployment to trigger the safety timeout and shut down.

We have checked for network overload and CPU/GPU resource issues on the external PC, but none of these seem to be the cause. For us, the deployment stops roughly once in 20–30 sessions, causing the robot to switch to damping mode, which is quite dangerous.

We are not sure whether this issue also occurs when running the deployment directly on the robot PC2 (NX), as we have not tested that setup extensively. We also don't know whether this could be related to Unitree DDS.

If anyone has experienced this or has more information about the root cause, we would really appreciate it. We are also curious whether the SONIC developers are seeing the same LowState dropout issue.