Skip to content

Commit daf3899

Browse files
ZhengyiLuoclaude
andcommitted
SONIC training release: code, data, docs, H2 support, and pre-release UX fixes
Training & Finetuning: - Full SONIC training code with Isaac Lab (sonic_release, sonic_bones_seed configs) - Training checkpoint and SMPL motion data on HuggingFace (nvidia/GEAR-SONIC) - download_from_hf.py: --training, --sample, --no-smpl modes - Sample data (1 walking sequence) for quick-start testing - Data processing pipeline (Bones-SEED conversion, SOMA extraction, filtering) - Evaluation and ONNX export scripts H2 Robot Support: - H2 URDF, meshes, MJCF model files - H2 robot mapping, order converter, experiment config (sonic_h2.yaml) Codebase Cleanup: - Remove unused grasp/object manipulation code (~3200 lines) - Remove compliance/CHIP and visual domain randomization code - Rename unitree_description to robot_description - BeyondMimic attribution in URDF and g1.py - Remove hardcoded nv-gear wandb entity Documentation: - Training on New Embodiments guide (KP/KD tuning, body names, retargeting) - Configuration guide (Hydra hierarchy, all tunable params, common recipes) - Troubleshooting guide (12 common issues from GitHub issues) - Coordinate frame and quaternion conventions reference - Download models page (checkpoint, SMPL data, sample data) - CONTRIBUTING.md and SECURITY.md Pre-release UX Fixes: - Isaac Lab import check in train/eval scripts with clear error message - Git LFS warning in README Setup section - check_environment.py pre-flight validation script - Fix pyproject.toml: Python >=3.10, torch>=2.4.0, trl==0.28.0, add easydict/loguru - Fix README mailto link - "Which environment do I need?" guide in README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3d3e593 commit daf3899

885 files changed

Lines changed: 91517 additions & 2497 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ instance/
6767
.scrapy
6868

6969
# Sphinx documentation
70-
70+
data/
7171

7272
# PyBuilder
7373
.pybuilder/
@@ -171,6 +171,11 @@ huggingface/
171171

172172
# Model checkpoints (download via download_from_hf.py)
173173
*.onnx
174+
*.pt
175+
*.pth
176+
*.ckpt
177+
*.safetensors
178+
*.engine
174179
!decoupled_wbc/sim2mujoco/resources/robots/g1/policy/GR00T-WholeBodyControl-Balance.onnx
175180
!decoupled_wbc/sim2mujoco/resources/robots/g1/policy/GR00T-WholeBodyControl-Walk.onnx
176181

@@ -188,7 +193,26 @@ external_dependencies/XRoboToolkit-PC-Service-Pybind/
188193
# UV
189194
uv.lock
190195

196+
# Virtual environments (created by install_scripts/)
197+
.venv_sim/
198+
.venv_teleop/
199+
.venv_data_collection/
200+
.venv_camera/
201+
191202
# XRoboToolkit-PC-Service-Pybind
192203
xrobotoolkit_sdk.cpython-*-*-*.so
193204
teleop_vids/
194-
*.code-workspace
205+
*.code-workspace
206+
# Motion data (large local datasets, not tracked in git)
207+
data/motion_lib_bones_seed
208+
209+
# Training output logs
210+
logs_rl/
211+
logs_eval/
212+
213+
# Model checkpoints directory
214+
models/
215+
216+
# Downloaded from HuggingFace (hf download)
217+
sample_data/
218+
sonic_release/

‎CONTRIBUTING.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Contributing to GR00T-WholeBodyControl
2+
3+
We welcome contributions from the community! Here's how to get started.
4+
5+
## Reporting Issues
6+
7+
- Search [existing issues](https://github.com/NVlabs/GR00T-WholeBodyControl/issues) first
8+
- Open a new issue with a clear description, error messages, and steps to reproduce
9+
- Include your Python version, OS, GPU, and Isaac Lab version
10+
11+
## Pull Requests
12+
13+
1. Fork the repository
14+
2. Create a feature branch (`git checkout -b my-feature`)
15+
3. Make your changes
16+
4. Run the pre-flight check: `python check_environment.py`
17+
5. Commit and push to your fork
18+
6. Open a pull request against `main`
19+
20+
### Guidelines
21+
22+
- Keep PRs focused on a single change
23+
- Follow existing code style (no linter is enforced, but be consistent)
24+
- Update documentation if your change affects user-facing behavior
25+
- Add yourself to the PR description if you'd like credit
26+
27+
## Development Setup
28+
29+
See the [Installation Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_training.html)
30+
for setting up the training environment.
31+
32+
## Questions
33+
34+
For questions, open a [GitHub Discussion](https://github.com/NVlabs/GR00T-WholeBodyControl/issues)
35+
or contact [gear-wbc@nvidia.com](mailto:gear-wbc@nvidia.com).
36+
37+
## License
38+
39+
By contributing, you agree that your contributions will be licensed under the
40+
[Apache 2.0 License](LICENSE).

‎README.md‎

Lines changed: 75 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
<div align="center">
1111

1212
[![License](https://img.shields.io/badge/License-Apache%202.0-76B900.svg)](LICENSE)
13-
[![IsaacLab](https://img.shields.io/badge/IsaacLab-2.3.0-orange.svg)](https://github.com/isaac-sim/IsaacLab/releases/tag/v2.3.0)
13+
[![IsaacLab](https://img.shields.io/badge/IsaacLab-2.3.2-orange.svg)](https://github.com/isaac-sim/IsaacLab/releases/tag/v2.3.2)
1414
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-76B900.svg)](https://nvlabs.github.io/GR00T-WholeBodyControl/)
1515

1616
</div>
@@ -29,17 +29,19 @@ This is the codebase for the **GR00T Whole-Body Control (WBC)** projects. It hos
2929

3030
## News
3131

32-
- **[2026-03-24]** Update the C++ inference stack: motor error monitoring and temperature reporting (with TTS alerts and MuJoCo heatmap visualization); support streamed token input via ZMQ protocol v4; idle-mode error-based readaptation; TRT engine output now co-located with ONNX model; **ZMQ header size changed to 1280 bytes**
33-
- **[2026-03-16]** [BONES-SEED](https://huggingface.co/datasets/bones-studio/seed) is now open-sourced! A large-scale human motion dataset (142K+ motions, ~288 hours) with Unitree G1 MuJoCo-compatible trajectories (a large subset of SONIC training data!).
34-
- **[2026-02-19]** Released GEAR-SONIC with pretrained policy checkpoints, C++ inference stack, VR teleoperation stack, and documentation.
35-
- **[2025-11-12]** Initial release of GR00T-WholeBodyControl with Decoupled WBC for GR00T N1.5 and N1.6.
32+
- **[2026-04-10]** Released SONIC training code, checkpoint, and SMPL data on [HuggingFace](https://huggingface.co/nvidia/GEAR-SONIC). Train from scratch or finetune. Added support for additional embodiments. See [Training Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/user_guide/training.html).
33+
- **[2026-03-24]** C++ inference stack update: motor error monitoring, TTS alerts, ZMQ protocol v4, idle-mode readaptation. **ZMQ header size changed to 1280 bytes.**
34+
- **[2026-03-16]** [BONES-SEED](https://huggingface.co/datasets/bones-studio/seed) open-sourced — 142K+ human motions (~288 hours) with G1 MuJoCo trajectories.
35+
- **[2026-02-19]** Released GEAR-SONIC: pretrained checkpoints, C++ inference, VR teleoperation, and documentation.
36+
- **[2025-11-12]** Initial release with Decoupled WBC for GR00T N1.5 and N1.6.
3637

3738
## Table of Contents
3839

3940
- [News](#news)
4041
- [GEAR-SONIC](#gear-sonic)
4142
- [VR Whole-Body Teleoperation](#vr-whole-body-teleoperation)
4243
- [Kinematic Planner](#kinematic-planner)
44+
- [SONIC Training](#sonic-training)
4345
- [TODOs](#todos)
4446
- [What's Included](#whats-included)
4547
- [Setup](#setup)
@@ -66,7 +68,7 @@ This is the codebase for the **GR00T Whole-Body Control (WBC)** projects. It hos
6668

6769
SONIC is a humanoid behavior foundation model that gives robots a core set of motor skills learned from large-scale human motion data. Rather than building separate controllers for predefined motions, SONIC uses motion tracking as a scalable training task, enabling a single unified policy to produce natural, whole-body movement and support a wide range of behaviors — from walking and crawling to teleoperation and multi-modal control. It is designed to generalize beyond the motions it has seen during training and to serve as a foundation for higher-level planning and interaction.
6870

69-
In this repo, we will release SONIC's training code, deployment framework, model checkpoints, and teleoperation stack for data collection.
71+
In this repo, we release SONIC's training code, deployment framework, model checkpoints, and teleoperation stack for data collection.
7072

7173

7274
## VR Whole-Body Teleoperation
@@ -157,13 +159,50 @@ SONIC includes a kinematic planner for real-time locomotion generation — choos
157159
</table>
158160
</div>
159161

162+
## SONIC Training
163+
164+
SONIC can be trained from scratch on the [Bones-SEED](https://huggingface.co/datasets/bones-studio/seed)
165+
motion capture dataset (142K+ motions, ~288 hours, Unitree G1 retargeted), or finetuned
166+
from the released checkpoint on [Hugging Face](https://huggingface.co/nvidia/GEAR-SONIC).
167+
168+
### Quick start
169+
170+
```bash
171+
# Install training dependencies (Isaac Lab must be installed separately — see docs)
172+
pip install -e "gear_sonic/[training]"
173+
174+
# Download checkpoint + SMPL data from Hugging Face
175+
pip install huggingface_hub
176+
python download_from_hf.py --training
177+
178+
# Download Bones-SEED G1 CSVs from bones-studio.ai/seed, then convert and filter
179+
python gear_sonic/data_process/convert_soma_csv_to_motion_lib.py \
180+
--input /path/to/bones_seed/g1/csv/ \
181+
--output data/motion_lib_bones_seed/robot --fps 30 --fps_source 120 --individual --num_workers 16
182+
python gear_sonic/data_process/filter_and_copy_bones_data.py \
183+
--source data/motion_lib_bones_seed/robot --dest data/motion_lib_bones_seed/robot_filtered
184+
185+
# Finetune from released checkpoint (64+ GPUs recommended)
186+
accelerate launch --num_processes=8 gear_sonic/train_agent_trl.py \
187+
+exp=manager/universal_token/all_modes/sonic_release \
188+
+checkpoint=sonic_release/last.pt \
189+
num_envs=4096 headless=True \
190+
++manager_env.commands.motion.motion_lib_cfg.motion_file=data/motion_lib_bones_seed/robot_filtered \
191+
++manager_env.commands.motion.motion_lib_cfg.smpl_motion_file=data/smpl_filtered
192+
```
193+
194+
For the full guide including multi-node training, evaluation, ONNX export, and SOMA encoder setup:
195+
📖 [Installation (Training)](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_training.html) |
196+
[Training Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/user_guide/training.html)
197+
198+
160199
## TODOs
161200

162201
- [x] Release pretrained SONIC policy checkpoints
163202
- [x] Open source C++ inference stack
164203
- [x] Setup documentation
165204
- [x] Open source teleoperation stack and demonstration scripts
166-
- [ ] Release training scripts and recipes for motion imitation and fine-tuning
205+
- [x] Release training scripts and recipes for motion imitation and fine-tuning
167206
- [ ] Open source large-scale data collection workflows and fine-tuning VLA scripts.
168207
- [ ] Publish additional preprocessed large-scale human motion datasets
169208

@@ -174,17 +213,38 @@ SONIC includes a kinematic planner for real-time locomotion generation — choos
174213
This release includes:
175214

176215
- **`gear_sonic_deploy`**: C++ inference stack for deploying SONIC policies on real hardware
177-
- **`gear_sonic`**: Teleoperation stack for collecting demonstration data (no training code, YET.)
216+
- **`gear_sonic`**: Full SONIC training stack — PPO training, data processing pipeline, and configuration system for training on Bones-SEED and custom motion datasets
178217

179218
### Setup
180219

181-
Clone the repository with Git LFS:
220+
> **Git LFS required.** This repo contains large binary assets (meshes, ONNX
221+
> models). Without Git LFS, you will get small pointer files instead of actual
222+
> data, causing silent failures. Install Git LFS first if you don't have it:
223+
> `sudo apt install git-lfs && git lfs install`
224+
182225
```bash
183226
git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git
184227
cd GR00T-WholeBodyControl
185228
git lfs pull
229+
230+
# Verify your environment
231+
python check_environment.py
186232
```
187233

234+
### Which environment do I need?
235+
236+
| I want to... | Environment | How to install |
237+
|---|---|---|
238+
| **Train / finetune SONIC** | Isaac Lab's Python env | [Install Isaac Lab](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html), then `pip install -e "gear_sonic/[training]"` |
239+
| **Run MuJoCo simulation** | `.venv_sim` (auto-created) | `bash install_scripts/install_mujoco_sim.sh` |
240+
| **VR teleoperation** | `.venv_teleop` (auto-created) | `bash install_scripts/install_pico.sh` |
241+
| **Collect data** | `.venv_data_collection` (auto-created) | `bash install_scripts/install_data_collection.sh` |
242+
| **Deploy on real robot** | C++ build | See [deployment docs](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_deploy.html) |
243+
244+
Each use case has its own lightweight environment. The install scripts use `uv`
245+
and create isolated venvs automatically — you don't need to manage them manually.
246+
Training is the only one that requires Isaac Lab (installed separately).
247+
188248
## Documentation
189249

190250
📚 **[Full Documentation](https://nvlabs.github.io/GR00T-WholeBodyControl/)**
@@ -200,6 +260,11 @@ git lfs pull
200260
- [ZMQ Communication](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/zmq.html)
201261
- [ZMQ Manager / PICO VR](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/vr_wholebody_teleop.html)
202262

263+
### Training
264+
- [Installation (Training)](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_training.html)
265+
- [Training Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/user_guide/training.html)
266+
- [Training Data](https://nvlabs.github.io/GR00T-WholeBodyControl/user_guide/training_data.html)
267+
203268
### Best Practices
204269
- [Teleoperation](https://nvlabs.github.io/GR00T-WholeBodyControl/user_guide/teleoperation.html)
205270

@@ -242,7 +307,7 @@ All required legal documents, including the Apache 2.0 license, 3rd-party attrib
242307

243308
## Support
244309

245-
For questions and issues, please contact the GEAR WBC team at [gear-wbc@nvidia.com](gear-wbc@nvidia.com) to provide feedback!
310+
For questions and issues, please contact the GEAR WBC team at [gear-wbc@nvidia.com](mailto:gear-wbc@nvidia.com) to provide feedback!
246311

247312
## Decoupled WBC
248313

‎SECURITY.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Security Policy
2+
3+
## Reporting a Vulnerability
4+
5+
NVIDIA is committed to the security of our products. If you believe you have
6+
found a security vulnerability in this project, please report it through
7+
[NVIDIA's coordinated vulnerability disclosure process](https://www.nvidia.com/en-us/security/)
8+
rather than opening a public issue.
9+
10+
You can also email [psirt@nvidia.com](mailto:psirt@nvidia.com).

0 commit comments

Comments
 (0)