Skip to content

docs(cosmos3): add hand pose forward dynamics example#253

Open
hans-yang-tps wants to merge 3 commits into
NVIDIA:mainfrom
hans-yang-tps:cookbook/hand-pose-fd
Open

docs(cosmos3): add hand pose forward dynamics example#253
hans-yang-tps wants to merge 3 commits into
NVIDIA:mainfrom
hans-yang-tps:cookbook/hand-pose-fd

Conversation

@hans-yang-tps

@hans-yang-tps hans-yang-tps commented Jul 1, 2026

Copy link
Copy Markdown

Description

Add a Cosmos3 Nano functional-dexterity hand-pose inference example to the Action cookbook, covering both Cosmos Framework and vLLM-Omni execution paths.

Depends on NVIDIA/cosmos-framework#71 for the public human hand-pose dataset definition.

Changes

  • Add one self-contained validation episode for the task unscrew nut with screwdriver as a LeRobot asset with one Parquet file and one MP4 file.
  • Reindex the public-facing episode metadata and remove source user IDs, original episode IDs, and private filesystem paths.
  • Add a hand-pose action-definition section to the Action cookbook README.
  • Add matching hand-pose setup, inference, and preview sections to run_fd_with_cosmos_framework.ipynb and run_fd_with_vllm.ipynb.

Validation

  • Selected the episode from the validation split, not the training split.
  • Asset integrity: 1,440 video frames at 30 FPS, 832x480; one matching Parquet episode.
  • Cosmos Framework notebook: 45 cells, 18 code cells, zero errors, and zero unexecuted non-empty code cells.
  • vLLM-Omni notebook: 38 cells, 17 code cells, zero errors; only the two pre-existing empty placeholder cells were unexecuted.
  • Hand-pose output through each path: 17 frames at 15 FPS, 832x480.
  • Notebook sources contain no stored outputs or execution counts.

@hans-yang-tps

Copy link
Copy Markdown
Author

Need Michael(Zhengyu) Huang's help to review it.

@hans-yang-tps hans-yang-tps marked this pull request as draft July 1, 2026 09:59
Signed-off-by: Hans Yang <hayang@nvidia.com>
@hans-yang-tps hans-yang-tps force-pushed the cookbook/hand-pose-fd branch from 8ca5456 to 274e84e Compare July 2, 2026 01:40
@michaelhu-hash

Copy link
Copy Markdown

We should better also update the readme in here to include the hand pose eg.

cosmos/README.md

Lines 748 to 749 in 274e84e

| Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) |
| Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) |

@lzhang57 lzhang57 requested review from lfengad, lzhang57 and mli0603 July 2, 2026 07:07
@hans-yang-tps hans-yang-tps marked this pull request as ready for review July 9, 2026 08:22
@hans-yang-tps

Copy link
Copy Markdown
Author

Also removed Mecka keywords.

@lfengad lfengad left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, could we make a test run regarding all these newly added code/notebooks, to execute from a clean env for a further verification of the updated workflow? THX!

"if str(COSMOS3_REPO) not in sys.path:\n",
" sys.path.insert(0, str(COSMOS3_REPO))\n",
"\n",
"from cosmos_framework.data.vfm.action.datasets import HumanHandPoseLeRobotDataset\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The vfm related folder already renamed to generator, here might need corresponding change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants