diff --git a/README.md b/README.md index 11c30eac..5104f769 100644 --- a/README.md +++ b/README.md @@ -745,8 +745,8 @@ We are building examples that show Cosmos 3 capabilities end to end, including w | Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | | Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | | Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | -| Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | -| Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | +| Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | +| Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | | Inverse dynamics with Cosmos Framework | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) | | Inverse dynamics with vLLM-Omni | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_vllm.ipynb) | | Transfer with Cosmos Framework | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | diff --git a/cookbooks/cosmos3/generator/action/README.md b/cookbooks/cosmos3/generator/action/README.md index 49169f55..ba761ea2 100644 --- a/cookbooks/cosmos3/generator/action/README.md +++ b/cookbooks/cosmos3/generator/action/README.md @@ -51,6 +51,7 @@ fingers. | Autonomous vehicle | Ego pose (9D) | 9D | Meter | Normalization | 60 frames @ 10FPS | | [DROID](https://arxiv.org/abs/2403.12945) | End-effector pose (9D) + gripper grasp state (1D) | 10D | Meter | Multiview concatenation, `to-OpenCV`, normalization | 16 frames @ 15FPS | | UMI | End-effector pose (9D) + gripper grasp state (1D) | 10D | Meter | Normalization | 16 frames @ 20FPS | +| Human hand pose | Ego pose (9D) + right wrist (9D) + right fingertips (15D) + left wrist (9D) + left fingertips (15D) | 57D | Meter | Wrist-frame alignment, normalization | 16 frames @ 15FPS | Action data samples across different embodiments can be inspected interactively in the [Cosmos3 Action Viewer](https://huggingface.co/spaces/nvidia/Cosmos3-Action-Viewer) Hugging Face Space. @@ -73,7 +74,7 @@ torchrun --nproc-per-node=1 \ ``` The input spec pairs a start image with an action trajectory. The notebooks -assemble ready-to-run specs for AV, DROID, and UMI examples from the checked-in +assemble ready-to-run specs for AV, DROID, UMI, and human hand-pose examples from the checked-in assets under [`assets/`](./assets). Outputs are written under the framework checkout. @@ -83,7 +84,7 @@ The Cosmos Framework build their input spec, run inference, and visualize the generated videos: - [`run_fd_with_cosmos_framework.ipynb`](./run_fd_with_cosmos_framework.ipynb) — - forward dynamics for AV, DROID, and UMI robotics examples using Cosmos3-Nano. + forward dynamics for AV, DROID, UMI, and human hand-pose examples using Cosmos3-Nano. - [`run_id_with_cosmos_framework.ipynb`](./run_id_with_cosmos_framework.ipynb) — inverse dynamics, predicting ego-motion trajectories from input AV videos using Cosmos3-Nano. - [`run_policy_with_cosmos_framework.md`](./run_policy_with_cosmos_framework.md) - policy, predicting future observations and action trajectories for DROID robot using Cosmos3-Nano-Policy-DROID. @@ -114,7 +115,7 @@ generation (see [`run_fd_with_vllm.ipynb`](./run_fd_with_vllm.ipynb) and | `guidance_scale` | `1.0` | | `flow_shift` | `10.0` | -The notebooks build the full request body for AV, DROID, and UMI examples, +The notebooks build the full request body for AV, DROID, UMI, and human hand-pose examples, including autoregressive chunked generation for the robotics examples. ### VLLM-Omni Notebook Walkthrough @@ -123,7 +124,7 @@ The vLLM-Omni notebooks send requests through the OpenAI-compatible video API an write outputs under `outputs/cosmos3_action_vllm/`: - [`run_fd_with_vllm.ipynb`](./run_fd_with_vllm.ipynb) — forward dynamics for AV, - DROID, and UMI robotics examples. + DROID, UMI, and human hand-pose examples. - [`run_id_with_vllm.ipynb`](./run_id_with_vllm.ipynb) — inverse dynamics, predicting ego-motion trajectories from input AV videos. diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/data/chunk-000/file-000.parquet b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/data/chunk-000/file-000.parquet new file mode 100644 index 00000000..7b81f373 Binary files /dev/null and b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/data/chunk-000/file-000.parquet differ diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/episodes/chunk-000/file-000.parquet b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/episodes/chunk-000/file-000.parquet new file mode 100644 index 00000000..28615355 Binary files /dev/null and b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/episodes/chunk-000/file-000.parquet differ diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/info.json b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/info.json new file mode 100644 index 00000000..eb14fb34 --- /dev/null +++ b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/info.json @@ -0,0 +1,431 @@ +{ + "codebase_version": "v3.0", + "robot_type": "human_demo", + "repo_id": "local/human_hand_pose_validation_episode", + "fps": 30, + "total_episodes": 1, + "total_frames": 1440, + "total_tasks": 1, + "total_videos": 1, + "total_chunks": 1, + "chunks_size": 1000, + "features": { + "observation.images.main": { + "dtype": "video", + "shape": [ + 480, + 832, + 3 + ], + "names": [ + "height", + "width", + "channel" + ], + "video_info": { + "video.fps": 30, + "video.codec": "h264", + "video.pix_fmt": "yuv420p", + "video.is_depth_map": false, + "has_audio": false + } + }, + "subtask_index": { + "dtype": "int64", + "shape": [ + 1 + ], + "names": null + }, + "timestamp": { + "dtype": "float32", + "shape": [ + 1 + ], + "names": null + }, + "frame_index": { + "dtype": "int64", + "shape": [ + 1 + ], + "names": null + }, + "episode_index": { + "dtype": "int64", + "shape": [ + 1 + ], + "names": null + }, + "index": { + "dtype": "int64", + "shape": [ + 1 + ], + "names": null + }, + "task_index": { + "dtype": "int64", + "shape": [ + 1 + ], + "names": null + }, + "observation.state.hand_left_cam": { + "dtype": "float32", + "shape": [ + 63 + ], + "names": [ + "wrist_x", + "wrist_y", + "wrist_z", + "thumb_cmc_x", + "thumb_cmc_y", + "thumb_cmc_z", + "thumb_mcp_x", + "thumb_mcp_y", + "thumb_mcp_z", + "thumb_ip_x", + "thumb_ip_y", + "thumb_ip_z", + "thumb_tip_x", + "thumb_tip_y", + "thumb_tip_z", + "index_mcp_x", + "index_mcp_y", + "index_mcp_z", + "index_pip_x", + "index_pip_y", + "index_pip_z", + "index_dip_x", + "index_dip_y", + "index_dip_z", + "index_tip_x", + "index_tip_y", + "index_tip_z", + "middle_mcp_x", + "middle_mcp_y", + "middle_mcp_z", + "middle_pip_x", + "middle_pip_y", + "middle_pip_z", + "middle_dip_x", + "middle_dip_y", + "middle_dip_z", + "middle_tip_x", + "middle_tip_y", + "middle_tip_z", + "ring_mcp_x", + "ring_mcp_y", + "ring_mcp_z", + "ring_pip_x", + "ring_pip_y", + "ring_pip_z", + "ring_dip_x", + "ring_dip_y", + "ring_dip_z", + "ring_tip_x", + "ring_tip_y", + "ring_tip_z", + "pinky_mcp_x", + "pinky_mcp_y", + "pinky_mcp_z", + "pinky_pip_x", + "pinky_pip_y", + "pinky_pip_z", + "pinky_dip_x", + "pinky_dip_y", + "pinky_dip_z", + "pinky_tip_x", + "pinky_tip_y", + "pinky_tip_z" + ] + }, + "observation.state.hand_right_cam": { + "dtype": "float32", + "shape": [ + 63 + ], + "names": [ + "wrist_x", + "wrist_y", + "wrist_z", + "thumb_cmc_x", + "thumb_cmc_y", + "thumb_cmc_z", + "thumb_mcp_x", + "thumb_mcp_y", + "thumb_mcp_z", + "thumb_ip_x", + "thumb_ip_y", + "thumb_ip_z", + "thumb_tip_x", + "thumb_tip_y", + "thumb_tip_z", + "index_mcp_x", + "index_mcp_y", + "index_mcp_z", + "index_pip_x", + "index_pip_y", + "index_pip_z", + "index_dip_x", + "index_dip_y", + "index_dip_z", + "index_tip_x", + "index_tip_y", + "index_tip_z", + "middle_mcp_x", + "middle_mcp_y", + "middle_mcp_z", + "middle_pip_x", + "middle_pip_y", + "middle_pip_z", + "middle_dip_x", + "middle_dip_y", + "middle_dip_z", + "middle_tip_x", + "middle_tip_y", + "middle_tip_z", + "ring_mcp_x", + "ring_mcp_y", + "ring_mcp_z", + "ring_pip_x", + "ring_pip_y", + "ring_pip_z", + "ring_dip_x", + "ring_dip_y", + "ring_dip_z", + "ring_tip_x", + "ring_tip_y", + "ring_tip_z", + "pinky_mcp_x", + "pinky_mcp_y", + "pinky_mcp_z", + "pinky_pip_x", + "pinky_pip_y", + "pinky_pip_z", + "pinky_dip_x", + "pinky_dip_y", + "pinky_dip_z", + "pinky_tip_x", + "pinky_tip_y", + "pinky_tip_z" + ] + }, + "observation.state.hand_left_cam_rotation": { + "dtype": "float32", + "shape": [ + 84 + ], + "names": [ + "wrist_qx", + "wrist_qy", + "wrist_qz", + "wrist_qw", + "thumb_cmc_qx", + "thumb_cmc_qy", + "thumb_cmc_qz", + "thumb_cmc_qw", + "thumb_mcp_qx", + "thumb_mcp_qy", + "thumb_mcp_qz", + "thumb_mcp_qw", + "thumb_ip_qx", + "thumb_ip_qy", + "thumb_ip_qz", + "thumb_ip_qw", + "thumb_tip_qx", + "thumb_tip_qy", + "thumb_tip_qz", + "thumb_tip_qw", + "index_mcp_qx", + "index_mcp_qy", + "index_mcp_qz", + "index_mcp_qw", + "index_pip_qx", + "index_pip_qy", + "index_pip_qz", + "index_pip_qw", + "index_dip_qx", + "index_dip_qy", + "index_dip_qz", + "index_dip_qw", + "index_tip_qx", + "index_tip_qy", + "index_tip_qz", + "index_tip_qw", + "middle_mcp_qx", + "middle_mcp_qy", + "middle_mcp_qz", + "middle_mcp_qw", + "middle_pip_qx", + "middle_pip_qy", + "middle_pip_qz", + "middle_pip_qw", + "middle_dip_qx", + "middle_dip_qy", + "middle_dip_qz", + "middle_dip_qw", + "middle_tip_qx", + "middle_tip_qy", + "middle_tip_qz", + "middle_tip_qw", + "ring_mcp_qx", + "ring_mcp_qy", + "ring_mcp_qz", + "ring_mcp_qw", + "ring_pip_qx", + "ring_pip_qy", + "ring_pip_qz", + "ring_pip_qw", + "ring_dip_qx", + "ring_dip_qy", + "ring_dip_qz", + "ring_dip_qw", + "ring_tip_qx", + "ring_tip_qy", + "ring_tip_qz", + "ring_tip_qw", + "pinky_mcp_qx", + "pinky_mcp_qy", + "pinky_mcp_qz", + "pinky_mcp_qw", + "pinky_pip_qx", + "pinky_pip_qy", + "pinky_pip_qz", + "pinky_pip_qw", + "pinky_dip_qx", + "pinky_dip_qy", + "pinky_dip_qz", + "pinky_dip_qw", + "pinky_tip_qx", + "pinky_tip_qy", + "pinky_tip_qz", + "pinky_tip_qw" + ] + }, + "observation.state.hand_right_cam_rotation": { + "dtype": "float32", + "shape": [ + 84 + ], + "names": [ + "wrist_qx", + "wrist_qy", + "wrist_qz", + "wrist_qw", + "thumb_cmc_qx", + "thumb_cmc_qy", + "thumb_cmc_qz", + "thumb_cmc_qw", + "thumb_mcp_qx", + "thumb_mcp_qy", + "thumb_mcp_qz", + "thumb_mcp_qw", + "thumb_ip_qx", + "thumb_ip_qy", + "thumb_ip_qz", + "thumb_ip_qw", + "thumb_tip_qx", + "thumb_tip_qy", + "thumb_tip_qz", + "thumb_tip_qw", + "index_mcp_qx", + "index_mcp_qy", + "index_mcp_qz", + "index_mcp_qw", + "index_pip_qx", + "index_pip_qy", + "index_pip_qz", + "index_pip_qw", + "index_dip_qx", + "index_dip_qy", + "index_dip_qz", + "index_dip_qw", + "index_tip_qx", + "index_tip_qy", + "index_tip_qz", + "index_tip_qw", + "middle_mcp_qx", + "middle_mcp_qy", + "middle_mcp_qz", + "middle_mcp_qw", + "middle_pip_qx", + "middle_pip_qy", + "middle_pip_qz", + "middle_pip_qw", + "middle_dip_qx", + "middle_dip_qy", + "middle_dip_qz", + "middle_dip_qw", + "middle_tip_qx", + "middle_tip_qy", + "middle_tip_qz", + "middle_tip_qw", + "ring_mcp_qx", + "ring_mcp_qy", + "ring_mcp_qz", + "ring_mcp_qw", + "ring_pip_qx", + "ring_pip_qy", + "ring_pip_qz", + "ring_pip_qw", + "ring_dip_qx", + "ring_dip_qy", + "ring_dip_qz", + "ring_dip_qw", + "ring_tip_qx", + "ring_tip_qy", + "ring_tip_qz", + "ring_tip_qw", + "pinky_mcp_qx", + "pinky_mcp_qy", + "pinky_mcp_qz", + "pinky_mcp_qw", + "pinky_pip_qx", + "pinky_pip_qy", + "pinky_pip_qz", + "pinky_pip_qw", + "pinky_dip_qx", + "pinky_dip_qy", + "pinky_dip_qz", + "pinky_dip_qw", + "pinky_tip_qx", + "pinky_tip_qy", + "pinky_tip_qz", + "pinky_tip_qw" + ] + }, + "observation.state.camera_position": { + "dtype": "float32", + "shape": [ + 3 + ], + "names": [ + "tx", + "ty", + "tz" + ] + }, + "observation.state.camera_rotation": { + "dtype": "float32", + "shape": [ + 4 + ], + "names": [ + "qx", + "qy", + "qz", + "qw" + ] + } + }, + "splits": { + "validation": "0:1" + }, + "data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet", + "video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4", + "episodes_path": "meta/episodes/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet" +} diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/subtasks.parquet b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/subtasks.parquet new file mode 100644 index 00000000..415620ce Binary files /dev/null and b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/subtasks.parquet differ diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/tasks.parquet b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/tasks.parquet new file mode 100644 index 00000000..dfad33ee Binary files /dev/null and b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/meta/tasks.parquet differ diff --git a/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/videos/observation.images.main/chunk-000/file-000.mp4 b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/videos/observation.images.main/chunk-000/file-000.mp4 new file mode 100644 index 00000000..c7a59fc3 Binary files /dev/null and b/cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example/videos/observation.images.main/chunk-000/file-000.mp4 differ diff --git a/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb b/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb index 78ebf75e..07bc9198 100644 --- a/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb +++ b/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb @@ -22,7 +22,7 @@ "python -m cosmos_framework.scripts.inference\n", "```\n", "\n", - "Forward dynamics predicts future visual observations from an initial image and an action trajectory. This notebook is written as a first-run cookbook: clone or locate the framework source, install dependencies from scratch, verify the GPU environment, build AV, robotics, and UMI input specs, run Nano inference, and visualize generated videos.\n", + "Forward dynamics predicts future visual observations from an initial image and an action trajectory. This notebook is written as a first-run cookbook: clone or locate the framework source, install dependencies from scratch, verify the GPU environment, build AV, robotics, UMI, and human hand-pose input specs, run Nano inference, and visualize generated videos.\n", "\n", "Tested path from the audit:\n", "\n", @@ -1390,6 +1390,166 @@ "print(f\"UMI Generated Video: fps = {fps}, num_frames = {stitched_preview.shape[0]}\")\n", "media.show_video(stitched_preview, width=512, fps=fps)" ] + }, + { + "cell_type": "markdown", + "id": "hand-pose-fd-md", + "metadata": {}, + "source": [ + "## 9. Human Hand-Pose Forward Dynamics\n", + "\n", + "This example uses one validation episode of a person assembling a wooden object with a screwdriver. The checked-in LeRobot asset contains synchronized ego video, camera pose, and bimanual 21-keypoint hand annotations. `HumanHandPoseLeRobotDataset` converts them into the released 57D Cosmos hand-pose representation.\n" + ] + }, + { + "cell_type": "markdown", + "id": "hand-pose-fd-spec-md", + "metadata": {}, + "source": [ + "### Create the Hand-Pose Forward-Dynamics Input Spec\n", + "\n", + "The example samples the 30 FPS source episode at 15 FPS and prepares one 16-action chunk. The action layout is camera pose, right wrist and fingertips, then left wrist and fingertips.\n" + ] + }, + { + "cell_type": "code", + "id": "hand-pose-fd-spec-code", + "metadata": {}, + "source": [ + "# `resolve_input` and the COSMOS3_* paths come from the configuration cell.\n", + "import json\n", + "import os\n", + "import sys\n", + "\n", + "from PIL import Image\n", + "\n", + "if str(COSMOS3_REPO) not in sys.path:\n", + " sys.path.insert(0, str(COSMOS3_REPO))\n", + "\n", + "from cosmos_framework.data.vfm.action.datasets import HumanHandPoseLeRobotDataset\n", + "\n", + "hand_pose_dataset_root = resolve_input(\n", + " \"cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example\"\n", + ")\n", + "hand_pose_dataset = HumanHandPoseLeRobotDataset(root=hand_pose_dataset_root)\n", + "hand_pose_sample = hand_pose_dataset[0]\n", + "hand_pose_chunk_length = 16\n", + "assert tuple(hand_pose_sample[\"action\"].shape) == (hand_pose_chunk_length, 57)\n", + "\n", + "COSMOS3_INPUT_DIR.mkdir(parents=True, exist_ok=True)\n", + "hand_pose_action_path = COSMOS3_INPUT_DIR / \"human_hand_pose_action_chunk_00.json\"\n", + "hand_pose_action_path.write_text(json.dumps(hand_pose_sample[\"action\"].cpu().tolist()))\n", + "hand_pose_vision_path = COSMOS3_INPUT_DIR / \"human_hand_pose_input_chunk_00.png\"\n", + "first_frame = hand_pose_sample[\"video\"][:, 0].permute(1, 2, 0).cpu().numpy()\n", + "Image.fromarray(first_frame).save(hand_pose_vision_path)\n", + "\n", + "hand_pose_record = {\n", + " \"action_chunk_size\": hand_pose_chunk_length,\n", + " \"action_path\": str(hand_pose_action_path),\n", + " \"domain_name\": \"hand_pose\",\n", + " \"fps\": int(hand_pose_sample[\"conditioning_fps\"]),\n", + " \"image_size\": 480,\n", + " \"view_point\": hand_pose_sample[\"viewpoint\"],\n", + " \"model_mode\": \"forward_dynamics\",\n", + " \"name\": \"human_hand_pose_action_cond_chunk_00\",\n", + " \"prompt\": hand_pose_sample[\"ai_caption\"],\n", + " \"seed\": 0,\n", + " \"vision_path\": str(hand_pose_vision_path),\n", + "}\n", + "hand_pose_fd_input_path = COSMOS3_INPUT_DIR / \"action_forward_dynamics_hand_pose.jsonl\"\n", + "hand_pose_fd_input_path.write_text(json.dumps(hand_pose_record) + \"\\n\")\n", + "hand_pose_fd_output_dir = COSMOS3_OUTPUT_ROOT / \"action_forward_dynamics_hand_pose\"\n", + "\n", + "os.environ[\"COSMOS3_HAND_POSE_FD_INPUT\"] = str(hand_pose_fd_input_path)\n", + "os.environ[\"COSMOS3_HAND_POSE_FD_OUTPUT\"] = str(hand_pose_fd_output_dir)\n", + "\n", + "print(\"loaded hand-pose sample from:\", hand_pose_dataset_root)\n", + "print(\"caption:\", hand_pose_record[\"prompt\"])\n", + "print(\"action shape:\", tuple(hand_pose_sample[\"action\"].shape))\n", + "print(\"wrote hand-pose spec:\", hand_pose_fd_input_path)\n", + "print(hand_pose_fd_input_path.read_text())\n" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "hand-pose-fd-run-md", + "metadata": {}, + "source": [ + "### Run Hand-Pose Forward-Dynamics Inference\n" + ] + }, + { + "cell_type": "code", + "id": "hand-pose-fd-run-code", + "metadata": {}, + "source": [ + "import os\n", + "import subprocess\n", + "\n", + "hand_pose_env = os.environ.copy()\n", + "hand_pose_env[\"CUDA_VISIBLE_DEVICES\"] = hand_pose_env.get(\"CUDA_VISIBLE_DEVICES\", \"0\")\n", + "hand_pose_env[\"MASTER_ADDR\"] = hand_pose_env.get(\"COSMOS3_MASTER_ADDR\", \"127.0.0.1\")\n", + "hand_pose_env[\"MASTER_PORT\"] = hand_pose_env.get(\"COSMOS3_NANO_IMAGE_MASTER_PORT\", \"29501\")\n", + "hand_pose_env[\"RANK\"] = \"0\"\n", + "hand_pose_env[\"WORLD_SIZE\"] = \"1\"\n", + "hand_pose_env[\"LOCAL_RANK\"] = \"0\"\n", + "hand_pose_env.setdefault(\"PYTORCH_CUDA_ALLOC_CONF\", \"expandable_segments:True\")\n", + "\n", + "subprocess.run(\n", + " [\n", + " str(COSMOS3_REPO / \".venv\" / \"bin\" / \"python\"),\n", + " \"-m\",\n", + " \"cosmos_framework.scripts.inference\",\n", + " \"--parallelism-preset=latency\",\n", + " \"--no-guardrails\",\n", + " \"-i\",\n", + " str(hand_pose_fd_input_path),\n", + " \"-o\",\n", + " str(hand_pose_fd_output_dir),\n", + " \"--checkpoint-path\",\n", + " os.environ[\"COSMOS3_CHECKPOINT_PATH\"],\n", + " \"--video-save-quality\",\n", + " \"8\",\n", + " \"--image_size\",\n", + " \"480\",\n", + " \"--seed\",\n", + " str(hand_pose_record[\"seed\"]),\n", + " \"--benchmark\",\n", + " ],\n", + " cwd=str(COSMOS3_REPO),\n", + " env=hand_pose_env,\n", + " check=True,\n", + ")\n", + "\n", + "hand_pose_generated_video = hand_pose_fd_output_dir / hand_pose_record[\"name\"] / \"vision.mp4\"\n", + "assert hand_pose_generated_video.exists(), f\"missing generated video: {hand_pose_generated_video}\"\n", + "print(\"generated:\", hand_pose_generated_video)\n" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "hand-pose-fd-preview-md", + "metadata": {}, + "source": [ + "### Visualize the Hand-Pose Generated Video\n" + ] + }, + { + "cell_type": "code", + "id": "hand-pose-fd-preview-code", + "metadata": {}, + "source": [ + "from IPython.display import Video, display\n", + "\n", + "assert hand_pose_generated_video.exists(), f\"missing: {hand_pose_generated_video}\"\n", + "display(Video(str(hand_pose_generated_video), embed=True))\n" + ], + "execution_count": null, + "outputs": [] } ], "metadata": { diff --git a/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb b/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb index 7daa89b8..38e0dfbd 100644 --- a/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb +++ b/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb @@ -5,7 +5,8 @@ "id": "license-header", "metadata": {}, "source": [ - "" + "" ] }, { @@ -27,7 +28,7 @@ "POST /v1/videos\n", "```\n", "\n", - "Forward dynamics predicts future visual observations from an initial image and an action trajectory. This notebook contains separate AV and robotics sections that each build their own input spec, run inference, and visualize generated videos.\n", + "Forward dynamics predicts future visual observations from an initial image and an action trajectory. This notebook contains separate AV, robotics, UMI, and human hand-pose sections that each build their own input spec, run inference, and visualize generated videos.\n", "\n", "Start the server in a terminal from the `cosmos` repo root. The container listens on port `8000`; Docker publishes it to host port `8001`, so the notebook uses `http://localhost:8001`.\n", "\n", @@ -1286,6 +1287,123 @@ "metadata": {}, "outputs": [], "source": [] + }, + { + "cell_type": "markdown", + "id": "fdvl-hand-pose-md", + "metadata": {}, + "source": [ + "## Human Hand Pose\n", + "\n", + "This section runs a 57D bimanual human hand-pose forward-dynamics example from a single validation episode. The source video and hand annotations are synchronized at 30 FPS and sampled at 15 FPS for Cosmos3-Nano.\n" + ] + }, + { + "cell_type": "markdown", + "id": "fdvl-hand-pose-spec-md", + "metadata": {}, + "source": [ + "### Create the Hand-Pose Forward-Dynamics Input Spec\n" + ] + }, + { + "cell_type": "code", + "id": "fdvl-hand-pose-spec-code", + "metadata": {}, + "source": [ + "import json\n", + "import sys\n", + "\n", + "from PIL import Image\n", + "\n", + "if str(COSMOS3_REPO) not in sys.path:\n", + " sys.path.insert(0, str(COSMOS3_REPO))\n", + "\n", + "from cosmos_framework.data.vfm.action.datasets import HumanHandPoseLeRobotDataset\n", + "\n", + "hand_pose_dataset_root = resolve_input(\n", + " \"cookbooks/cosmos3/generator/action/assets/human_hand_pose_lerobot_example\"\n", + ")\n", + "hand_pose_dataset = HumanHandPoseLeRobotDataset(root=hand_pose_dataset_root)\n", + "hand_pose_sample = hand_pose_dataset[0]\n", + "hand_pose_chunk_length = 16\n", + "assert tuple(hand_pose_sample[\"action\"].shape) == (hand_pose_chunk_length, 57)\n", + "\n", + "hand_pose_action_path = COSMOS3_INPUT_DIR / \"human_hand_pose_action_chunk_00.json\"\n", + "hand_pose_action_path.write_text(json.dumps(hand_pose_sample[\"action\"].cpu().tolist()))\n", + "hand_pose_vision_path = COSMOS3_INPUT_DIR / \"human_hand_pose_input_chunk_00.png\"\n", + "first_frame = hand_pose_sample[\"video\"][:, 0].permute(1, 2, 0).cpu().numpy()\n", + "Image.fromarray(first_frame).save(hand_pose_vision_path)\n", + "\n", + "hand_pose_record = {\n", + " \"action_chunk_size\": hand_pose_chunk_length,\n", + " \"action_path\": str(hand_pose_action_path),\n", + " \"domain_name\": \"hand_pose\",\n", + " \"fps\": int(hand_pose_sample[\"conditioning_fps\"]),\n", + " \"image_size\": 480,\n", + " \"view_point\": hand_pose_sample[\"viewpoint\"],\n", + " \"model_mode\": \"forward_dynamics\",\n", + " \"name\": \"human_hand_pose_action_cond_chunk_00\",\n", + " \"prompt\": hand_pose_sample[\"ai_caption\"],\n", + " \"seed\": 0,\n", + " \"vision_path\": str(hand_pose_vision_path),\n", + "}\n", + "hand_pose_fd_input_path = COSMOS3_INPUT_DIR / \"action_forward_dynamics_hand_pose.jsonl\"\n", + "hand_pose_fd_input_path.write_text(json.dumps(hand_pose_record) + \"\\n\")\n", + "hand_pose_fd_output_dir = COSMOS3_OUTPUT_ROOT / \"action_forward_dynamics_hand_pose\"\n", + "\n", + "print(\"loaded hand-pose sample from:\", hand_pose_dataset_root)\n", + "print(\"caption:\", hand_pose_record[\"prompt\"])\n", + "print(\"action shape:\", tuple(hand_pose_sample[\"action\"].shape))\n", + "print(hand_pose_fd_input_path.read_text())\n" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "fdvl-hand-pose-run-md", + "metadata": {}, + "source": [ + "### Run Hand-Pose Forward-Dynamics Inference\n" + ] + }, + { + "cell_type": "code", + "id": "fdvl-hand-pose-run-code", + "metadata": {}, + "source": [ + "hand_pose_result = submit_forward_dynamics(\n", + " hand_pose_record,\n", + " hand_pose_fd_output_dir,\n", + " disable_guardrails=True,\n", + ")\n", + "hand_pose_generated_video = hand_pose_result[\"video_path\"]\n", + "assert hand_pose_generated_video.exists(), f\"missing generated video: {hand_pose_generated_video}\"\n", + "print(\"generated:\", hand_pose_generated_video)\n" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "fdvl-hand-pose-preview-md", + "metadata": {}, + "source": [ + "### Visualize the Hand-Pose Generated Video\n" + ] + }, + { + "cell_type": "code", + "id": "fdvl-hand-pose-preview-code", + "metadata": {}, + "source": [ + "from IPython.display import Video, display\n", + "\n", + "display(Video(str(hand_pose_generated_video), embed=True))\n" + ], + "execution_count": null, + "outputs": [] } ], "metadata": {