Cosmos-H-Surgical

A surgical video world foundation model suite based on NVIDIA Cosmos and SurgWorld, part of the NVIDIA MedTech Open Models.

Overview

Cosmos-H-Surgical delivers high-quality video prediction and transfer for surgical scenes, including future-state simulation and control-conditioned generation across modalities. The Cosmos 3 release unifies Predict and Transfer in a single mixed-capability checkpoint that supports both image-to-video prediction and multi-modal control-based generation. This project was conducted by NVIDIA in collaboration with Chinese University of Hong Kong, National University of Singapore, and Shanghai Jiao Tong University.

News

• [July 2026] — Released Cosmos-H-Surgical, built on NVIDIA Cosmos 3, with Predict and Transfer unified in a single mixed-capability checkpoint.
• [July 2026] — Added OpenMDW-1.1 checkpoint editions for Predict and Transfer while retaining the original checkpoint paths and license terms.
• [March 2026] — Released SurgΣ: a large-scale multimodal surgical dataset and foundation model suite for surgical intelligence.
• [March 2026] — Released BSA: generalized recognition of basic surgical actions enabling skill assessment and VLM-based surgical planning.
• [March 2026] — Released Cosmos-H-Surgical-Predict and Cosmos-H-Surgical-Transfer as part of the NVIDIA MedTech Open Models.

The complete Cosmos 2.5 release remains available on the cosmos-2.5 branch and under the v0.2.0 tag.

Repository Structure

Cosmos-H-Surgical/
|-- cosmos_h_surgical/          # Surgical package and CLI
|-- datasets/                   # Synthetic toy post-training dataset
|-- examples/
|   |-- post_training/          # Public post-training recipes
|-- inputs/                     # Predict, Transfer, and prompt examples
|-- docs/                       # Setup, inference, and migration guides
|-- tests/                      # Release and integration checks
|-- pyproject.toml              # Package and uv configuration
|-- uv.lock                     # Reproducible dependency lock
|-- UPSTREAM.md                 # Framework provenance and update policy
`-- release-manifest.json       # Public release artifact inventory

Scaling World State Diversity Examples

scaling_world_state_diversity.mp4

Architecture

Cosmos-H-Surgical is a focused package layered on an immutable public revision of NVIDIA Cosmos Framework. The framework is installed as a commit-pinned dependency. See UPSTREAM.md.

Documentation

Guide	Description
Setup	CUDA 13/12.8 installation and environment verification.
Inference	Structured prompts, checkpoints, I2V, and all transfer controls.
Prompt upsampling	Convert short surgical prompts into Cosmos 3 JSON prompts.
Environment variables	Inference, prompt-upsampling, and training variables.
Troubleshooting	Installation, checkpoint, input, and distributed failures.
Code structure	Package architecture and framework ownership boundary.
Cosmos 2.5 migration	Archive locations, command mapping, and compatibility.
Post-training	Predict and Transfer LoRA recipes and dataset contracts.

License

Code and model weights in this release are provided under OpenMDW-1.1. Third-party software remains subject to its own license terms. See NOTICE and ATTRIBUTIONS.md.

Citation

@misc{he2026cosmoshsurgicallearningsurgicalrobot,
  title={Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling},
  author={Yufan He and Pengfei Guo and Mengya Xu and Zhaoshuo Li and Andriy Myronenko and Dillan Imans and Bingjie Liu and Dongren Yang and Mingxue Gu and Yongnan Ji and Yueming Jin and Ren Zhao and Baiyong Shen and Daguang Xu},
  year={2026},
  eprint={2512.23162},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2512.23162},
}

@misc{zeng2026surgsigma,
  title={Surg$\Sigma$: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence},
  author={Zhitao Zeng and Mengya Xu and Jian Jiang and Pengfei Guo and Yunqiu Xu and Zhu Zhuo and Chang Han Low and Yufan He and Dong Yang and Chenxi Lin and Yiming Gu and Jiaxin Guo and Yutong Ban and Daguang Xu and Qi Dou and Yueming Jin},
  year={2026},
  eprint={2603.16822},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2603.16822},
}

@misc{xu2026generalizedrecognitionbasicsurgicalactions,
  title={Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning},
  author={Mengya Xu and Daiyun Shen and Jie Zhang and Hon Chi Yip and Yujia Gao and Cheng Chen and Dillan Imans and Yonghao Long and Yiru Ye and Yixiao Liu and Rongyun Mai and Kai Chen and Hongliang Ren and Yutong Ban and Guangsuo Wang and Francis Wong and Chi-Fai Ng and Kee Yuan Ngiam and Russell H. Taylor and Daguang Xu and Yueming Jin and Qi Dou},
  year={2026},
  eprint={2603.12787},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.12787},
}

Resources

• Cosmos-H-Surgical Paper (arXiv)
• Basic Surgical Actions Dataset Paper (arXiv)
• HuggingFace Collection
• NVIDIA Cosmos Platform
• Cosmos-H-Surgical-Simulator — Sister repo (action-conditioned surgical simulation)
• NVIDIA MedTech Open Models