CrashTwin: A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

Nuo Chen^1*, Lulin Liu^1,2*, Zihao Li¹, Ziyao Zeng³, Zihao Zhu¹, Wenyan Cong⁴, Junyuan Hong⁴, Yunhao Yang⁴, Zhengzhong Tu¹, Yan Wang⁵, Boris Ivanovic⁵, Marco Pavone^5,6, Zhangyang Wang⁴, Yang Zhou¹, Zhiwen Fan^1
1Texas A&M University    ²University of Minnesota    ³Yale University    ⁴University of Texas at Austin    ⁵NVIDIA    ⁶Stanford University
^*Equal contribution.

Overview

Official code for CrashTwin, a physics-grounded evaluation framework for world-model collision rollouts. Given a model's generated collision videos on CrashTwin-Eval, the evaluator reconstructs metric-scale physical attributes from monocular videos and reports the diagnostic scores used in the paper.

CrashTwin-Eval contains 300 synthetic and 44 real-world collision videos. Each clip contains two collision actors and is evaluated with the same public reconstruction-and-scoring protocol.

Use CrashTwin-Data for training and post-training data. Use CrashTwin-Eval for the fixed 344-scenario evaluation set used for quantitative benchmarking.

Installation

The host machine needs Bash, Docker, NVIDIA Container Toolkit, and huggingface-cli. All CrashTwin Python code runs inside Docker.

git clone https://github.com/phai-lab/CrashTwin.git
cd CrashTwin

docker pull nuochen1203/crashtwin-preprocess:v1.0.0
docker pull nuochen1203/crashtwin-reconstruct:v1.0.0

The Docker images are hosted on Docker Hub:

• Preprocessing image: https://hub.docker.com/r/nuochen1203/crashtwin-preprocess
• Reconstruction image: https://hub.docker.com/r/nuochen1203/crashtwin-reconstruct

Data Preparation

Training and post-training data are released separately as CrashTwin-Data:

https://huggingface.co/datasets/nnuochen/crashtwin_data/blob/main/README.md

To run the public evaluation protocol, download the CrashTwin-Eval files used for quantitative benchmarking from Hugging Face:

https://huggingface.co/datasets/nnuochen/crashtwin_eval

From the repository root, run:

huggingface-cli download nnuochen/crashtwin_eval \
  --repo-type dataset \
  --local-dir .

The downloaded files include the benchmark CSV, per-video metadata, synthetic first-frame images, first-frame processing scripts, and a checkpoint archive. The evaluator does not download checkpoints automatically; unzip the checkpoint archive before running evaluation:

unzip checkpoints/checkpoints.zip -d .

Data Layout

After downloading CrashTwin-Eval and adding one model's generated videos, the repository should look like this:

CrashTwin/
├── benchmark/
│   ├── crashtwin_eval.csv
│   ├── auto_json/
│   │   └── <video_id>_auto.json
│   └── vehicle_specs/
│       └── <video_id>_vehicle_specs.json
├── checkpoints/
│   ├── droid.pth
│   ├── metric_depth_vit_giant2_800k.pth
│   ├── nuScenes_3Dtracking.pth
│   └── searaft/
│       └── Tartan-C-T-TSKH-kitti432x960-M.pth
├── first_frames/
│   └── <video_id>_frame_raw.png
├── assets/
│   └── trim_head_tail_package.zip
└── predictions/
    └── <model_name>/
        ├── <video_id>.mp4
        ├── <video_id>.mp4
        └── ...

The expected video_id values are listed in benchmark/crashtwin_eval.csv. Generated videos may use any resolution or frame rate; the evaluator normalizes them internally. Camera intrinsics are estimated from each input video, so fixed camera parameters are not required.

Run Evaluation

Evaluate one model from the host with:

bash scripts/evaluate.sh \
  --method-name <model_name> \
  --predictions predictions/<model_name> \
  --output outputs/<model_name> \
  --gpus 0,1,2,3

scripts/evaluate.sh starts the released Docker images with docker run. It mounts the repository root to /crashtwin inside each container, so benchmark/, checkpoints/, predictions/, and outputs/ must all live inside the cloned repository. The script also mounts .cache/ to /cache so downloaded model-hub files are reused across runs.

Use --gpus 0 for a single GPU, or pass a comma-separated list such as --gpus 0,1,2,3 for multiple GPUs. For multi-GPU runs, the script splits the benchmark rows across GPUs and starts one Docker container per GPU. Logs are written to outputs/<model_name>/logs/.

Outputs

The main result files are:

outputs/<model_name>/
├── per_video_metrics.csv
├── summary_metrics.csv
├── failed_videos.csv
├── logs/
└── per_video/

summary_metrics.csv reports the aggregate CrashTwin-Eval scores. per_video_metrics.csv contains one row per evaluated video, and failed_videos.csv records clips that did not complete. Intermediate preprocessing and reconstruction outputs are stored under outputs/<model_name>/per_video/.

Each completed per-video folder keeps the JSON files used for scoring, the bounding-box anchor PNG, the tracking/SAM2 videos, the Kalman-smoothed trajectory video, and the visualization.

Acknowledgements

CrashTwin builds on the following open-source projects:

• DROID-SLAM for camera motion estimation.
• droid_metric for DROID-SLAM and metric-depth integration utilities.
• Metric3D for metric depth reconstruction.
• MapAnything for intrinsic estimation.
• SAM 2 for video object segmentation.
• SEA-RAFT for optical flow.
• CenterTrack for monocular 3D vehicle detection and tracking.
• OpenCLIP for appearance-feature extraction.

We thank the authors of these projects for releasing their code and models. Please refer to the corresponding files under third_party/ for upstream licenses and notices.