WIP: Formal pull request containing CR3DT Changes

.gitignore

@@ -75,9 +75,19 @@ vis4d.egg-info
 # local test logs and scripts
 log/
 /*.sh
+!eval_nusc.sh
+!main_cc3dtpp_track.sh
+!full_eval.sh
+!sec_docker.sh
 docs/source/api/*
 docs/source/tutorials/.ipynb_checkpoints/*
 wandb/
 
 # No lightning logs
 lightning_logs/
+
+# data folders
+vis4d/data/nuscenes/
+vis4d/data/nuscenes_mini/
+vis4d/data/nuscenes_test/
+qd_models/

Dockerfile.cc3dtpp

@@ -0,0 +1,33 @@
+# freezing to the correct cuda installation
+FROM nvidia/cuda:11.8.0-devel-ubuntu20.04
+
+ADD . /root/cc3dt
+
+ARG DEBIAN_FRONTEND=noninteractive
+RUN apt-get clean && apt-get update &&\
+    apt-get install -y curl git libgl1-mesa-glx libglib2.0-0 
+
+# CONDA
+RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
+    chmod +x ~/miniconda.sh && \
+    ~/miniconda.sh -b -p /opt/conda && \
+    rm ~/miniconda.sh && \
+    /opt/conda/bin/conda install -y python=3.9 conda-build pyyaml numpy ipython cython typing typing_extensions mkl mkl-include ninja && \
+    /opt/conda/bin/conda clean -ya
+
+
+# CONDA ENVIRONMENT
+
+RUN /opt/conda/bin/conda init bash 
+
+RUN /opt/conda/bin/conda create --name cc3dt python=3.9
+
+RUN . ~/.bashrc && conda activate cc3dt && pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
+
+RUN git clone https://github.com/SysCV/vis4d_cuda_ops.git && cd vis4d_cuda_ops \
+    && . ~/.bashrc && conda activate cc3dt && pip install -v -e .
+
+RUN . ~/.bashrc && conda activate cc3dt && cd /root/cc3dt/vis4d && pip install vis4d && pip install motmetrics==1.1.3 && pip install pandas==1.1.5
+
+# Download the QD model
+RUN mkdir -p /root/cc3dt/qd_models && cd /root/cc3dt/qd_models && curl -O https://dl.cv.ethz.ch/vis4d/cc_3dt/cc_3dt_frcnn_r101_fpn_24e_nusc_f24f84.pt

README.md

@@ -1,131 +1,96 @@
-<p align="center">
-    <!-- pypi-strip -->
-    <picture>
-    <!-- /pypi-strip -->
-    <img alt="vis4d" src="https://dl.cv.ethz.ch/vis4d/vis4d_logo.svg" width="400">
-    <!-- pypi-strip -->
-    </picture>
-    <!-- /pypi-strip -->
-    <br/>
-    A modular library for 4D scene understanding
-</p>
-
-## Quickstart
-
-You can checkout our [documentation](https://docs.vis.xyz/4d/index.html).
-
-You can use the [template](https://github.com/SysCV/vis4d-template) here to start your own project with Vis4D.
+# CC-3DT++: The Velocity-Similarity Enhanced Tracker of CR3DT
+This packages contains the isolated CC-3DT++ tracker of [CR3DT](https://arxiv.org/abs/2403.15313) and is meant to be run in combination with the [CR3DT](https://github.com/ETH-PBL/CR3DT) detector.
 
 ## Installation
 
-Installation is as easy as
-
+Step 1: Clone the repository
 ```bash
-python3 -m pip install vis4d
+git clone https://github.com/ETH-PBL/cc-3dt-pp.git
+cd cc-3dt-pp
 ```
-
-[For more detailed information, check out our installation guide](docs/source/user_guide/install.rst)
-
-## Basic CLI usage
-
-- To train a model, e.g. Faster-RCNN on COCO
-
+Step 2: Build the docker image
 ```bash
-# vis4d.engine
-vis4d fit --config vis4d/zoo/faster_rcnn/faster_rcnn_coco.py --gpus 1
-
-# vis4d.pl
-vis4d-pl fit --config vis4d/zoo/faster_rcnn/faster_rcnn_coco.py --gpus 1
+docker build -t cc3dt -f Dockerfile.cc3dtpp .
 ```
 
-- To test a model
-
-```bash
-# vis4d.engine
-vis4d test --config vis4d/zoo/faster_rcnn/faster_rcnn_coco.py --gpus 1
+Step 3. Create the folder structure below anywhere on your file system. You can chose to populate the folders with the nuScenes dataset and our provided pkl-files and checkpoints ([Google Drive with checkpoint and pkls](https://drive.google.com/drive/folders/1gHPZMUCDObDTHqbU_7Drw0CILx4pu_7i)), or just with the dataset and to create any pkl-files and checkpoints yourself. At least one of the three dataset folders (`v1.0-mini`, `v1.0-trainval`, or `v1.0-test`) needs to be populated.
+```shell script
+...
+├── <your data directory>
+│   ├── v1.0-mini
+│   ├── v1.0-trainval
+│   ├── v1.0-test
+│   └── checkpoints
+└ ...
+```
 
-# vis4d.pl
-vis4d-pl test --config vis4d/zoo/faster_rcnn/faster_rcnn_coco.py --gpus 1
+Step 4. Start the docker container with the necessary flags using the provided utility script. After that you can open a second interactive shell to the docker using `sec_docker.sh`.
+```shell script
+./main_cc3dtpp_track.sh <path to your data directory>
+./sec_docker.sh
 ```
 
-## DDP
+Step 5. Make sure you have gnerated a results json file for the nuScenes dataset. You can use the following command to generate the results json file and place it in the corresponding datasets folder. Or you can download our provided detection results [here](https://drive.google.com/drive/folders/1gHPZMUCDObDTHqbU_7Drw0CILx4pu_7i).
 
-### Training
+Step 6. Inside the docker container, you can run the following command to evaluate the tracking performance on nuscense dataset. This will mount the datasets into the container in the correct naming convention.
+```shell script
+# For mini
+./full_eval.sh mini vis4d/data/nuscenes_mini/<results json file>
 
-- Local machine / SLURM interactivate job (`job-name=bash`)
+# For trainval
+./full_eval.sh trainval vis4d/data/nuscenes/<results json file>
 
-```bash
-# vis4d.engine
-./scripts/dist_train.sh <config-file> <num-gpus>
-
-# vis4d.pl
-vis4d-pl fit --config <config-file> --gpus <num-gpus>
+# For test
+./full_eval.sh test vis4d/data/nuscenes_test/<results json file>
 ```
 
-- SLURM
+## Expected Performance
 
-```bash
-# vis4d.engine
-srun vis4d fit --config <config-file> --gpus <num-gpus> --slurm True
-
-# vis4d.pl
-srun vis4d-pl fit --config <config-file> --gpus <num-gpus>
+### CC-3DT++ On nuScenes trainval
 ```
-
-### Testing
-
-- Local machine / SLURM interactivate job (`job-name=bash`)
-
-```bash
-# vis4d.engine
-./scripts/dist_test.sh <config-file> <num-gpus>
-
-# vis4d.pl
-vis4d-pl test --config <config-file> --gpus <num-gpus>
+AMOTA   0.381
+AMOTP   1.366
+RECALL  0.461
+MOTAR   0.729
+GT      14556
+MOTA    0.342
+MOTP    0.711
+MT      2120
+ML      2505
+FAF     38.5
+TP      55658
+FP      10808
+FN      45034
+IDS     1205
+FRAG    2012
+TID     1.44
+LGD     2.52
 ```
 
-- SLURM
-
-```bash
-# vis4d.engine
-srun vis4d test --config <config-file> --gpus <num-gpus> --slurm True
-
-# vis4d.pl
-srun vis4d-pl test --config <config-file> --gpus <num-gpus>
+### CC-3DT++ On nuScenes mini
+```
+AMOTA   0.477
+AMOTP   1.226
+RECALL  0.587
+MOTAR   0.803
+GT      611
+MOTA    0.476
+MOTP    0.703
+MT      62
+ML      53
+FAF     96.2
+TP      2132
+FP      413
+FN      1437
+IDS     100
+FRAG    124
+TID     1.42
+LGD     5.02
 ```
 
 ## Acknowledgement
+For CC-3DT++ we build upon the original Vis4D codebase, which is a group effort by our team at ETH Zurich. We would like to thank the authors for their contribution to the open-source community.
 Vis4D is a group effort by our team at ETH Zurich.
-[Yung-Hsu Yang](https://royyang0714.github.io/) built the current version and will be the main maintainer of the codebase.
-
-Vis4D was originally written by [Tobias Fischer](https://tobiasfshr.github.io/) during the first three years of his Ph.D. at ETH Zurich, [Thomas E. Huang](https://www.thomasehuang.com/) helped contribute many models, [Tao Sun](https://www.suniique.com/) implemented the ViT models and designed the evaluation pipeline, and[René Zurbrügg](https://github.com/renezurbruegg) designed the config system.
-
-
-## Contributors
-**Project Leads**
-- [Yung-Hsu Yang](https://royyang0714.github.io/)*
-- [Tobias Fischer](https://tobiasfshr.github.io/)*
-
-**Core Contributors**
-- [Thomas E. Huang](https://www.thomasehuang.com/)
-- [Tao Sun](https://www.suniique.com/)
-- [René Zurbrügg](https://renezurbruegg.github.io/)
-
-**Advisors**
-- [Fisher Yu](https://www.yf.io/)
-
-`*` denotes equal contribution
-
-**We are open to contributions and suggestions, feel free to reach out to us.**
-
-[Check out our contribution guidelines for this project](docs/source/dev_guide/contribute.rst)
-
-**Community Contributors**
-
-<a href="https://github.com/SysCV/vis4d/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=SysCV/vis4d" />
-</a>
-
 
 ## Citation
 
@@ -139,3 +104,14 @@ If you find Vis4D is useful for your research, please consider citing the follow
   year = {2024}
 }
 ```
+
+If you find CC-3DT++ is useful for your research, please consider citing the following BibTeX entry.
+
+```bibtex
+@article{baumann2024cr3dt,
+  title={CR3DT: Camera-RADAR Fusion for 3D Detection and Tracking},
+  author={Baumann, Nicolas and Baumgartner, Michael and Ghignone, Edoardo and K{\"u}hne, Jonas and Fischer, Tobias and Yang, Yung-Hsu and Pollefeys, Marc and Magno, Michele},
+  journal={arXiv preprint arXiv:2403.15313},
+  year={2024}
+}
+```

eval_nusc.py

@@ -0,0 +1,117 @@
+"""nuScenes evaluation pipeline for Vis4D."""
+import argparse
+import os
+
+from nuscenes import NuScenes
+from nuscenes.eval.detection.evaluate import NuScenesEval
+from nuscenes.eval.detection.config import config_factory
+
+from nuscenes.eval.tracking.evaluate import TrackingEval as track_eval
+from nuscenes.eval.common.config import config_factory as track_configs
+
+
+def parse_arguments() -> argparse.Namespace:
+    """Parse arguments."""
+    parser = argparse.ArgumentParser(description="Vis4D for nuScenes eval.")
+    parser.add_argument(
+        "--input",
+        "-i",
+        help=("Path to save nuScenes format dection / tracking results."),
+    )
+    parser.add_argument(
+        "--version",
+        "-v",
+        choices=["v1.0-trainval", "v1.0-test", "v1.0-mini"],
+        help="NuScenes dataset version to convert.",
+    )
+    parser.add_argument(
+        "--dataroot",
+        "-d",
+        help="NuScenes dataset root.",
+    )
+    parser.add_argument(
+        "-m",
+        "--mode",
+        default="tracking",
+        choices=["tracking", "detection"],
+        help="Conversion mode: detection or tracking.",
+    )
+    return parser.parse_args()
+
+
+def eval_detection(
+    version: str,
+    dataroot: str,
+    output_dir: str,
+    result_path: str,
+    eval_set: str,
+) -> None:
+    """Evaluate detection results."""
+    nusc = NuScenes(version=version, dataroot=dataroot, verbose=True)
+
+    nusc_eval = NuScenesEval(
+        nusc,
+        config=config_factory("detection_cvpr_2019"),
+        result_path=result_path,
+        eval_set=eval_set,
+        output_dir=output_dir,
+        verbose=True,
+    )
+    _ = nusc_eval.main(render_curves=False)
+
+
+def eval_tracking(
+    version: str, output_dir: str, result_path: str, root: str, eval_set: str
+) -> None:
+    """Evaluate tracking results."""
+    nusc_eval = track_eval(
+        config=track_configs("tracking_nips_2019"),
+        result_path=result_path,
+        eval_set=eval_set,
+        output_dir=output_dir,
+        verbose=True,
+        nusc_version=version,
+        nusc_dataroot=root,
+    )
+    _ = nusc_eval.main()
+
+
+def evaluate(
+    version: str,
+    dataroot: str,
+    mode: str,
+    output_dir: str,
+    result_path: str,
+    root: str,
+) -> None:
+    """nuScenes evaluation."""
+    if "mini" in version:
+        eval_set = "mini_val"
+    elif "test" in version:
+        eval_set = "test"
+    else:
+        eval_set = "val"
+
+    if mode == "tracking":
+        eval_tracking(version, output_dir, result_path, root, eval_set)
+    else:
+        eval_detection(version, dataroot, output_dir, result_path, eval_set)
+
+
+if __name__ == "__main__":
+    """Main."""
+    args = parse_arguments()
+
+    if args.mode == "detection":
+        metric = "detect_3d"
+    else:
+        metric = "track_3d"
+
+    evaluate(
+        args.version,
+        args.dataroot,
+        args.mode,
+        args.input,
+        os.path.join(args.input, f"{metric}_predictions.json"),
+        args.dataroot,
+    )