SPGroup3D

This project provides the code and results for 'SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection', AAAI2024.

Anchors: Yun Zhu, Le Hui, Yaqi Shen, Jin Xie*

PaperLink: https://arxiv.org/abs/2312.13641

Introduction

Current 3D object detection methods for indoor scenes mainly follow the voting-and-grouping strategy to generate proposals. However, most methods utilize instance-agnostic groupings, such as ball query, leading to inconsistent semantic information and inaccurate regression of the proposals. To this end, we propose a novel superpoint grouping network for indoor anchor-free one-stage 3D object detection. Specifically, we first adopt an unsupervised manner to partition raw point clouds into superpoints, areas with semantic consistency and spatial similarity. Then, we design a geometry-aware voting module that adapts to the centerness in anchor-free detection by constraining the spatial relationship between superpoints and object centers. Next, we present a superpoint-based grouping module to explore the consistent representation within proposals. This module includes a superpoint attention layer to learn feature interaction between neighboring superpoints, and a superpoint-voxel fusion layer to propagate the superpoint-level information to the voxel level. Finally, we employ effective multiple matching to capitalize on the dynamic receptive fields of proposals based on superpoints during the training. Experimental results demonstrate our method achieves state-of-the-art performance on ScanNet V2, SUN RGB-D, and S3DIS datasets in the indoor one-stage 3D object detection.

Preparation

• For installing the environment, we mainly follow TR3D. Besides, we also provide our_env.yaml for details check.

• Alternatively, you can install all required packages manually. This implementation is based on mmdetection3d framework. Please refer to the installation guide getting_started.md, including MinkowskiEngine installation.

# If you can not install MinkowskiEngine with pip successfully,
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install --blas=openblas --force_cuda

• Install torch_scatter.

pip install torch-scatter=2.0.9 -f https://data.pyg.org/whl/torch-1.10.1+${CUDA}.html

• All the SPGroup3D-related code locates in the folder projects.

Data Preparation

• We mainly follow the mmdetection3d data preparation protocol described in scannet, sunrgbd, and s3dis, including superpoint generation. Other modifies are in scannet_data_utils, s3dis_data_utils, and sunrgbd_data_utils, which are related to pkl flies generation.

• Since superpoint generation needs many dependencies. We recommend you use the official data processing and use the processed superpoint and pkl flies we provided in GoogleDrive / BaiduDrive. By doing this, you just need to put them into your files and replace the origin pkl files.

• Please DON'T do any sampling operation in Data preparation, otherwise it will result in a mismatch with the superpoint we provide.

Training

To start training, run train with SPGroup3D configs:

# Remember to modify the data_root
# scannet v2
CUDA_VISIBLE_DEVICE={} bash tools/dist_train.sh projects/configs/SPGroup_scannet.py 4 --work-dir work_dirs/{YOUR PATH}
# sunrgbd
CUDA_VISIBLE_DEVICE={} bash tools/dist_train.sh projects/configs/SPGroup_sunrgbd.py 4 --work-dir work_dirs/{YOUR PATH}
# s3dis
CUDA_VISIBLE_DEVICE={} bash tools/dist_train.sh projects/configs/SPGroup_s3dis.py 4 --work-dir work_dirs/{YOUR PATH}

Testing

Test pre-trained model using test with SPGroup3D configs:

# scannet v2
python tools/test.py projects/configs/SPGroup_scannet.py \
    work_dirs/{YOUR PATH}.pth --eval mAP
# sunrgbd
python tools/test.py projects/configs/SPGroup_sunrgbd.py \
    work_dirs/{YOUR PATH}.pth --eval mAP
# s3dis
python tools/test.py projects/configs/SPGroup_s3dis.py \
    work_dirs/{YOUR PATH}.pth --eval mAP

Visualization

Visualizations can be created with test script. For better visualizations, you may set score_thr in configs to 0.25:

# scannet v2
python tools/test.py projects/configs/SPGroup_scannet.py \
    work_dirs/{YOUR PATH} --eval mAP --show \
    --show-dir work_dirs/{YOUR PATH}

Main Results

All models are trained with 4 3090 GPUs.

Dataset	[email protected]	[email protected]	Download	config
ScanNet V2	74.3 (73.5)	59.6 (58.3)	GoogleDrive / BaiduDrive	config
SUN RGB-D	65.4 (64.8)	47.1 (46.4)	GoogleDrive / BaiduDrive	config
S3DIS	69.2 (67.7)	47.2 (43.6)	GoogleDrive / BaiduDrive	config

Due to the size of these datasets and the randomness that inevitably exists in the model, the results on these datasets fluctuate significantly. It's normal for results to fluctuate within a range.

Citation

If you find this work useful for your research, please cite our paper:

@article{zhu2023spgroup3d,
  title={SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection},
  author={Zhu, Yun and Hui, Le and Shen, Yaqi and Xie, Jin},
  journal={arXiv preprint arXiv:2312.13641},
  year={2023}
}

Acknowledgments

This project is based on the following codebases.

• mmdetection3D
• TR3D
• CAGroup3D
• 3D-WSIS
• segmentator
• SPG/SSP
• pclpy

If you find this project helpful, Please also cite the codebases above. Thanks.