[Bug] img_path = data_info['images'][cam_type]['img_path'] KeyError: 'CAM2'

[weinixiong8964]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux
Python: 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: NVIDIA GeForce RTX 4090 D
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 9.5.0-1ubuntu1~22.04) 9.5.0
PyTorch: 1.12.0+cu113
PyTorch compiling details: PyTorch built with:

• GCC 9.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.3.2 (built against CUDA 11.5)
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.13.0+cu113
OpenCV: 4.9.0
MMEngine: 0.10.4
MMDetection: 3.1.0
MMDetection3D: 1.4.0+
spconv2.0: True

Reproduces the problem - code sample

python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py ${CHECKPOINT_FILE} --show --cam-type CAM_BACK

Reproduces the problem - command or script

python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth --show --cam-type CAM_BACK

Reproduces the problem - error message

This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized.
/home/yuanyue2021/anaconda3/envs/mm3d/lib/python3.8/site-packages/mmengine/visualization/visualizer.py:196: UserWarning: Failed to add <class 'mmengine.visualization.vis_backend.LocalVisBackend'>, please provide the save_dir argument.
warnings.warn(f'Failed to add {vis_backend.class}, '
Traceback (most recent call last):
File "demo/mono_det_demo.py", line 98, in
main()
File "demo/mono_det_demo.py", line 88, in main
inferencer(**call_args)
File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/base_3d_inferencer.py", line 204, in call
ori_inputs = self._inputs_to_list(inputs, cam_type=cam_type)
File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/mono_det3d_inferencer.py", line 110, in _inputs_to_list
img_path = data_info['images'][cam_type]['img_path']
KeyError: 'CAM2'

Additional information

No response

[KopiSoftware]

The cam-type is somehow failed to pass its value here. I add cam_type = 'CAM0' before img_path = data_info['images'][cam_type]['img_path']. It passed, while another problem raised:
KeyError: 'cam2img'
So I output the content of demo/data/sunrgbd/sunrgbd_000017_infos.pkl and found that there is no cam2img. It seems like intrinsics for a camera or something.
I ran this on kitti too. It lack of a parameter called depth2img, whereas pkl file of sunrgbd has.
Wired.

[KopiSoftware]

I solved it. Just as I posted before, some important parameters were missing. Some changes:
in mmdetection3d/mmdet3d/apis/inferencers/multi_modality_det3d_inferencer.py (Ln 117, 125, and etc.), it failed to aquire inner and extrinsics, I directly get them in SUNRGBD dataset(intrinsics.txt and extrinsics/20151113224133.txt):

                # cam2img = np.asarray(
                #     data_info['images'][cam_type]['cam2img'], dtype=np.float32)
                # lidar2cam = np.asarray(
                #     data_info['images'][cam_type]['lidar2cam'],
                #     dtype=np.float32)
                # if 'lidar2img' in data_info['images'][cam_type]:
                #     lidar2img = np.asarray(
                #         data_info['images'][cam_type]['lidar2img'],
                #         dtype=np.float32)
                # else:
                #     lidar2img = cam2img @ lidar2cam
                    # lidar2img = cam2img @ lidar2cam[0:3]

                # for sunrgbd
                cam2img = np.asarray([[574.334300, 0.000000, 320.000000, 0.000000],
                                      [0.000000, 574.334300, 240.000000, 0.000000],
                                      [0.000000, 0.000000, 1.000000, 0.000000],
                                      [0.000000, 0.000000, 0.000000, 1.000000]]
                                    )
                lidar2cam = np.asarray(
                    [[0.999997, 0.002445, 0.000612, 0.000000],
                    [-0.002445, 0.881972, 0.471295, 0.000000],
                    [0.000612, -0.471295, 0.881975, 0.000000],
                    [0, 0, 0, 1]]
                )

                lidar2img = cam2img @ lidar2cam

                input['cam2img'] = cam2img
                input['lidar2cam'] = lidar2cam
                input['lidar2img'] = lidar2img

in mmdetection3d/mmdet3d/models/layers/fusion_layers/vote_fusion.py, it failed to aquire depth to img parameter. I directly wrote the data in sunrgbd (ln 62) here:

            depth2img = torch.tensor([[530.5366320782341, 362.0406589161139, -32.432176765054464], [35.10014939727262, 164.8833093419671, -567.605659916997], [0.005785000044852495, 0.9835000038146973, -0.18082000315189362]]).cuda()
            # depth2img = xyz_depth.new_tensor(img_meta['depth2img'])