Issue when running test.py using a Custom dataset

[Mo7amedNajjar]

Hello all, I was finally able to train my custom dataset follwing the custom_dataset tutorial on OPENPCDET.

I trainsed my dataset of 733 samples for about 5 epochs, When I wanted to test the evaualtion and see evaualtion metrics for the "Pedestrian" class I am trying to detect.

I run the following the code:

python tools/test.py --cfg_file tools/cfgs/custom_models/second.yaml --batch_size 1 --ckpt output/cfgs/custom_models/second/default/ckpt/checkpoint_epoch_5.pth

This is the output below: I am not sure if this error is a CUDA error or if there is something I have set up not correctly when setting up the data.

Can anyone please advice me if encountered similar issue ?

/root/OpenPCDet/pcdet/utils/loss_utils.py:152: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float16)
2025-03-13 14:36:34,347 INFO Start logging
2025-03-13 14:36:34,348 INFO CUDA_VISIBLE_DEVICES=ALL
2025-03-13 14:36:34,348 INFO cfg_file tools/cfgs/custom_models/second.yaml
2025-03-13 14:36:34,349 INFO batch_size 1
2025-03-13 14:36:34,349 INFO workers 4
2025-03-13 14:36:34,349 INFO extra_tag default
2025-03-13 14:36:34,349 INFO ckpt output/cfgs/custom_models/second/default/ckpt/checkpoint_epoch_5.pth
2025-03-13 14:36:34,349 INFO pretrained_model None
2025-03-13 14:36:34,349 INFO launcher none
2025-03-13 14:36:34,349 INFO tcp_port 18888
2025-03-13 14:36:34,349 INFO local_rank None
2025-03-13 14:36:34,349 INFO set_cfgs None
2025-03-13 14:36:34,349 INFO max_waiting_mins 30
2025-03-13 14:36:34,349 INFO start_epoch 0
2025-03-13 14:36:34,349 INFO eval_tag default
2025-03-13 14:36:34,349 INFO eval_all False
2025-03-13 14:36:34,349 INFO ckpt_dir None
2025-03-13 14:36:34,349 INFO save_to_file False
2025-03-13 14:36:34,349 INFO infer_time False
2025-03-13 14:36:34,350 INFO cfg.ROOT_DIR: /root/OpenPCDet
2025-03-13 14:36:34,350 INFO cfg.LOCAL_RANK: 0
2025-03-13 14:36:34,350 INFO cfg.CLASS_NAMES: ['Pedestrian']
2025-03-13 14:36:34,350 INFO ----------- DATA_CONFIG -----------
2025-03-13 14:36:34,350 INFO cfg.DATA_CONFIG.DATASET: CustomDataset
2025-03-13 14:36:34,350 INFO cfg.DATA_CONFIG.DATA_PATH: /root/OpenPCDet/data/custom
2025-03-13 14:36:34,350 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
2025-03-13 14:36:34,350 INFO ----------- MAP_CLASS_TO_KITTI -----------
2025-03-13 14:36:34,350 INFO cfg.DATA_CONFIG.MAP_CLASS_TO_KITTI.Pedestrian: Pedestrian
2025-03-13 14:36:34,350 INFO ----------- DATA_SPLIT -----------
2025-03-13 14:36:34,350 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val
2025-03-13 14:36:34,351 INFO ----------- INFO_PATH -----------
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['custom_infos_train.pkl']
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['custom_infos_val.pkl']
2025-03-13 14:36:34,351 INFO ----------- POINT_FEATURE_ENCODING -----------
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2025-03-13 14:36:34,351 INFO ----------- DATA_AUGMENTOR -----------
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2025-03-13 14:36:34,351 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['custom_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Pedestrian:3']}, 'SAMPLE_GROUPS': ['Pedestrian:25'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2025-03-13 14:36:34,352 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 150000, 'test': 150000}}]
2025-03-13 14:36:34,352 INFO cfg.DATA_CONFIG.BASE_CONFIG: tools/cfgs/dataset_configs/custom_dataset.yaml
2025-03-13 14:36:34,352 INFO ----------- MODEL -----------
2025-03-13 14:36:34,352 INFO cfg.MODEL.NAME: SECONDNet
2025-03-13 14:36:34,352 INFO ----------- VFE -----------
2025-03-13 14:36:34,352 INFO cfg.MODEL.VFE.NAME: MeanVFE
2025-03-13 14:36:34,352 INFO ----------- BACKBONE_3D -----------
2025-03-13 14:36:34,352 INFO cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x
2025-03-13 14:36:34,352 INFO ----------- MAP_TO_BEV -----------
2025-03-13 14:36:34,352 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2025-03-13 14:36:34,352 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2025-03-13 14:36:34,352 INFO ----------- BACKBONE_2D -----------
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2025-03-13 14:36:34,353 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2025-03-13 14:36:34,353 INFO ----------- DENSE_HEAD -----------
2025-03-13 14:36:34,353 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2025-03-13 14:36:34,353 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2025-03-13 14:36:34,353 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2025-03-13 14:36:34,353 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2025-03-13 14:36:34,353 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Pedestrian', 'anchor_sizes': [[0.6, 0.6, 1.6]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.4, 'unmatched_threshold': 0.25}]
2025-03-13 14:36:34,354 INFO ----------- TARGET_ASSIGNER_CONFIG -----------
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2025-03-13 14:36:34,354 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2025-03-13 14:36:34,354 INFO ----------- LOSS_CONFIG -----------
2025-03-13 14:36:34,354 INFO ----------- LOSS_WEIGHTS -----------
2025-03-13 14:36:34,355 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2025-03-13 14:36:34,355 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2025-03-13 14:36:34,355 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2025-03-13 14:36:34,355 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2025-03-13 14:36:34,355 INFO ----------- POST_PROCESSING -----------
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2025-03-13 14:36:34,355 INFO ----------- NMS_CONFIG -----------
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.5
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 1024
2025-03-13 14:36:34,355 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 300
2025-03-13 14:36:34,355 INFO ----------- OPTIMIZATION -----------
2025-03-13 14:36:34,355 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 2
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.LR: 0.001
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2025-03-13 14:36:34,356 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2025-03-13 14:36:34,356 INFO cfg.TAG: second
2025-03-13 14:36:34,357 INFO cfg.EXP_GROUP_PATH: cfgs/custom_models
2025-03-13 14:36:34,358 INFO Loading Custom dataset.
2025-03-13 14:36:34,361 INFO Total samples for CUSTOM dataset: 147
/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/torch/functional.py:534: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3595.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
2025-03-13 14:36:35,953 INFO ==> Loading parameters from checkpoint output/cfgs/custom_models/second/default/ckpt/checkpoint_epoch_5.pth to GPU
/root/OpenPCDet/pcdet/models/detectors/detector3d_template.py:367: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
checkpoint = torch.load(filename, map_location=loc_type)
2025-03-13 14:36:36,123 INFO ==> Checkpoint trained from version: pcdet+0.6.0+8caccce
2025-03-13 14:36:36,139 INFO ==> Done (loaded 163/163)
2025-03-13 14:36:36,156 INFO *************** EPOCH 5 EVALUATION *****************
eval: 0%| | 0/147 [00:00<?, ?it/s]/root/OpenPCDet/pcdet/ops/iou3d_nms/iou3d_nms_utils.py:66: UserWarning: The torch.cuda.DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:78.)
overlaps_bev = torch.cuda.FloatTensor(torch.Size((boxes_a.shape[0], boxes_b.shape[0]))).zero() # (N, M)
eval: 100%|████████████████████████████████████████████████████████████████████████████████████| 147/147 [00:19<00:00, 7.59it/s, recall_0.3=(0, 145) / 147]
2025-03-13 14:36:55,540 INFO *************** Performance of EPOCH 5 *****************
2025-03-13 14:36:55,546 INFO Generate label finished(sec_per_example: 0.1319 second).
2025-03-13 14:36:55,546 INFO recall_roi_0.3: 0.000000
2025-03-13 14:36:55,547 INFO recall_rcnn_0.3: 0.986395
2025-03-13 14:36:55,547 INFO recall_roi_0.5: 0.000000
2025-03-13 14:36:55,548 INFO recall_rcnn_0.5: 0.918367
2025-03-13 14:36:55,548 INFO recall_roi_0.7: 0.000000
2025-03-13 14:36:55,549 INFO recall_rcnn_0.7: 0.061224
2025-03-13 14:36:55,549 INFO Average predicted number of objects(147 samples): 4.279
Traceback (most recent call last):
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 254, in ensure_initialized
self.cuInit(0)
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
self._check_ctypes_error(fname, retcode)
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/OpenPCDet/tools/test.py", line 210, in
main()
File "/root/OpenPCDet/tools/test.py", line 206, in main
eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
File "/root/OpenPCDet/tools/test.py", line 65, in eval_single_ckpt
eval_utils.eval_one_epoch(
File "/root/OpenPCDet/tools/eval_utils/eval_utils.py", line 125, in eval_one_epoch
result_str, result_dict = dataset.evaluation(
File "/root/OpenPCDet/pcdet/datasets/custom/custom_dataset.py", line 136, in evaluation
ap_result_str, ap_dict = kitti_eval(eval_det_annos, eval_gt_annos, self.map_class_to_kitti)
File "/root/OpenPCDet/pcdet/datasets/custom/custom_dataset.py", line 118, in kitti_eval
from ..kitti.kitti_object_eval_python import eval as kitti_eval
File "/root/OpenPCDet/pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 6, in
from .rotate_iou import rotate_iou_gpu_eval
File "/root/OpenPCDet/pcdet/datasets/kitti/kitti_object_eval_python/rotate_iou.py", line 18, in
def trangle_area(a, b, c):
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/decorators.py", line 131, in _jit
disp.compile_device(argtypes, restype)
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 882, in compile_device
cc = get_current_device().compute_capability
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/api.py", line 443, in get_current_device
return current_context().device
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 220, in get_context
return _runtime.get_or_create_context(devnum)
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 138, in get_or_create_context
return self._get_or_create_context_uncached(devnum)
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 153, in _get_or_create_context_uncached
with driver.get_active_context() as ac:
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 495, in enter
driver.cuCtxGetCurrent(byref(hctx))
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 292, in getattr
self.ensure_initialized()
File "/root/OpenPCDet/openpcdet-venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 258, in ensure_initialized
raise CudaSupportError(f"Error at driver init: {description}")
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_NO_DEVICE (100)
(openpcdet-venv) root@NAJJAR:~/OpenPCDet#

[Petros626]

reinstall CUDA clearly is recommendation.
https://gist.github.com/kmhofmann/cee7c0053da8cc09d62d74a6a4c1c5e4
https://medium.com/virtual-force-inc/a-step-by-step-guide-to-install-nvidia-drivers-and-cuda-toolkit-855c75efcdb6