The val results does not correspond to the paper

[gleeeee1]

I tested the network using the mini dataset val and weight centerfusion_e60.pth. And I got the following results. There is a gap between these results and those in paper. I think it might be because the dataset is too small. Do you use the val of the full dataset or the val from the mini dataset? How can the evaluation result be matched to the paper?
`mAP: 0.3129
mATE: 0.6590
mASE: 0.4630
mAOE: 0.6008
mAVE: 0.7622
mAAE: 0.3084
NDS: 0.3771
Eval time: 7.8s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.542 0.418 0.159 0.120 0.152 0.073
truck 0.442 0.542 0.168 0.133 0.141 0.122
bus 0.549 0.484 0.081 0.079 1.400 0.090
trailer 0.000 1.000 1.000 1.000 1.000 1.000
construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000
pedestrian 0.468 0.561 0.262 0.491 0.512 0.118
motorcycle 0.318 0.689 0.342 1.167 0.061 0.016
bicycle 0.187 0.448 0.299 0.417 1.832 0.048
traffic_cone 0.624 0.447 0.319 nan nan nan
barrier 0.000 1.000 1.000 1.000 nan nan`

here's the results from the paper:
Method Dataset C R L NDS ↑ mAP ↑ mA TE mASE mAOE mA VE mAAE Ours (DLA) val X X 0.453 0.332 0.649 0.263 0.535 0.540 0.142
Method Dataset C R L Car Truck Bus Trailer Const. Pedest. Motor. Bicycle Traff. Barrier Ours (DLA) val X X 0.524 0.265 0.362 0.154 0.055 0.389 0.305 0.229 0.563 0.470
please give me some advice, thank you!

[gleeeee1]

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

[niranjanravilevelup]

@gleeeee1 Hi, I have some trouble with using the mini-dataset. The issue is at #62 . can you please help me with this? Thank you.

[JYS997760473]

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

Hi, do you evaluate on nuScenes validation dataset?