Cannot reproduce training results

[AlexeyG]

Hello

I have been trying to reproduce the training results. However, I am not getting anywhere close to the results reported in the paper, or the checkpoint released in this repo (I get an mAP of 20, compared to the public model that gets 31.0 on AVA).

Can you provide some assistance here, in reproducing the paper's results. Or provide explanations for why the code does not reproduce?

These are the steps I have taken:

Firstly, I had to apply the changes from this issue to make the code work. Otherwise, the code provided would crash on loading the data.

I then followed the instructions in the readme. However, the performance on AVA after just a few epochs was very bad:

Epoch	1	3
mAP on AVA	1.43	1.98

Looking into the provided config further, we can see that MODEL.PRETRAINED = False, which means that the weights of the backbone are not loaded here

By loading the backbone pretrained weights, the performance did improve. But after training completed, the results are still nowhere close to what was reported in the paper (33.6), or what the publicly released checkpoint gets (31.0)

Epoch	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
mAP on AVA	11.28	15.08	17.77	18.50	18.81	19.59	20.34	20.98	21.00	20.43	20.94	21.77	20.76	20.03	20.35	20.42	21.14	20.79	20.39	19.91

Can you please explain why the training code does not reproduce the results of the paper?
Also, why is the default training setting to not load the backbone weights? This setting was never explored in the paper.
I have also attached the config that I used to achieve the 20.0 mAP result on AVA above.

[AlexeyG]

Attaching the config gist

[Hutaf]

Thank you for your comments, which have been immensely helpful in improving the results of my work. However, I must admit that I am still struggling to reproduce the results of the paper. I was wondering if you have found a way to successfully replicate the results of the paper. If so, I would greatly appreciate any guidance or advice you could provide me with. @AlexeyG

[sibonjia]

I cann't reproduce the results of the paper, too. And there are some bugs in the code.