Results on QVHighlights val set

[gkv91]

Hi,

Thanks for sharing the code for your amazing work!!! Can you share the results on the QVHighlights val set?

Thanks,
Goutham

[QinghongLin]

Hi @gkv91 ,

Thanks for interest, the QVHL val set result should be shared on the best_qvhighlights_val_preds_nms_thd_0.7.jsonl AND training log (e.g., tensorboard) in model zoo page: https://github.com/showlab/UniVTG/blob/main/model.md
please take a look.

[gkv91]

Thanks for your fast response. I tried to replicate the results but, I am getting very poor values. Specifically, I trained the model by running "bash scripts/qvhl_pretrain.sh" with the pre-trained checkpoint. Can you help me to get similar results as yours? Attaching my results screenshot for your reference:

[QinghongLin]

@gkv91 Hi the parameter in qvhl_pretrain.sh is not the optimal one (now I have updated them).
Please try this parameters:

b_loss_coef=10.0
g_loss_coef=1.0
f_loss_coef=10.0
s_loss_intra_coef=0.05
s_loss_inter_coef=0.01

You can find them by referring the paper supp. OR the model zoo config.

https://github.com/showlab/UniVTG/blob/main/model.md

[gkv91]

Sure, I will try. Thanks :)

[QinghongLin]

@gkv91 Can you please try this parameters and update the result here? thanks!

[gkv91]

Hi, I tried using parameter setting as you mentioned. But, I did not get much performance improvement. Specifically, I have initialised the UniVTG model using the pre-trained checkpoint from here (by giving the checkpoint path in the "resume" arg) and then trained the model on the QVHighlight dataset using "scripts/qvhl_pretrain.sh" with the parameters that you mentioned. Can you help me to find where I am doing wrong? Attaching the results screenshot for your reference. Thanks

[QinghongLin]

Hi @gkv91 ,

Can you attach your completed script (sh) and the opt.json here.

without any pretraining, we should able to reproduce the result based on this parameters settings. Can you compare the opt.json to the opt.json in the following: https://drive.google.com/drive/folders/1EqwZSOVeKBCjcHe6SfeUjxM4fN6xrPf3

Thanks!

[gkv91]

Sure, please find the attached file. Thanks
Archive.zip

[QinghongLin]

@gkv91 Thanks for attachment! I have checked the results and found most parameters are aligned.
This is weired to me, Does the result you repo is the best one or the latest one?

Besides, can you attach the eval log to me, if the pretrained checkpoint load successfully, it should be receive proper performance in the zero-shot stage (i.e., epoch=-1).

OR

Can you load my provided QVHL checkpoints can see can it reproduce the same results :)
https://drive.google.com/drive/folders/1ms53Lfm__zrzlvBsadIT6b17vUjFhRG7

[gkv91]

Hi, the results are the best one.

I have checked the log file and seems like the checkpoint file is not loading. But, the "if opt.resume is not None" condition in the Config.py is executing fine .
Please find the log file attached.
Thanks
eval.log.txt.zip