Problems in reproducing the results

[JiyangZhang]

Hi,

Thanks for the great work and the detailed documentation! I would like to check with you the way to reproduce the results in the paper.

Goal: I want get the results for your best model that fine-tuned on the Tufano_etal_ICSE21 code&comment-to-code dataset. And generate the predictions with beam search 20 but only let model return top 1 prediction among 20 beams as the final prediction.

Steps I took:

1. Get model checkpoint:
   python3 ./tf_2_pytorch_T5.py --tf_checkpoint_path ../models/T5_pre-trained_Tufano_etal_dataset_code\&comment-to-code/model.ckpt --config_file ./config.json --pytorch_dump_path ./dumps

2. Change script to do prediction: I changed the beam size to 20, and the num_return_sequences=1 in t5_mlm.generate() then there will be one prediction for each data point.

3. Compute the metrics: I got around 29% top-1 match

Does it sound good to you?

Thank you.