Not able to replicate NaLP paper results?

[siddhu001]

Hi We were trying to replicate the results of NaLP paper-

We ran the following commands-

1. python JF17K2rv_json.py
2. python builddata.py --sub_dir JF17K_version1 --dataset_name JF17K_version1
3. python train_only.py --sub_dir JF17K_version1 --dataset_name JF17K_version1 --wholeset_name JF17K_version1_permutate --model_name JF17K_version1_opt --embedding_dim 100 --n_filters 200 --n_gFCN 1000 --batch_size 128 --learning_rate 0.00005 --n_epochs 5000 --saveStep 100
4. CUDA_VISIBLE_DEVICE=0,1 python see_eval.py --sub_dir JF17K_version1 --dataset_name JF17K_version1 --wholeset_name JF17K_version1_permutate --model_name JF17K_version1_opt --embedding_dim 100 --n_filters 200 --n_gFCN 1000 --batch_size 512 --n_epochs 5000 --start_epoch 4800 --evalStep 100 --valid_or_test 2 --gpu_ids 0,1

However the results we obtain are -
Value Prediction - MRR: 0.21133465 Hits@10:0.33177453 Hits@3:0.22270729 Hits@1:0.15060749
Role Prediction - MRR: 0.82452982 Hits@10:0.89603836 Hits@3:0.85997366 Hits@1:0.7798825

Can you advise on what could be the cause of this discrepancy?

[gsp2014]

It seems that you miss this command: "python builddata.py --sub_dir JF17K_version1 --dataset_name JF17K_version1 --if_permutate True --bin_postfix _permutate". And if you have trained the model until the evaluation result on the validation set drops continuously or reaching the max number of epochs, i.e., 5000?