Is the BertScore metric used with idf weighted averaging or not

[La-SilverLand]

since in the BertScore paper, the evaluation result differs w/o idf weighted averaging, I'm wondering whether it's related to which kind of BertScore (or sadly both )

[Alex-Fabbri]

Hi @La-SilverLand !

For Table 4 we used idf weighted averaging. For the correlation calculations, we did not use idf weighted averaging as we ran this separately on just the 100 examples where the idf weighting may not be representative, but we could also have gotten the idf from the whole corpus and then used that. I will run this when I get a free cycle.

[La-SilverLand]

btw, there seems a little error in the table 3 column Consistency, the point 4.96 of M12 is not included in the top 3 ranking...

[Alex-Fabbri]

Thank you for pointing that out!

[cyr19]

Hi @La-SilverLand !

For Table 4 we used idf weighted averaging. For the correlation calculations, we did not use idf weighted averaging as we ran this separately on just the 100 examples where the idf weighting may not be representative, but we could also have gotten the idf from the whole corpus and then used that. I will run this when I get a free cycle.

Hi,

thanks for your great work!
So the basic.config file in the examples folder is not the real configurations used and the correlations in Table 2 were computed without idf, right？

Thanks + best regards,