Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BERTScore #3

Open
forrestbao opened this issue Jun 5, 2022 · 0 comments
Open

BERTScore #3

forrestbao opened this issue Jun 5, 2022 · 0 comments
Labels
metric

Comments

@forrestbao
Copy link
Contributor

forrestbao commented Jun 5, 2022 •
edited
Loading

It seems that the default values of keyword arguments in Huggingface's BERTScore API do not give the best of BERTScore.

  1. idf: By default, it is off. We should probably turn it on. See "Importance Weighting" on page 4 of BERTScore paper However, since we use the same setting for both traditional and new approach, I am not sure whether it matters.
  2. model_type: Default language model is roberta-large when lang=en. According to BERTScore's lead board, other models have higher correlation with human ratings. However, since we use the same language model for both traditional/ref-based and new/DocAsRef approach, I am not sure whether it matters.
  3. use_fast_tokenizer. Default is off. Please turn on to speed up. Huggingface's fast tokenizer is implemented in Rust instead of Python.

@NKWBTB @lihebi Let me know your thoughts.
@forrestbao forrestbao assigned NKWBTB and unassigned NKWBTB Jun 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
metric
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@forrestbao @NKWBTB