Metric | Description |
---|---|
BLEU | Sacrebleu & BLEU from HuggingFace |
METEOR | HuggingFace |
REUSE | Follow Instructions on author's github |
SDC* | Already in this Repo |
SMS | Already in this Repo |
Bart Score | Already in this Repo |
- For BLEU METEOR, REUSE, SDC*, initialize a class first and call
compute
function seeadditional_metrics.py
- SMS
from wmd_master import SMD_scorer
, and usecalculate_score
function. Note: SMS is hard coded with GloVe embeddings and SMS Metric
- When loading METEOR from HF may display network errors warning messages. This is due to
nltk.download
not working. Probably because of bad internet - METEOR needs some addons such as wordnet. If
nltk.download('wordnet')
doesn't work. Go google nltk_data github and pull the repo to some local folders specified by the lookup error. - SMS (Sentence Mover Score) uses 2 embeddings, ELMo and GloVe. ELMo uses
allennlp.commands.elmo
which doesn't work for some (most) people. Issue by some other people. Therefore, GloVe is hard coded. - You need
en_core_web_md
for SMS, this can be done with huggingface. This requires SpaCy to be >=3.5.0,<3.6.0 or else when loadingen_core_web_md
, it will display a warning message. The performance isn't affected (I tried with like 5 sentences) but it is annoying. Tried with 3.2.2 and with 3.5.0.
!pip install https://huggingface.co/spacy/en_core_web_md/resolve/main/en_core_web_md-any-py3-none-any.whl
# Using spacy.load().
import spacy
nlp = spacy.load("en_core_web_md")
# Importing as module.
import en_core_web_md
nlp = en_core_web_md.load()
- SMS,
pip install wmd
will fail with you don't have a C++ compiler. It is suggested to run SMS on Linux.