Performance by named entity category of spacy/en_core_web_trf

[exfalsoquodlibet]

Hi,

I'm looking for a breakdown of performance scores (ENTS_P, ENTS_R, and ENTS_F) by entity categories for your spacy/en_core_web_trf model. I cannot seem to find this in the official documentation (https://spacy.io/models/en#en_core_web_trf).

It'd be really useful to benchmark the performance of the NER model my team has been training (some categories overlap).

Is this information available anywhere or could it be made available?

Thank you.

[svlandeg]

Hi, good question! This information is recorded in the file accuracy.json that is shipped as part of the model when you download it. So when you've done spacy download en_core_web_trf you can navigate to where this package sits in your virtual environment, for instance on my system that would be ...\venv\Lib\site-packages\en_core_web_trf\en_core_web_trf-3.5.0. I'll paste the relevant content here for your convenience:

  "ents_p": 0.8977938984,
  "ents_r": 0.9048978365,
  "ents_f": 0.9013318701,
  "ents_per_type": {
    "DATE": {
      "p": 0.8915360502,
      "r": 0.9028571429,
      "f": 0.8971608833
    },
    "GPE": {
      "p": 0.9549524342,
      "r": 0.9520223152,
      "f": 0.9534851236
    },
    "ORDINAL": {
      "p": 0.7771260997,
      "r": 0.8229813665,
      "f": 0.7993966817
    },
    "ORG": {
      "p": 0.8985736926,
      "r": 0.9019088017,
      "f": 0.9002381582
    },
    "FAC": {
      "p": 0.5635359116,
      "r": 0.7846153846,
      "f": 0.6559485531
    },
    "QUANTITY": {
      "p": 0.7823529412,
      "r": 0.7307692308,
      "f": 0.7556818182
    },
    "LOC": {
      "p": 0.8443708609,
      "r": 0.8121019108,
      "f": 0.8279220779
    },
    "CARDINAL": {
      "p": 0.8491208168,
      "r": 0.8900118906,
      "f": 0.8690856313
    },
    "PERSON": {
      "p": 0.9398162813,
      "r": 0.9683420366,
      "f": 0.953865938
    },
    "NORP": {
      "p": 0.9253968254,
      "r": 0.9328,
      "f": 0.9290836653
    },
    "PRODUCT": {
      "p": 0.6804733728,
      "r": 0.5450236967,
      "f": 0.6052631579
    },
    "TIME": {
      "p": 0.7663043478,
      "r": 0.8245614035,
      "f": 0.7943661972
    },
    "EVENT": {
      "p": 0.7898550725,
      "r": 0.6264367816,
      "f": 0.6987179487
    },
    "WORK_OF_ART": {
      "p": 0.6514285714,
      "r": 0.587628866,
      "f": 0.6178861789
    },
    "LAW": {
      "p": 0.6363636364,
      "r": 0.65625,
      "f": 0.6461538462
    },
    "MONEY": {
      "p": 0.9109506619,
      "r": 0.893742621,
      "f": 0.9022646007
    },
    "PERCENT": {
      "p": 0.9224683544,
      "r": 0.8928024502,
      "f": 0.9073929961
    },
    "LANGUAGE": {
      "p": 1.0,
      "r": 0.71875,
      "f": 0.8363636364
    }

All that said - when you're setting out to compare two different models, it's always best to rerun the evaluation yourself for both models, on a totally independent dev set that has zero overlap with the training sets of either models. If you want to accomplish this, you can use spacy benchmark accuracy.

[exfalsoquodlibet]

Thanks @svlandeg - most useful.

when you're setting out to compare two different models, it's always best to rerun the evaluation yourself for both models, on a totally independent dev set that has zero overlap with the training sets of either models.

Yes, we are doing this and we know en_core_web_trf performs rather poorly on our corpus because the language is rather specific. So, we were interested to know how en_core_web_trf performed when the linguistic contexts of train and dev sets were the same.