How to remove evaluation of frozen components ?

[probavee]

Hello everyone !

I'm training NER components. I already made one the "classic" way, using only "Transformer" and "NER" components with the default config file provided by spacy. It worked well.
Now I'm trying to train another NER component using annotations from the pre-trained french pipeline fr_dep_news_trf.
I just want to see the performance if i'm freezing everything except the NER component when the component has access to all previous annotations.
The training dataset contains only the named entity label. (no pos, dep, lemma, ...).
When I launch the training, I got an error because the labels for parser evaluation do not exist. Which was unexpected for me since I only wanted to train and evaluate the NER component, so I don't have any other label in my dataset.

ValueError: [E024] Could not find an optimal move to supervise the parser. Usually, this means that the model can't be updated in a way that's valid and satisfies the correct annotations specified in the GoldParse. For example, are all labels added to the model? If you're training a named entity recognizer, also make sure that none of your annotated entity spans have leading or trailing whitespace or punctuation. You can also use the `debug data` command to validate your JSON-formatted training data.

My base config file :

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = null
[system]
gpu_allocator = "pytorch"

[nlp]
lang = "fr"
pipeline = ["transformer", "morphologizer", "parser", "attribute_ruler", "lemmatizer", "ner"]
batch_size = 128

[components]


[components.transformer]
source = "fr_dep_news_trf"

[components.morphologizer]
source = "fr_dep_news_trf"

[components.parser]
source = "fr_dep_news_trf"

[components.attribute_ruler]
source = "fr_dep_news_trf"

[components.lemmatizer]
source = "fr_dep_news_trf"

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
frozen_components = ["transformer", "morphologizer", "parser", "attribute_ruler","lemmatizer"]
annotating_components = ["transformer", "morphologizer", "parser", "attribute_ruler","lemmatizer"]

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[initialize]
vectors = ${paths.vectors}

So my questions are: why does it behave like this? How can I handle this? Is what I'm trying to do relevant?

[adrianeboyd]

The error message is a little confusing, since there's an internal "parser" that is used for both parser and ner, so my first guess is that there is some problem with the NER training data that leads to this particular error.

But to back up a bit first: you have the right overall idea here, but in practice it doesn't work to train ner with a frozen transformer component. You'll need to use a separate transformer component if you want to add ner to fr_dep_news_trf (yes, it will be twice as big and twice as slow, which is why we don't publish a fr_core_news_trf pipeline right now).

I'd recommend:

• train only transformer+ner using the ner GPU config from the training quickstart or init config
• if you see the same parser error, run spacy debug data on your NER-only config and data
• train the NER-only pipeline
• add the new ner component to fr_dep_news_trf and replace the listeners so that the ner component uses an internal transformer model (otherwise you'd have two components named transformer):

  fr_ner = spacy.load("/path/to/fr_ner_trf")  # your trained ner pipeline
  fr_ner.replace_listeners("transformer", "ner", ["model.tok2vec"])
  nlp = spacy.load("fr_dep_news_trf")
  nlp.add_pipe("ner", source=fr_ner)
  nlp.to_disk("/path/to/fr_core_news_trf")

I don't think it will make a large difference, but if you want to try your original idea, you can start out with the transformer component from fr_dep_news_trf in your NER config to see if this improves the training. To do this, replace the entire [components.transformer] block with:

[components.transformer]
source = "fr_dep_news_trf"

[probavee]

I don't think the problem came from the dataset as it work perfectly with the method you provided. But anyway the result are good enough without using other pipeline components and with default spaCy's hyper-parameters. I also tried training with the source fr_dep_news_trf but it was a little worse. Thank you !

[adrianeboyd]

Oh, then I'm not sure what was going on , but I'm glad to hear that you got it working!