How can I add a custom list of words to the tokenizer?

[bpersevere]

Hello, I'm a novice at NLP and spaCy. I have some user-defined words, most of which are biological names, such as Polygonum cuspidatum, or similar terms composed of two or three words. There are about 2000 or 3000 user-defined words. How can I add these custom words in the form of a custom dictionary so that Tokenizer can recognize these multi-word words as a single token, rather than split them into multiple words.

I use spaCy v3.4.

Thank you.

[polm]

If you have a list of phrases that you'd like to be single tokens, what I'd recommend you do is use the EntityRuler to mark your phrases as entities, and then use merge_entities to make them into single tokens.

[bpersevere]

Hello, I read the spaCy document according to your tips, and then I wrote the following code:

import spacy

nlp = spacy.load("en_core_web_sm")

text = "Here are some Polygonum cuspidatum."

patterns = [{"label": "BIO", "pattern": "Polygonum cuspidatum"}]

ruler = nlp.add_pipe("entity_ruler")

ruler.add_patterns(patterns)

nlp.add_pipe("merge_entities")

doc = nlp(text)

for ent in doc.ents:
    print(ent.text, ent.label_)

print([w.text for w in doc])

The output is:
Polygonum ORG
['Here', 'are', 'some', 'Polygonum', 'cuspidatum', '.']

"Polygonum cuspidatum" is still split. In addition, and it seems that the entity I created only extracts the word in front of the space. What's wrong with my code? Maybe I didn't understand the usage of merge_entities.

[polm]

The problem here is that en_core_web_sm has an NER component, so it's setting different entities. Since your ruler is added after the NER component, by default it doesn't overwrite existing entites, so after "Polygonum" is detected as an entity the full entity isn't detected.

A simple fix here is to use spacy.blank("en") instead of spacy.load("en_core_web_sm"). Another thing you can do is to put your entity ruler and the merge rule early in the pipeline, like this:

ruler = nlp.add_pipe("entity_ruler", first=True)
... add your patterns ...
nlp.add_pipe("merge_entities", after="entity_ruler")

[bpersevere]

polm, thank you very much. Your clear explanation and guidance helped me solve this problem and help me understand some concepts that I did not understand before.

In addition, I saw the following article when I was reading spaCy documents and searching for information on the Internet to solve problems:

https://ner.pythonhumanities.com/02_03_mwt_with_finditer.html , it seems that you can also use Span to Token entities with multiple words. But to be honest, I didn't understand this article, but I probably knew that it meant that using Span was also a solution.

In addition, when I read the spaCy document, I found that the following code can also implement multi word Tokens according to the document prompts:

import spacy
from spacy.symbols import ORTH

nlp = spacy.load("en_core_web_sm")

special_case = [{ORTH: "Polygonum cuspidatum"}]
nlp.tokenizer.add_special_case("Polygonum cuspidatum", special_case)

doc = nlp("Here are some Polygonum cuspidatum.")
print([w.text for w in doc])

The output is:

['Here', 'are', 'some', 'Polygonum cuspidatum', '.']

I don't know the three methods: entity_ ruler、span、special_ case, what is the difference between them? In terms of the situation I am facing, which one is more appropriate?

[polm]

• EntityRuler is a component in a pipeline which creates entities using rules. One simple way it can create rules is from a list of strings.
• A Span is just a part of a Doc object, like a substring is part of a string. doc[0:1] creates a Span with the first token. Entities are represented using Spans, but a Span doesn't have to be an entity.
• Special Cases modify the behavior of the tokenizer, which runs before any components in the pipeline.

If all you want to do is make your list of phrases into single tokens, either special cases or the EntityRuler + merge_entities will basically give you the same result. However, since your list of phrases seem to be entities, in this case I would use the EntityRuler based solution. In particular, this will be important if you want to re-use other components like the tagger or parser - if you change the tokenization before they run, then they'll see the merged tokens, but since their training data didn't include words with spaces, they might behave strangely.

[bpersevere]

Great. Thanks for your explanation. I will use EntityRuler. Thank you for your very useful help.