How to add new language to spacy_lookups_data?

[vistamou]

How can I integrate a new language in spacy-lookups-data?
I keep getting a KeyError: 'spacy_lookups_data/data/hr_lemma_lookup.json'

[polm]

The latest version of spacy-lookups-data includes hr_lemma_lookup.json. Maybe you need to upgrade?

If upgrading doesn't fix it, can you clarify:

1. What version of spaCy are you using?
2. What version of spacy-lookups-data are you using?
3. What code is giving you that error?

[polm]

Also to answer the question about adding new languages: you need to get the data, get it in the right format (not complicated), and make a PR. For us the main issues when adding files are 1. is the license OK? 2. is the content reasonable (not too large, decent quality)?

This recent PR (explosion/spacy-lookups-data#28) might be a decent reference.

[vistamou]

Dear Paul,

I have spacy-lookups-data : version = "1.0.3" (downloaded from github)
and spaCy version 3.1.4

actually what I did it is I made the respective changes to files included in the data folder (json file), in the init.py,
and in the setup.cfg -- including the language code for new language and the data, but it doesn't read the language
and throws an error (although I have defined the language 'xx' and i can import without problem!):

raise ValueError(Errors.E955.format(table=", ".join(tables), lang=lang))
ValueError: [E955] Can't find table(s) lemma_lookup for language 'xx' in spacy-lookups-data. Make sure you have the package installed or provide your own lookup tables if no default lookups are available for your language.

the command I ran is the one required to train a pipeline: python -m train config.cfg outputpath --paths.train --paths.dev

Then to skip this issue, I replaced the content of the hr language with my data
and I got the KeyError problem

[adrianeboyd]

The tables are identified through entry points, so you need to be sure that the new entry point is in setup.cfg and then reinstall spacy-lookups-data from source so that the entry points are registered as part of the package.

[vistamou]

Dear Adriane,

Thanks a lot!! I followed those steps and everything is working as expected,
but the problem is that it's working only inside in the directory of spacy_lookups_data,
how can I make the changes to be visible also in the directory of spacy?

[adrianeboyd]

That sounds like it's still not installed correctly. Try uninstalling with pip uninstall spacy-lookups-data several times in a row (especially if you had an editable install) until it says it's not installed, and then install again from source with pip install ..

[vistamou]

Thank you!! solved :)