Lemma of masculine nouns inconsistent in Norwegian

[Woldbrand]

Lemmatization of nouns sometimes gives out lemma of words that are non-existent in the Norwegian language. The typical way to write a masculine noun would be to remove the suffix "-en", "-er" and "-ene". In the example below, the lemma returns the suffix "-e". That might be correct for feminine lemmas ("kvinner" -> "kvinne"), but not for masc ("biler" -> "bil"). There exists masculine nouns that in singular ends with -e, but often they do not have it.

As far as I have seen, it seems to be an issue with the masculine nouns, as an example I have included the nouns:
"bilen" (def, sing, masc) and "biler" (ind, plur, masc), which gives the lemma "bile". It should be "bil", which is the ordinary way to conjugate masc nouns.
Also, "skatten" gives "skatt", which is correct, but "formueskatten" gives "formueskatte". Again, the problem is the extra "e".

How to reproduce the behaviour

# -*- coding: utf-8 -*-

import spacy
nlp = spacy.load("nb_core_news_lg")

for s in nlp("Kjøp bilen og kjør biler"):
    print(s, s.lemma_, s.tag_)

for s in nlp("Betal skatten og formueskatten"):
    print(s, s.lemma_, s.tag_)

Output

Kjøp kjøpe VERB__Mood=Imp|VerbForm=Fin
bilen bile NOUN__Definite=Def|Gender=Masc|Number=Sing
kjør kjøre VERB__Mood=Imp|VerbForm=Fin
biler bile NOUN__Definite=Ind|Gender=Masc|Number=Plur

Betal betale VERB__Mood=Imp|VerbForm=Fin
skatten skatt NOUN__Definite=Def|Gender=Masc|Number=Sing
og og CCONJ
formueskatten formueskatte NOUN__Definite=Def|Gender=Masc|Number=Sing

Your Environment

• spaCy version: 2.3.4
• Platform: Windows-10-10.0.18362-SP0
• Python version: 3.7.1
• spaCy model: nb_core_news_lg-2.3.0

[Woldbrand]

After some research I think I understand why this issue comes up. It seems to be nouns that finds the lemma by using the lemma_rules. I found the lemma_rules for Norwegian, and I see that the rule for nouns is just based upon the suffix, and not the gender.

This is the issue: Very often both female and the male conjugated nouns ends with -en, so the rule now gives out -e. The gender of the word is therefore important when defining the lemma by rules. Would you consider rules that includes both suffix and other information (gender/number/definite)? That would make the rules a lot more precise.

In the Norwegian language, it is a rule to group nouns into one noun, but the rule of conjugating follows the last noun. I therefore have another suggestion: Do the lookup in a reversed order of letters in the word and check if the letter becomes two existing nouns if splitted. As with the example above, skatten -> skatt, and formueskatten -> skatte. If you would look for words in a reversed order, then you would match the word "skatt" in formeskatt, and therefore follow the lemma of the more common base "skatt".

[adrianeboyd]

Yeah, the current built-in rule-based lemmatizer doesn't work very well for languages where gender or other features play a role like this. You would really need a custom lemmatizer that has rules based on gender and probably a lot of additional lexical information to handle irregular forms and exceptions.

I'm more familiar with German, which is probably somewhat similar, and even relying on the morphological analysis is tricky because gender is not marked in the plural, so without good lexical resources, it's very easy to get the analysis wrong.

[Woldbrand]

Understandable. A lookup-based lemmatizer would probably be better/more precise, but I have yet to find a method that replaces the existing solution. Could you point out for me where/how I can replace the existing lemmatizer with my own solution?

[adrianeboyd]

There is a lookup lemmatizer table for Norwegian, but the quality may not be great. The easiest way to try it out in a pipeline is to remove the lemma_rules table from nlp.vocab.lookups to force it to switch internally to the lookups. (There's no explicit lemmatizer config in v2, it just checks whether the tagger and certain lookups tables are present.) You can customize the lemmatizer by adding a custom lemmatizer class and create_lemmatizer to the language defaults. You can see examples for French, Dutch, or Polish, anything where there's a custom spacy/lang/__/lemmatizer.py.

The configuration is much easier in v3 because it's a separate pipeline component. Again, look for languages with a spacy/lang/__/lemmatizer.py, which override or add new lemmatizer modes. You can add a lemmatizer with a custom whatever_lemmatize method with its data tables defined for that mode="whatever" in get_lookups_config. Polish is a good example of the config setup with custom tables. Or you can do something completely separate with a 3rd party lemmatizer, like for Russian or Ukrainian.

[Woldbrand]

Great, thank you! I will look into it. Just so I understand: In both v2 and v3, is it a preference for lookup, and then rules as a secondary choice when the word does not exists in a table?

[adrianeboyd]

No, by default in v2, if there is a lemma_rules table, it switches entirely to the rule-based approach and doesn't use the lookups table at all. You can write a custom lemmatizer to handle this differently (I think French does this?). The lookups approach isn't context-sensitive and the lookups table quality is not great, so the rule-based method that uses POS is considered to be the "better" approach, at least for English. Depending on your task, though, having lookup lemmas can be easier if you're using the Matcher or want things to be less dependent on the not-100%-accurate POS tags.

In v3, the separate lemmatizer component only has one mode, so if you use lookup it only uses the lookup methods, if you use rule, it only uses the rule-based method, etc. You can write a custom method rule_lemmatize or whatever_lemmatize that uses whatever tables you'd like (again, I think French is an example here).

[Woldbrand]

Great! Thank you so much for the answers!