SpanCategorizer gives 0 scores for Danish language

[mirfan899]

How to reproduce the behaviour

I have converted the Danish datasets to SpaCy SpanCategorizer format. Here is screenshot of format

Here is debug data command output.

python -m spacy debug data config.cfg --ignore-warnings --verbose --no-format --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy 
Data file validation
Pipeline can be initialized with data
Corpus is loadable
Training stats
Language: da
Training pipeline: tok2vec, spancat
148 training docs
40 evaluation docs
It's recommended to use at least 2000 examples (minimum 100)
Vocab & Vectors
7130 total word(s) in the data (522 unique)
10 most common words: 'de' (318), '...' (280), 'det' (196), 'er' (174), 'og' (174), '.' (172), ',' (168), 'har' (166), 'en' (130), 'deres' (116)
No word vectors present in the package
Span Categorization

Spans Key   Labels                        
---------   ------------------------------
sc          {'Price', 'People', 'Proces', 'Product', 'Place'}

Label counts in train data: 
Key: sc, 'Product' (70), 'Place' (6), 'People' (66), 'Proces' (4), 'Price' (2)
Span characteristics for spans_key 'sc'
SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type      Length     SD     BD    N
------------   ------   ----   ----   --
Product          2.20   1.98   1.79   70
Place            3.00   3.56   3.06    6
People           2.19   2.19   1.98   66
Proces           1.73   4.60   4.90    4
Price            1.00   8.18   5.52    2
------------   ------   ----   ----   --
Wgt. Average     2.20   2.29   2.06    -

Over 90% of spans have lengths of 1 -- 4 (min=1, max=5). The most common span lengths are: 1 (22.97%), 2 (28.38%), 3 (33.78%), 4 (10.81%). If you are using the n-gram suggester, note that omitting infrequent n-gram lengths can greatly improve speed and memory usage.
Full distribution of span lengths: 1 (22.97%), 2 (28.38%), 3 (33.78%), 4 (10.81%), 5 (4.05%)
Spans are distinct from the rest of the corpus
10 most common span tokens: 'omkostninger', 'det', 'ikke', 'interesse', 'en', 'med', 'højsæson', 'materialet', 'deres', 'og'
Boundary tokens are distinct from the rest of the corpus
10 most common span boundary tokens: ',', 'har', 'masse', 'som', 'lige', 'igang', 'sendt', 'videre', '...', '.'
To train a new span type, your data should include at least 50 instances of the new label
Examples without ocurrences available for all labels
Summary
5 checks passed

My config.cfg

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "da"
pipeline = ["tok2vec","spancat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3,4,5,6,7,8]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Model training logs

python -m spacy train config.cfg --output ./output --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy
ℹ Saving to output directory: output
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2023-02-17 22:30:10,359] [INFO] Set up nlp object from config
[2023-02-17 22:30:10,368] [INFO] Pipeline: ['tok2vec', 'spancat']
[2023-02-17 22:30:10,371] [INFO] Created vocabulary
[2023-02-17 22:30:10,371] [INFO] Finished initializing nlp object
[2023-02-17 22:30:10,576] [INFO] Initialized pipeline components: ['tok2vec', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'spancat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
  0       0         67.87        806.15        0.13        0.06       40.00    0.00
  2     200        121.03       1831.67        0.00        0.00        0.00    0.00
  5     400          0.76        458.56        0.00        0.00        0.00    0.00
  9     600          0.28        484.45        0.00        0.00        0.00    0.00
 13     800          0.20        539.33        0.00        0.00        0.00    0.00
 19    1000          0.22        607.30        0.00        0.00        0.00    0.00
 26    1200          0.31        661.08        0.00        0.00        0.00    0.00
 35    1400          0.40        764.32        0.00        0.00        0.00    0.00
 47    1600          0.77        837.79        0.00        0.00        0.00    0.00
✔ Saved pipeline to output directory
output/model-last

But model is giving very strange spans.

import spacy
from spacy import displacy
nlp = spacy.load("output/model-best")
doc = nlp("Slet ikke behov for er bare en lille forening med ikke mange indtægter eller udgifter.")
doc.spans

{'sc': [Slet, Slet, Slet, Slet, ikke, ikke, behov, behov, for, for, er, bare, en, lille, lille, lille, forening, forening, forening, forening, med, mange, mange, mange, indtægter, eller, eller, eller, udgifter, udgifter, udgifter, udgifter, ., ., ., ikke behov, behov for, behov for, for er, bare en, en lille, en lille, lille forening, lille forening, forening med, ikke mange, mange indtægter, mange indtægter, indtægter eller, eller udgifter, eller udgifter, udgifter., Slet ikke behov, ikke behov for, ikke behov for, for er bare, for er bare, er bare en, er bare en, bare en lille, bare en lille, en lille forening, lille forening med, forening med ikke, med ikke mange, mange indtægter eller, indtægter eller udgifter, eller udgifter., Slet ikke behov for, ikke behov for er, ikke behov for er, behov for er bare, for er bare en, er bare en lille, bare en lille forening, bare en lille forening, en lille forening med, forening med ikke mange, forening med ikke mange, ikke mange indtægter eller, mange indtægter eller udgifter, indtægter eller udgifter., indtægter eller udgifter., ikke behov for er bare, behov for er bare en, for er bare en lille, er bare en lille forening, bare en lille forening med, lille forening med ikke mange, lille forening med ikke mange, lille forening med ikke mange, ikke mange indtægter eller udgifter, mange indtægter eller udgifter., mange indtægter eller udgifter., mange indtægter eller udgifter., ikke behov for er bare en, ikke behov for er bare en, for er bare en lille forening, for er bare en lille forening, er bare en lille forening med, bare en lille forening med ikke, en lille forening med ikke mange, lille forening med ikke mange indtægter, forening med ikke mange indtægter eller, Slet ikke behov for er bare en, ikke behov for er bare en lille, ikke behov for er bare en lille, behov for er bare en lille forening, for er bare en lille forening med, er bare en lille forening med ikke, bare en lille forening med ikke mange, Slet ikke behov for er bare en lille, Slet ikke behov for er bare en lille, ikke behov for er bare en lille forening, ikke behov for er bare en lille forening, ikke behov for er bare en lille forening, for er bare en lille forening med ikke, er bare en lille forening med ikke mange, er bare en lille forening med ikke mange, er bare en lille forening med ikke mange, er bare en lille forening med ikke mange, bare en lille forening med ikke mange indtægter, lille forening med ikke mange indtægter eller udgifter]}

Your Environment

Info about spaCy

• spaCy version: 3.5.0
• Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.27
• Python version: 3.8.9
• Pipelines: da_core_news_md (3.5.0)

[mirfan899]

Is there anyone here?

[svlandeg]

Hi! You have submitted this question on a Friday after European working hours, only to ping us again on Monday morning before European working hours because you haven't received a response yet. Please consider how this comes across.

I will move your question to the discussion board as the issue tracker is only meant for bug reports.

[mirfan899]

I think I have found the issue.

If text has multiple spans and text is repeated for each span, it will not work i.e.

Slet ikke behov for er bare en lille forening med ikke mange indtægter eller udgifter. {"sc": [bare en lille forening med ikke]}
Slet ikke behov for er bare en lille forening med ikke mange indtægter eller udgifter. {"sc": [med ikke mange indtægter eller udgifter]}
Slet ikke behov for er bare en lille forening med ikke mange indtægter eller udgifter. {"sc": [mange indtægter eller udgifter]}

Text should not be repeated for the spans. I tried it and it works.

Slet ikke behov for er bare en lille forening med ikke mange indtægter eller udgifter. {'sc': [bare en lille forening med ikke, med ikke mange indtægter eller udgifter, mange indtægter eller udgifter]}