Using TextCategorizer on Span instead of Doc

[NixBiks]

I have a pipeline that first split my large document into sentences using the sentenziser. Then I want to classify each sentence. I could have a sentence pipeline (tokenize and predict) and use that on the sentence but I assume I could save the tokenization (since its already done).

My current idea is to extend TextCategorizer and override __call__ to do nothing, add Span._.cats as an extension attribute and add that component to my one and only pipeline. Not sure if I am over complicating things though?

Info about spaCy

• spaCy version: 2.2.3
• Platform: Linux-5.3.0-45-generic-x86_64-with-glibc2.2.5
• Python version: 3.8.2

[NixBiks]

Actually now when I think about it. I guess I should

text_cat = TextCategorizer(nlp.vocab)
text_cat.from_disk("/model/textcat", exlude=["vocab"])
text_cat.predict([span])  # and add some logic to link with labels

[svlandeg]

Can't you use span.as_doc() ?

[NixBiks]

Now that I know of span.as_doc() then yes (didn't notice that one). So I'd change the last line to

text_cat(span.as_doc()).cats

Right?

[svlandeg]

Yep!

[NixBiks]

Thanks a bunch!