Explainability on Spacy trained custom models

[geethaRam]

We are using Spacy's pretrained model to fine tune a custom model using Spacy's custom training. Checking to find out how to explain the Spacy's custom trained model's predictions. Have checked in other stack overflow space and could not find any reference to this topic.

Info about spaCy

• spaCy version: 2.2.4
• Platform: Linux-4.14.171-105.231.amzn1.x86_64-x86_64-with-glibc2.9
• Python version: 3.6.5
• Models: en

[itsshavar]

It's not an issue. It's a feature more like an enhancement.

[geethaRam]

I was not sure if this this is an Enhancement or Training. That is why have not tagged on any type

[geethaRam]

@ines and @honnibal - Explainability frameworks like alibi require prediction function and not sure how to expose that from the spacy's functions. Would this be a new feature added to Spacy or could we use existing functions ? Interpretability on custom trained models is a topic that many will be interested for enterprise applications. Spacy works extremely well for fine tuning a pretrained model with smaller labeled dataset. Ability to explain and interpret the model's predictions will be an extremely valuable feature.
Any pointers on this is greatly appreciated .

[svlandeg]

Definitely agree that explainability of ML models is an important field. And actually alibi uses spaCy under the hood, for tokenization, though I don't know the details. I think it should also work for explaining spaCy models?

Either way, this kind of functionality is probably best implemented as an external plugin. It's not currently on our road map to be part of the core library. If anyone would implement such an external plugin though, to analyse spaCy models, we'd be very happy to add it to the spacy universe (alibi is already there).