Multi-word static vectors

[bpben]

I'm designing a text classification model that I'd like to be able to pass some additional features to (beyond those learned by the tok2vec component). These features are typically not token-specific but apply to Spans of tokens. Based on my reading/playing with the static vector functionality in model training, it seems like static vectors need to be keyed on individual words or tokens. I was looking into sense2vec, but it seems like the existing model architectures aren't designed to access sense2vec features. So I guess I have a couple questions:

1. Is this a reasonable thing to try and do with spacy? I feel like I might be trying to shoehorn a multi-modal problem into an NLP solution.
2. Am I right here in thinking I'd need to design my own model architecture to be able to access Span-level vectors?
3. Has anyone done work here before? It'd be great to see how others have implemented something like this or generally solved an issue like this.

Thanks!

[polm]

I haven't heard of anyone doing this before, but maybe you could just merge the tokens before the tok2vec step? If you use your vectors instead of (as opposed to in addition to) the standard spaCy vectors and train from scratch that should work. It would be like using a custom tokenizer.

Based on your description it's not clear if your vectors have good coverage or if they're only for select entities. If they're only for select entities, you could use them as a seed and pretrain to get better coverage while retaining some of whatever you've learned. Getting a balance might be difficult though.

Using your vectors with the existing tok2vec, as a separate feature source, is a multimodal task and there isn't really a straightforward way to do that at the moment. If you're feeling adventurous you can look at how HashEmbed is implemented (particularly the way shape features are used) and write something similar, but I think it might be a lot of work.

[bpben]

Super helpful suggestions, thanks. Yeah I guess I'm thinking of the vectors + tok2vec, but you think that probably will be more difficult? The tok2vec component seems to learn some useful things, but I think the vectors would boost that.

Re: merging tokens, would the attributes still work in that case? I was using a baseline set of features for the hash embedding, not sure how that works if I'm doing my own tokenization.

[polm]

Yeah I guess I'm thinking of the vectors + tok2vec, but you think that probably will be more difficult? The tok2vec component seems to learn some useful things, but I think the vectors would boost that.

Combining your vectors would be the most complicated thing if working within spaCy, yes. You could make a model with a special vectors member that concatenates your values with pretrained ones and train a tok2vec on top of that. You'd have to figure out how you want to handle multi-word tokens. If your vectors don't have good coverage I'm not sure how effective this would be.

Re: merging tokens, would the attributes still work in that case? I was using a baseline set of features for the hash embedding, not sure how that works if I'm doing my own tokenization.

The attributes used in hash embed are things like shape/prefix/suffix so they work the same after merging tokens as before merging tokens. The main difference would be that since tokens can contain spaces that might make the features sparse for multi-word tokens.