Out now: spaCy v3.0 nightly

[ines]

spaCy v3.0 is available as a nightly pre-release 🎉 Give it a try and let us know what you think! Feel free to ask questions in this thread, and report any bugs you come across on the issue tracker.

💫 spaCy v3.0 in a nutshell

The new version features all new transformer-based pipelines that bring spaCy's accuracy right up to the current state-of-the-art. You can use any pretrained transformer to train your own pipelines, and even share one transformer between multiple components with multi-task learning. Training is now fully configurable and extensible, and you can define your own custom models using PyTorch, TensorFlow and other frameworks. The new spaCy projects system lets you describe whole end-to-end workflows in a single file, giving you an easy path from prototype to production, and making it easy to clone and adapt best-practice projects for your own use cases.

📖 Read more

• Introducing spaCy v3.0 nightly: Blog post introducing the nightly release
• spaCy nightly: New website and documentation
• What's new in spaCy v3.0 (nightly): New features and all changes
• v3.0.0rc release notes: Official GitHub release
• Migration guide from v2.x to v3.x: How to update your code
• spaCy models directory: Download trained pipelines
• spaCy project templates: End-to-end NLP workflows

[thiippal]

I am somewhat confused about the relationship between spaCy and spacy-transformers when moving to v3.0, particularly when it comes to some of the neat functionalities in spacy-transformers.

Is it still possible to extract Token level representations from the pre-trained Transformer pipeline en_core_web_trf so that they would be aligned with spaCy Tokens?

[adrianeboyd]

The underlying transformer data including the alignment is saved in doc._.trf_data as TransformerData: https://nightly.spacy.io/api/transformer#transformerdata

[thiippal]

@adrianeboyd thanks for the quick reply!

If I understood correctly, the Token representations are under doc._.trf_data.tensors[0] – but (1) what are the input_ids under doc._.trf_data.tokens and (2) how can they be mapped to the tensors?

Thanks again!

[adrianeboyd]

The data in trf_data.align[index] shows you the indices of the transformer tokens from trf_data.tokens['input_texts'] that align with doc[index]. trf_data.tokens['input_ids'] are the matching internal token ids for the transformer tokenizer. They're not related to any of spacy's internal integer IDs.