Training sharing transformer layer

[fcggamou]

What is the recommended approach to train multiple pipeline components that would share the same transformer layer?
Should I freeze the transformer layer after training one component and then train the rest of them?

Thanks for any feedback

[svlandeg]

If you have data annotated with multiple tasks, there is no reason why you can't train the transformer simultaneously on all these tasks. You can even set the grad_factor to different values to weigh some components more than others (cf https://spacy.io/api/architectures#TransformerListener).

If you're training in multiple stages, then yes one solution is to first train a transformer+component, then freeze and train a second component on top of the frozen transformer. It's difficult to say upfront what the effect on accuracy would be though.