Does it make sense to change the # hidden layers of a pretrained transformer?

[bhartm3]

Hello,
does it make sense to increase or decrease the number of hidden layers of a pretrained transformer for training a model for a new entity for example to achieve better performance? Or are the hidden layers only set once when training models from scratch?
Thank you!

[rmitsch]

Hi @bhartm3, no, it doesn't make sense to do that. At this point the model architecture is fixed.