Request: New LM Adapted checkpoints for T5

[Xirider]

🌟 New LM Adapted checkpoints for T5

Description

Google released a new set of checkpoints for T5 v1.1. here:
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511

Especially interesting for most people will be the checkpoints below, as finetuning T5 with a prompt or using T5 for conditional text generation is quite common and these checkpoints promise better performance. The default T5 v1.1 checkpoints have never seen sequences without sentinel tokens.

LM-Adapted: t5.1.1.lm100k (copied from the readme)

These "LM adapted" models are initialized from t5.1.1 (above) and train for an
additional 100K steps on the LM objective discussed in the [T5 paper][paper].
This adaptation improves the ability of the model to be used for prompt
tuning.

• t5.1.1.lm100k.small (~77 million parameters): gs://t5-data/pretrained_models/t5.1.1.lm100k.small
• t5.1.1.lm100k.base (~250 million parameters): gs://t5-data/pretrained_models/t5.1.1.lm100k.base
• t5.1.1.lm100k.large (~800 million parameters): gs://t5-data/pretrained_models/t5.1.1.lm100k.large
• t5.1.1.lm100k.xl (~3 billion parameters): gs://t5-data/pretrained_models/t5.1.1.lm100k.xl
• t5.1.1.lm100k.xxl (~11 billion parameters): gs://t5-data/pretrained_models/t5.1.1.lm100k.xxl

Open source status

• the model implementation is available: t5 v1.1. with geglu
• the model weights are available: see links above
• who are the authors: Brian Lester, Rami Al-Rfou, Noah Constant