Inference Speed for Long Articles

[saptarshi059]

Is there any way to increase the generation speed for extremely long articles, such as 5000 tokens long? I've been trying to apply several optimization tricks, but none seems to work. Or is it just the case that text-generation in general for such long spans WILL be slow and there's no way around it?

[magicknight]

Is there any way to increase the generation speed for extremely long articles, such as 5000 tokens long? I've been trying to apply several optimization tricks, but none seems to work. Or is it just the case that text-generation in general for such long spans WILL be slow and there's no way around it?

How to generate 5000 tokens?

[saptarshi059]

So there's no way to directly generate ~5000 tokens,.. that's a limitation of any decoder-based models since they can only process tokens up to their maximum input length which in this case is 2048,.. what I was doing then is,. generate 2048 tokens,.. and then use the last N (say 150) tokens as input to generate new text (almost like a sliding window).. in this way I saw that the ultimate text was reasonably coherent..