请问RWKV Context长度可以无限外推么。 #179
-
|
RWKV Context长度可以无限外推么，对于超出训练长度的文本有没有性能的损失呢？ |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Hello! The context length of RWKV is theoretically infinity, while in reality, it is limited by the context length on which it was trained, as well as numerical precision, and the size of the hidden state. |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/RWKV/RWKV-infctx-trainer Is this project capable of training with unlimited context length, and is it officially supported? |
Beta Was this translation helpful? Give feedback.
Hello! The context length of RWKV is theoretically infinity, while in reality, it is limited by the context length on which it was trained, as well as numerical precision, and the size of the hidden state.
As a result, you might encounter some performance decrease when the context length is longer than that being trained.
See 128k context length here:
https://huggingface.co/xiaol/rwkv-7B-world-novel-128k
(Possibly the open-source model with the longest context length)