Context Size + extra 128 tokens

[Nabokov86]

Hey! I've noticed something a bit weird with the context size. It seems to be running 128 tokens larger than what I'm setting, even when I use the --noshift --nofastforward flags. This isn't happening with llamacpp, so I thought I'd bring it up.

--contextsize 8192 --noshift --nofastforward

llama_context: n_ctx_per_seq (8320) < n_ctx_train (131072) -- the full capacity of the model will not be utilized

The GUI just shows the value I set. Any chance I can actually run the model with just the context size I'm specifying? I'm guessing it might be something to do with context shifting?

Thanks!

[LostRuins]

Hi, this is actually a (semi-hidden) feature, yes it is due to context shifting. There were cases last time where the shift failed to work with large batch sizes when there are too few KV slots left. Although this is less of an issue now, the extra padding was kept for consistency reasons since the extra memory used is basically negligible. Removing it again will require a lot of regression testing to ensure that nothing breaks across various old architectures and models so I've opted to keep it in.

To have it set to exactly 8192 you can simply set --contextsize to 8064. But I wouldn't worry about it - is it causing any issues for you?

[Nabokov86]

@LostRuins Thanks for the detailed explanation! That makes a lot of sense. It's not actually causing me any issues, I was just curious since I don't use context shifting and expected --noshift would disable the padding. But you're right, the memory difference is negligible.

Regarding the --contextsize parameter, I wanted to clarify – should I set it to the maximum context size supported by the model (e.g., 8192 for Llama 3) or should I subtract 128 tokens (e.g., 8064) to account for the padding? Thanks again!

llama_context: n_ctx_per_seq (8320) > n_ctx_train (8192) -- possible training context overflow

[LostRuins]

--contextsize is the maximum context size you want to use the model at. It simply affects the amount of memory reserved to handle that context size. You don't have to "adjust" it to the model, that's handled automatically. If it's set to larger than the model's training context, rope scaling will be applied to permit that.