Memory management for large batch sizes

[pj-ml]

Does SGLang automatically manage memory when a large batch (e.g. 100 batch size with avg. token per sequence 6,000) is submitted for processing?

I have been getting OOM errors when increasing my batch sizes, which is surprising because I thought the memory management was automatic.

[hnyls2002]

@pj-ml, could you try reducing the argument --mem-fraction-static? When the batch size increases, the backend may require more GPU memory for temporary usage.

https://github.com/sgl-project/sglang/?tab=readme-ov-file#additional-arguments