Add Bamba Model

[fabianlim]

This is the companion PR to an huggingface PR for adding Bamba, which is a hybrid mamba2 architecture with SwiGLU. The checkpoints are jointly trained by IBM, Princeton, and UIUC.

In this PR we have:

• Created the bamba model inference architecture, which we would like acknowledge the jamba team for referencing their implementation, whereby we modified to support full attention layers with RoPE and mamba v2.
• Ensured that we have TP support.
• Ensured we support chunked prefill. Currently we have a partial solution, which works only when the cont batch boundaries line up with the chunked boundaries. This is now completely fixed.
• Ensured that we conform to the recent PR for adding pipeline support for SSM models.
• Adapted the mamba v2 scan kernels into vllm/model_executor/layers/mamba/ops. Only the fwd kernels are extracted. Some modifications and fixes are made.
• created tests/models/decoder_only/language/test_bamba.py with an initial ibm-fms/Bamba-9.8b-1.8T-hf. This is practically identical to test_mamba.py, only chunked prefill tests are disabled as it is currently not supported.

Currently only FlashAttention backend is supported, as we check fields like context_lens_tensor. Have not yet investigated other backends.

We would like to also acknowledge the draft codestral mamba PR from @tlrmchlsmth, which we also referenced the mixer.

• we made a few simplications for bamba (simplified mixer from mamba v2)
• Cuda graph capturing seems to be working, but we understand that cudagraphs are disabled for long sequence lengths. For SSM models the strength is in this regime, so can we handle it better?

Hope to discuss the following with the maintainers

1. do we have to remove all the bwd kernels? yes we should
2. for the full attention layers, we increase the sin_cos cache to cover the sequence length, if it is longer than max_sequence_len. This differs for other current models (e.g., llama). How can we better support long sequence lengths? we should keep this consistent with other models, so we propose to allow the sin_cos cache extension only when VLLM_ALLOW_LONG_MAX_MODEL_LEN is specified.
3. have some ideas to support chunked pre-fill, but will appreciate some discussion with the maintainers on how to proceed. working on changing the kernels to support chunked prefill.
4. since the mixer2 is simplified from mamba, should we rename it? we can keep it as is, but we should document the differences from mamba_ssm

cc: @ani300, @raghukiran1224, @cyang49, @njhill

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[tlrmchlsmth]

Hi @fabianlim, thanks for the PR! It's really great to see progress being made on state-space models, especially for me as I unfortunately haven't been able to prioritize support for Mamba2

I'm happy to shepherd this PR and discuss any questions you have, especially to support chunked prefill. If you haven't already, can you join the developer slack for quicker discussion? (https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack)

[fabianlim]

@tlrmchlsmth I cleaned up the PR quite abit, perhaps it might be a good time to get some early eyes. The chunked prefill implementation is incomplete ATM, as we discussed offline.

[tlrmchlsmth]

first pass, just a few comments. At a high level it looks good.

Will you add a test for tensor parallelism?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fabianlim.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[fabianlim]

@tlrmchlsmth i have addressed most of your comments now, not rebasing yet, waiting for you to look first. But I realized test_jamba.py has changed so I will need to do the rename and test again.

[tlrmchlsmth]

@fabianlim At a high level, the changes look good, and the PR looks good overall. I'll do a more thorough review once it's unmarked as draft.

Could you add unit tests for the added kernels in layers/mamba/ops?

[fabianlim]

@tlrmchlsmth thank you so much for your comments. I have fixed the pre-commit and reverted changes in a bunch of files; i kept the changes that are still needed to test the PR. Also merging in upstream/main helped github to display the changes correctly. Regarding the new unit test, I left a question for you

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fabianlim.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[tlrmchlsmth]

I've been trying to get this PR green. Some of the issues are due to huggingface timeouts, but the kernels-test-3 failure is real. The log is too large to link to the problematic lines, but there is an illegal memory access during test_mamba_chunk_scan_cont_batch.

[fabianlim]

@tlrmchlsmth oh man i cant reproduce the illegal memory access in my machine. Im trying to debug this by modifying the CI file to just trigger this particular test in the CI

Update: yes if I run these tests only in the CI it passes, suggesting that the invalid memory access might be coming from somewhere else.

Update: when disabling the test case it passes .

python -m pytest -x tests/kernels/test_mamba_ssm_ssd.py::test_mamba_chunk_scan_cont_batch
============================================== test session starts ==============================================
platform linux -- Python 3.9.19, pytest-8.3.4, pluggy-1.5.0
rootdir: /gpfs/users/flim/data/vllm
configfile: pyproject.toml
plugins: anyio-4.8.0
collected 192 items

tests/kernels/test_mamba_ssm_ssd.py ..................................................................... [ 35%]
......................................................................................................... [ 90%]
..................                                                                                        [100%]

======================================= 192 passed in 2713.89s (0:45:13) ========================================