Skip to content

Gemma3 Multimodal optimization #404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Gemma3 Multimodal optimization #404

wants to merge 8 commits into from
+450 −22

Conversation

jiminha
Copy link
Contributor

@jiminha jiminha commented Oct 14, 2025

This is to optimize gemma3 multimodal memory/performance.

  • bucket vision tower based on batch bucket to reduce recompile overhead
  • modify merge_multimodal to use torch.where instead of masked_scatter for performance issue
  • add warmup multimodal bucket to precompile vision tower
  • port PT_HPU_SDPA_QKV_SLICE_MODE_FWD feature from vllm-fork v0 : this is necessary to reduce the memory for the longer sequence length.

@github-actions GitHub Actions
Copy link

🚧 CI Blocked

The main CI workflow was not started for the following reason:

Your branch is behind the base branch. Please merge or rebase to get the latest changes.

@jiminha jiminha marked this pull request as draft October 14, 2025 14:16
jiminha and others added 6 commits October 15, 2025 11:54
@jiminha
Initial code for multimodal warmup for Gemma
9f6c2b8 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Update warmup
4ade49e 
Signed-off-by: Jimin Ha <[email protected]>
@MohitIntel @jiminha
Overiding the _process_image_input for Gemma3
d5b7c49 
Signed-off-by: Mohit Deopujari <[email protected]>
@jiminha
Compile error fix on latest vllm merge
e63d1ef 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Update _merge_multimodal_embedding for perf issue in HPU
29d3f18 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Add FusedSDPA kernel with sliding window support for Gemma3
e2238fc 
Reduces memory usage for long sequences by eliminating dual attention
mask creation. Improves capacity from 150 to 400 images with 8K prompts
by avoiding OOM issues.
Limitation: Only available when block_list is None.

Signed-off-by: Jimin Ha <[email protected]>
@github-actions GitHub Actions
Copy link

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@jiminha jiminha marked this pull request as ready for review October 15, 2025 20:17
@jiminha
Copy link
Contributor Author

jiminha commented Oct 15, 2025

@xuechendi Could you review this? This includes model file and some utils changes which are necessary for gemma3 model optimization.
@adobrzyn Could you review this? This includes multimodal warmup, added vision bucket into bucketing. Also ported PT_HPU_SDPA_QKV_SLICE_MODE_FWD feature from V0.

@github-actions GitHub Actions
Copy link

✅ CI Passed

All checks passed successfully against the following vllm commit:
f57438338d819c8e3e7e70293281c575ebd77411

@github-actions GitHub Actions
Copy link

🚧 CI Blocked

The main CI workflow was not started for the following reason:

Your branch is behind the base branch. Please merge or rebase to get the latest changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kzawora-intel kzawora-intel Awaiting requested review from kzawora-intel kzawora-intel is a code owner

@xuechendi xuechendi Awaiting requested review from xuechendi xuechendi is a code owner

@mswiniarsk mswiniarsk Awaiting requested review from mswiniarsk mswiniarsk is a code owner

@adobrzyn adobrzyn Awaiting requested review from adobrzyn adobrzyn is a code owner

@mgawarkiewicz-intel mgawarkiewicz-intel Awaiting requested review from mgawarkiewicz-intel mgawarkiewicz-intel is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

@afierka-intel afierka-intel Awaiting requested review from afierka-intel afierka-intel is a code owner

@michalkuligowski michalkuligowski Awaiting requested review from michalkuligowski michalkuligowski is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jiminha @MohitIntel