Skip to content

opencl: transposed gemm/gemv moe kernel with mxfp4,f32 #16602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

opencl: transposed gemm/gemv moe kernel with mxfp4,f32 #16602

wants to merge 2 commits into from
+562 −4

Conversation

shawngu-quic
Copy link
Contributor

Added redesigned moe-mxfp4 kernels optimized for Adreno:

  • Added pref-transpose for experts, fused with SOA convert kernel.
  • Preprocess router table on CPU side.
  • Separated decoding and prefill new kernels for MoE-mxfp4.

Achieved large perf uplift for prefill, especially for long prompts.

@github-actions github-actions bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Oct 15, 2025
@shawngu-quic shawngu-quic force-pushed the moe-mxfp4-trans-reorder branch from cb1e386 to 2ca4fe1 Compare October 16, 2025 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lhez lhez Awaiting requested review from lhez lhez is a code owner

@max-krasnyansky max-krasnyansky Awaiting requested review from max-krasnyansky max-krasnyansky is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shawngu-quic