Skip to content

fix(kernels): catch C++ exceptions at the FFI boundary and surface the message#539

Merged
xiaguan merged 4 commits into
openinfer-project:mainfrom
FeathBow:fix/kernels-ffi-exception-catch
Jul 4, 2026
Merged

fix(kernels): catch C++ exceptions at the FFI boundary and surface the message#539
xiaguan merged 4 commits into
openinfer-project:mainfrom
FeathBow:fix/kernels-ffi-exception-catch

Conversation

@FeathBow

@FeathBow FeathBow commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Description

Fixes #520

FlashInfer can throw std::exception from host-side dispatch before any CUDA kernel launch. When that exception crosses an extern "C" boundary, the process aborts without surfacing the message. This was hit by Qwen3-14B: 40/8 heads -> GQA group size 5, which is not instantiated by DISPATCH_GQA_GROUP_SIZE.

  • Add csrc/shared/ffi_guard.cuh + ffi_last_error.cu: guarded FFI entries catch C++ exceptions at the boundary, store what() in a thread_local, clear stale messages on entry, and return a -1 sentinel.
  • Rust wrappers append the stored message on -1; ffi_exception_message is public for direct-FFI callers outside the kernels crate. The qwen35 hd256 prefill path and qwen3 kernel bench now append the message too.
  • SUPPORTED_GQA_GROUP_SIZES (&[1, 2, 3, 4, 8], matching the FlashInfer dispatch table) is exported from the kernels crate; qwen3 config load rejects unsupported group sizes up front.
  • glm52 top-k now returns i32 at the FFI declaration so the -1 sentinel cannot materialize an invalid CUresult.

The new FFI test drives decode at group 7 and asserts -1 plus a message naming the group size, instead of a process abort.

Test Env

Single GPU (sm_89, x86_64):

  • FFI test green.
  • Qwen3-4B (group 4) loads and serves unchanged.
  • Qwen3-14B now fails at load with unsupported GQA group size 40/8; the attention kernels instantiate [1, 2, 3, 4, 8].
  • kimi-k2 and glm52 feature builds compile clean with the guard additions (cargo check targeting sm_90 with the full moe dependency chain).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 154764075e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread openinfer-kernels/src/ffi/glm52/topk.rs
@FeathBow FeathBow force-pushed the fix/kernels-ffi-exception-catch branch from 1547640 to b188365 Compare July 3, 2026 18:25
@xiaguan xiaguan merged commit 4779407 into openinfer-project:main Jul 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernels: C++ exceptions crossing the extern "C" FFI boundary abort with no logged message

2 participants