Skip to content
Merged
Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
950fe7e
feat(quant): block-scaled element + scale codecs (E2M1/E4M3/E5M2/E8M0)
ekryski Jun 1, 2026
24001a8
feat(quant): host packer + dequant oracle for block-scaled formats
ekryski Jun 1, 2026
a59a297
feat(quant): mxfp4 block-scaled dequant kernel + GPU correctness test
ekryski Jun 1, 2026
2814cf6
feat(quant): nvfp4 / mxfp8 (e4m3,e5m2) / nvfp8 dequant kernels + GPU …
ekryski Jun 1, 2026
ac2c549
feat(quant): mxfp4 dequantizing GEMV (Phase B core matmul) + GPU test
ekryski Jun 1, 2026
e8fd990
feat(quant): nvfp4 / mxfp8 (e4m3,e5m2) / nvfp8 dequantizing GEMVs + G…
ekryski Jun 1, 2026
f086388
feat(quant): block-scaled dequantizing GEMM (qmm) for all 5 formats +…
ekryski Jun 1, 2026
7caeda4
feat(quant): block-scaled MoE gather-GEMM for all 5 formats + GPU tests
ekryski Jun 1, 2026
ecd70aa
feat(quant): mxfp4 simdgroup-matrix (MMA) dequant GEMM + GPU test
ekryski Jun 1, 2026
164c5f9
feat(codegen): block-scaled decode DSL intrinsics (e2m1/e4m3/e5m2)
ekryski Jun 1, 2026
13d4a0b
feat(quant): nvfp4 simdgroup-matrix (MMA) dequant GEMM + GPU test
ekryski Jun 1, 2026
0224b15
feat(quant): mxfp8 (e4m3,e5m2) + nvfp8 simdgroup-matrix (MMA) GEMM + …
ekryski Jun 1, 2026
f8f5bbc
feat(bench): block-scaled qgemv precision benches (decode shape)
ekryski Jun 1, 2026
fbe3d7d
feat(bench): block-scaled qmm + MMA precision benches
ekryski Jun 1, 2026
4e082dd
docs(spec): mark Appendix B block-scaled formats implemented
ekryski Jun 1, 2026
94eecbc
feat(quant): unify framework for int8 + legacy fp4/fp8 (full-matrix f…
ekryski Jun 1, 2026
73d29b3
feat(quant): block-scaled fused RMSNorm+GEMV for all 5 formats + GPU …
ekryski Jun 1, 2026
cf0d136
feat(quant): block-scaled fused gated-RMSNorm+GEMV for all 5 formats …
ekryski Jun 1, 2026
2aab42f
feat(quant): block-scaled batched Q/K/V GEMV for all 5 formats + GPU …
ekryski Jun 1, 2026
545a5b4
feat(quant): block-scaled batched-4 GEMV + batched Q/K/V GEMM, all 5 …
ekryski Jun 1, 2026
d92bef2
feat(quant): block-scaled batched-4 GEMM for all 5 formats + GPU tests
ekryski Jun 1, 2026
e91f913
feat(quant): block-scaled dequant-gather (embeddings) for all 5 forma…
ekryski Jun 1, 2026
197f49c
feat(quant): block-scaled flash SDPA (quantized KV) for all 5 formats…
ekryski Jun 1, 2026
81d407b
feat(quant): legacy fp4/fp8 + symmetric int8 qgemv (full-matrix 2nd a…
ekryski Jun 1, 2026
7f8ad08
feat(quant): legacy fp4/fp8 + int8 on dequant / qmm / MoE gather + GP…
ekryski Jun 1, 2026
0c5e9dd
feat(quant): legacy fp4/fp8 + int8 across all fused/attention/gather …
ekryski Jun 2, 2026
d3b1bab
feat(quant): legacy fp4/fp8 + int8 on qmm-MMA (simdgroup) — full matr…
ekryski Jun 2, 2026
fd3d18d
docs(spec): mark Appendix B full op×format matrix complete
ekryski Jun 2, 2026
f5578a9
feat(quant): block-scaled MPP qmm + expert-indexed GEMV (all 9 format…
ekryski Jun 2, 2026
354e87d
feat(quant): block-scaled NAX qmm (all 9 formats) + GPU tests
ekryski Jun 2, 2026
323d296
feat(quant): block-scaled MoE gather-qmm via MPP (all 9 formats) + GP…
ekryski Jun 2, 2026
ce54ffc
docs(spec): add MPP/NAX/MoE-MPP/expert-indexed block-scaled backends …
ekryski Jun 2, 2026
3446dba
feat(quant): block-scaled MoE-MPP at bm8 + bm64 tiles (all 9 formats)…
ekryski Jun 2, 2026
e73580c
feat(quant): block-scaled quantized patch embedding (all 9 formats) +…
ekryski Jun 2, 2026
325fbb1
docs(spec): bm8/bm64 + patch_embed coverage; classify ops where quant…
ekryski Jun 2, 2026
887f67d
feat(quant): block-scaled quantized conv2d + depthwise + audio conv1d…
ekryski Jun 2, 2026
0d33cfd
feat(quant): block-scaled quantized conv3d + dilated conv1d (all 9 fo…
ekryski Jun 2, 2026
5db73eb
fix(test+quant): make the #[test_kernel] gate real, fix the bugs it e…
ekryski Jun 2, 2026
0eddc27
docs(spec): record conv coverage + the test-gate linkage fix + fp4-MM…
ekryski Jun 2, 2026
a16f2f2
feat(quant): block-scaled cooperative conv2d/conv3d-MMA (all 9 format…
ekryski Jun 2, 2026
32e39f9
docs(spec): conv MMA block-scaled done; only winograd deferred
ekryski Jun 2, 2026
24274e6
fix(quant): correct fp4 simdgroup-MMA on f32 activations
ekryski Jun 2, 2026
76e9e46
docs: document full quant-format × kernel matrix + int8 coverage
ekryski Jun 2, 2026
0286bd5
feat(quant): block-scaled flash-SDPA KV for all production head dims
ekryski Jun 2, 2026
a92e75b
feat(quant): block-scaled simdgroup-MMA patch-embed (all 9 formats)
ekryski Jun 2, 2026
299fbc9
docs: flash all-head-dim + MMA patch-embed coverage; affine int is cu…
ekryski Jun 2, 2026
4b6c049
feat(quant): int8 block-scaled flash-SDPA KV at d96 via ragged tail b…
ekryski Jun 3, 2026
0136fd9
docs(audit): trim fp_quantized_mma note to mechanism, not change history
ekryski Jun 3, 2026
545dcc9
feat(quant): symmetric intN + MXINT format foundation (codec, packing…
ekryski Jun 3, 2026
1d97af3
feat(quant): GPU dequant kernels for symmetric int2-6 + MXINT2-8
ekryski Jun 3, 2026
10e8f02
feat(quant): integer formats for qgemv + qmm + embedding-gather (wave 1)
ekryski Jun 3, 2026
000b38b
feat(quant): integer formats for fused-norm/patch-embed/batched qGEMV…
ekryski Jun 3, 2026
9c790ab
feat(quant): integer formats for batched-qmm + conv families (wave 3)
ekryski Jun 3, 2026
42345e1
feat(quant): integer formats for qmm MMA/MPP/NAX perf paths (wave 4)
ekryski Jun 3, 2026
3568d5d
feat(quant): integer formats for patch-embed-MMA + conv2d/3d-MMA (wav…
ekryski Jun 3, 2026
6e95863
feat(quant): integer formats for flash-SDPA + MoE gather + expert-ind…
ekryski Jun 3, 2026
d72b426
feat(quant): integer formats for MoE-MPP gather (bm16/bm8/bm64) (wave 7)
ekryski Jun 3, 2026
21afae5
docs: reframe quant coverage to element/scale/zero-point axes; full i…
ekryski Jun 3, 2026
3a94577
feat(quant): fp16 group-scale axis foundation + GPU dequant proof (Ph…
ekryski Jun 3, 2026
e16a2dd
feat(quant): fp16-scale formats for qgemv + qmm (Phase 2 wave 1)
ekryski Jun 3, 2026
0e9622c
feat(quant): fp16-scale formats for gather/fused-norm/patch-embed/bat…
ekryski Jun 3, 2026
ff99511
feat(quant): fp16-scale formats for batched-qmm + conv families (Phas…
ekryski Jun 3, 2026
3a236bc
feat(quant): fp16-scale formats for MMA/MPP/NAX perf paths (Phase 2 w…
ekryski Jun 3, 2026
936d226
feat(quant): fp16-scale formats for MoE gather + expert-indexed + MoE…
ekryski Jun 3, 2026
1e9eb2b
docs: fp16-scale axis implemented — quant matrix now 30 Track-1 formats
ekryski Jun 3, 2026
eb53781
feat(quant): fp16-scale formats for flash-SDPA — completes the fp16 a…
ekryski Jun 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions crates/metaltile-codegen/src/msl/features.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ pub(super) struct KernelFeatures {
pub needs_erf: bool,
pub needs_erfinv: bool,
pub needs_expm1: bool,
/// Block-scaled dequant decode helpers (`mt_decode_e2m1/e4m3/e5m2/int8`).
pub needs_decode_e2m1: bool,
pub needs_decode_e4m3: bool,
pub needs_decode_e5m2: bool,
pub needs_decode_int8: bool,
pub needs_simd_product: bool,
/// MetalPerformancePrimitives (`mpp::tensor_ops::matmul2d` / NAX) needed.
/// Detected by scanning `Op::InlineMsl::source` for `"mpp::"` — kernels
Expand All @@ -71,6 +76,10 @@ impl MslGenerator {
needs_erf: false,
needs_erfinv: false,
needs_expm1: false,
needs_decode_e2m1: false,
needs_decode_e4m3: false,
needs_decode_e5m2: false,
needs_decode_int8: false,
needs_simd_product: false,
needs_mpp: false,
};
Expand Down Expand Up @@ -189,6 +198,10 @@ impl MslGenerator {
Op::UnaryOp { op: UnaryOpKind::Erf, .. } => feat.needs_erf = true,
Op::UnaryOp { op: UnaryOpKind::ErfInv, .. } => feat.needs_erfinv = true,
Op::UnaryOp { op: UnaryOpKind::Expm1, .. } => feat.needs_expm1 = true,
Op::UnaryOp { op: UnaryOpKind::DecodeE2m1, .. } => feat.needs_decode_e2m1 = true,
Op::UnaryOp { op: UnaryOpKind::DecodeE4m3, .. } => feat.needs_decode_e4m3 = true,
Op::UnaryOp { op: UnaryOpKind::DecodeE5m2, .. } => feat.needs_decode_e5m2 = true,
Op::UnaryOp { op: UnaryOpKind::DecodeInt8, .. } => feat.needs_decode_int8 = true,
Op::FusedElementwise { ops } =>
for inner in ops {
self.analyze_op(inner, feat);
Expand Down
50 changes: 50 additions & 0 deletions crates/metaltile-codegen/src/msl/preamble.rs
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,56 @@ impl super::MslGenerator {
wl!(out, "template<typename T>");
wl!(out, "inline T mt_expm1_impl(T x) {{ return T(mt_expm1_impl(float(x))); }}");
}
// Block-scaled dequant decodes (mirror crates/metaltile-std quant::codec).
// Operand is the raw integer code; result is always f32.
if feat.needs_decode_e2m1 {
wl!(out);
wl!(out, "// E2M1 (fp4): codebook {{0,.5,1,1.5,2,3,4,6}}, sign in bit 3.");
wl!(out, "inline float mt_decode_e2m1(uint code) {{");
wl!(out, " uint m = code & 7u;");
wl!(
out,
" float mag = (m < 1u) ? 0.0f : (m < 2u) ? 0.5f : (m < 3u) ? 1.0f : \
(m < 4u) ? 1.5f : (m < 5u) ? 2.0f : (m < 6u) ? 3.0f : (m < 7u) ? 4.0f : 6.0f;"
);
wl!(out, " return (code & 8u) ? -mag : mag;");
wl!(out, "}}");
}
if feat.needs_decode_e4m3 {
wl!(out);
wl!(out, "// E4M3 (fp8): (1+m/8)*2^(e-7); subnormal m*2^-9; sign in bit 7.");
wl!(out, "inline float mt_decode_e4m3(uint bits) {{");
wl!(out, " uint e = (bits >> 3u) & 15u;");
wl!(out, " uint m = bits & 7u;");
wl!(
out,
" float mag = (e < 1u) ? (float(m) * 0.001953125f) \
: ((1.0f + float(m) * 0.125f) * exp2(float(e) - 7.0f));"
);
wl!(out, " return ((bits >> 7u) & 1u) ? -mag : mag;");
wl!(out, "}}");
}
if feat.needs_decode_e5m2 {
wl!(out);
wl!(out, "// E5M2 (fp8): (1+m/4)*2^(e-15); subnormal m*2^-16; sign in bit 7.");
wl!(out, "inline float mt_decode_e5m2(uint bits) {{");
wl!(out, " uint e = (bits >> 2u) & 31u;");
wl!(out, " uint m = bits & 3u;");
wl!(
out,
" float mag = (e < 1u) ? (float(m) * 0.0000152587890625f) \
: ((1.0f + float(m) * 0.25f) * exp2(float(e) - 15.0f));"
);
wl!(out, " return ((bits >> 7u) & 1u) ? -mag : mag;");
wl!(out, "}}");
}
if feat.needs_decode_int8 {
wl!(out);
wl!(out, "// Symmetric int8: reinterpret the byte as i8 (sign-extend low 8 bits).");
wl!(out, "inline float mt_decode_int8(uint bits) {{");
wl!(out, " return float(int(bits << 24u) >> 24);");
wl!(out, "}}");
}
if feat.needs_simd_product {
wl!(out);
// simd_size is only accessible as a kernel attribute, not in free functions.
Expand Down
10 changes: 10 additions & 0 deletions crates/metaltile-codegen/src/passes/type_check.rs
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,16 @@ fn infer_block(
env.insert(vid, TypedValue { dtype: tv.dtype, shape: tv.shape.clone() });
},

// Block-scaled decodes consume an integer code and always produce
// f32, so their result dtype is f32 regardless of the operand dtype.
Op::UnaryOp { op, value } if op.yields_f32() =>
if let Some(tv) = env.get(value) {
env.insert(vid, TypedValue {
dtype: metaltile_core::dtype::DType::F32,
shape: tv.shape.clone(),
});
},

Op::UnaryOp { value, .. } | Op::Activation { value, .. } => {
if let Some(tv) = env.get(value) {
env.insert(vid, TypedValue { dtype: tv.dtype, shape: tv.shape.clone() });
Expand Down
29 changes: 29 additions & 0 deletions crates/metaltile-core/src/ir/op.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,19 @@ pub enum UnaryOpKind {
Log10,
/// Inverse error function: `erfinv(x)`.
ErfInv,
/// Decode a 4-bit E2M1 (fp4) code → signed f32 (codebook
/// `{0,.5,1,1.5,2,3,4,6}`, sign in bit 3). Operand is the `u32` code;
/// result is always f32. See `quant::codec::e2m1_decode`.
DecodeE2m1,
/// Decode an 8-bit E4M3 (fp8) byte → f32 (`(1+m/8)·2^(e-7)`, ±448, no inf).
/// Used for fp8 elements and nvfp4 micro-scales. See `quant::codec::e4m3_decode`.
DecodeE4m3,
/// Decode an 8-bit E5M2 (fp8) byte → f32 (IEEE-half high byte).
/// See `quant::codec::e5m2_decode`.
DecodeE5m2,
/// Decode a symmetric-int8 code (a `u8` reinterpreted as `i8`) → f32. The
/// per-group float scale is applied separately. See `quant::codec::int8_decode`.
DecodeInt8,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't these be expressed via DSL? Ideally we don't hardcode algos into the DSL, but rather provide all needed primitives to easily do whatever you need.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anything can be expressed in the DSL. Just isn't right now and I didn't focus on optimizing as I'm still adding more kernels in this PR. Just got rebased after all the CI churn so I could ensure build is passing and see the clear diff.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be a good opurtunity for starting primitives file? Kernels that other kernels use?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with shipping this for now, can revisit ops later, lets just try to avoid new ops where DSL can easily express it.

}

impl UnaryOpKind {
Expand Down Expand Up @@ -97,8 +110,24 @@ impl UnaryOpKind {
UnaryOpKind::Expm1 => format!("mt_expm1_impl({arg})"),
UnaryOpKind::Log10 => format!("log10({arg})"),
UnaryOpKind::ErfInv => format!("mt_erfinv_impl({arg})"),
UnaryOpKind::DecodeE2m1 => format!("mt_decode_e2m1({arg})"),
UnaryOpKind::DecodeE4m3 => format!("mt_decode_e4m3({arg})"),
UnaryOpKind::DecodeE5m2 => format!("mt_decode_e5m2({arg})"),
UnaryOpKind::DecodeInt8 => format!("mt_decode_int8({arg})"),
}
}

/// True for the quant decode ops, whose result is always f32 regardless of
/// the (integer) operand dtype.
pub fn yields_f32(self) -> bool {
matches!(
self,
UnaryOpKind::DecodeE2m1
| UnaryOpKind::DecodeE4m3
| UnaryOpKind::DecodeE5m2
| UnaryOpKind::DecodeInt8
)
}
}

/// Neural activation function kind.
Expand Down
5 changes: 5 additions & 0 deletions crates/metaltile-macros/src/kernel/body.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2181,6 +2181,11 @@ fn unary_op_kind(name: &str) -> Option<TokenStream> {
"expm1" => quote! { UnaryOpKind::Expm1 },
"log10" => quote! { UnaryOpKind::Log10 },
"erfinv" => quote! { UnaryOpKind::ErfInv },
// Block-scaled dequant decodes: u32 code → f32 (see quant::codec).
"e2m1_decode" => quote! { UnaryOpKind::DecodeE2m1 },
"e4m3_decode" => quote! { UnaryOpKind::DecodeE4m3 },
"e5m2_decode" => quote! { UnaryOpKind::DecodeE5m2 },
"int8_decode" => quote! { UnaryOpKind::DecodeInt8 },
_ => return None,
})
}
Expand Down
Loading
Loading