Skip to content

portfolio_case_study.md

Latest commit

 

History

History
173 lines (118 loc) · 6.35 KB

portfolio_case_study.md

File metadata and controls

173 lines (118 loc) · 6.35 KB
Raw

CUDA SGEMM Optimization: Naive vs Shared-Memory Tiled Matrix Multiplication

By: Sana Ul Mustafa Qadri
Hardware: Tesla T4 (sm_75) + NVIDIA H100 (sm_90)
Language: CUDA C++ (C++17), portable CC 7.0+
Status: Completed — Tested on 2 GPU architectures

Project Summary

Implemented and optimized a CUDA SGEMM kernel (C = A x B, float32) from a naive global-memory baseline to a shared-memory tiled implementation. Performed full correctness audit (13-category checklist), performance audit with roofline analysis, and verified on two GPU architectures — Tesla T4 and NVIDIA H100.

Key results:

  • Tesla T4: Tiled kernel achieved 393.86 GFLOP/s (Google Colab, verified)
  • NVIDIA H100: Tiled kernel achieved 6,813.25 GFLOP/s — ACCEPTED on Tensara.org
  • 17.3x scaling confirmed from T4 to H100 on non-square matrices up to 8192x8192

Problem Statement

Matrix multiplication (GEMM) is the core operation in deep learning training and inference. On GPU hardware, naive implementations leave most of the device's compute and memory bandwidth unused. The goal was to:

  1. Implement a correct naive CUDA kernel (baseline)
  2. Optimize using shared-memory tiling to increase arithmetic intensity
  3. Verify correctness rigorously (not just "output looks right")
  4. Analyze performance using roofline model on real hardware
  5. Explain the gap between theoretical and measured speedup

Implementation

Kernel 1 — Naive (Baseline)

Each thread computes one output element directly from global memory.

Design decisions:

  • size_t for row/col — prevents signed integer overflow at N > 46,341
  • restrict on all pointers — enables compiler aliasing optimization

Kernel 2 — Tiled Shared Memory (Optimized)

Each thread block cooperatively loads 32x32 tiles into shared memory, reusing each element TILE_WIDTH=32 times. Uses 4 independent accumulators to break the FMA dependency chain.

Design decisions:

  • Boundary handling via zero-padding — all threads reach syncthreads() every tile
  • 4 independent accumulators — breaks 32-deep FMA dependency chain
  • pragma unroll — compiler fully unrolls 8 iterations, zero branch overhead
  • Non-square support (M, N, K) — works on rectangular matrices up to 8192x8192

Correctness Verification

Tesla T4 Results (Google Colab)

Size Naive Tiled
512x512 PASS (0 mismatches) PASS (0 mismatches)
1024x1024 PASS (0 mismatches) PASS (0 mismatches)
2048x2048 PASS (0 mismatches) PASS (0 mismatches)

NVIDIA H100 Results (Tensara.org — Real Hardware)

Matrix Shape Status
4096x4096x4096 ACCEPTED
8192x8192x4096 ACCEPTED
4096x4096x8192 ACCEPTED
8192x8192x8192 ACCEPTED

Correctness Audit (13-Category Checklist)

Category Status
OOB global memory access No issue
Races on shared memory No issue
Divergent barrier / deadlock No issue
Uninitialized shared memory No issue
Integer overflow Fixed — size_t promotion applied
Architecture-dependent constructs N/A
Warp primitive masks N/A
Memory ordering / fence scope N/A
Async error surfacing Noted
Cross-kernel seam No issue
Pointer aliasing No issue
Numerical correctness No issue
Latent coupling bugs No issue

No Tier 0 or Tier 1 issues found.

Performance Results

Multi-Architecture Benchmark

Architecture GPU Test Size GFLOP/s Status
sm_75 Tesla T4 N=2048 393.86 PASS
sm_90 NVIDIA H100 4096x4096x4096 6,769.54 ACCEPTED
sm_90 NVIDIA H100 8192x8192x4096 6,822.33 ACCEPTED
sm_90 NVIDIA H100 4096x4096x8192 6,817.78 ACCEPTED
sm_90 NVIDIA H100 8192x8192x8192 6,843.56 ACCEPTED

Average H100: 6,813.25 GFLOP/s (6.8 TFLOPS)
T4 to H100 scaling: 17.3x confirmed

Compiler Statistics (ptxas -v, sm_75)

Kernel Registers/thread Shared Memory Spills
Naive 58 0 bytes 0
Tiled 62 8,192 bytes 0

Performance Analysis — Why 1.17x on T4 Instead of 32x?

Roofline Model (T4)

T4 Ridge Point = 8,100 GFLOP/s / 320 GB/s = 25.3 FLOPs/byte

Naive structural AI: 0.25 FLOPs/byte — BW ceiling 80 GFLOP/s — Measured 332.88 GFLOP/s (4.2x above ceiling) Tiled structural AI: 8.0 FLOPs/byte — BW ceiling 2,560 GFLOP/s — Measured 393.86 GFLOP/s (far below ceiling)

Root Cause 1: Naive Benefits from L2 Cache Reuse

Naive measured 332 GFLOP/s — 4.2x above structural ceiling of 80 GFLOP/s. Warp broadcast pattern on A and T4's 4MB L2 cache absorb most HBM traffic. Effective AI greater than 1.0 FLOPs/byte at HBM — gap that tiled was supposed to close was already 80% closed by hardware cache.

Root Cause 2: Synchronized Barriers Eliminate Latency Hiding

Tiled used only 49.2 GB/s of 320 GB/s peak HBM (15.4%) — latency-bound, not bandwidth-bound. After every syncthreads(), all 32 warps simultaneously stall on HBM loads. Compute phase (32 FMA = 32 cycles) is 6-12x shorter than HBM latency (200-400 cycles).

Why H100 Scales 17x

  • HBM3: 3.35 TB/s vs 320 GB/s (10.5x bandwidth)
  • 132 SMs vs 40 SMs (3.3x more parallelism)
  • 2,048 max threads/SM — allows 2 blocks/SM, partial latency recovery

Future Work

Thread coarsening (2 output elements per thread, 16x32 block):

  • Doubles compute phase to 64 FMA per tile
  • Restores latency hiding via multiple blocks per SM
  • Register impact: NEEDS PROFILING

Key Learnings

  1. Structural roofline does not equal actual behavior — L2 cache closes the gap before shared memory helps
  2. Synchronized barriers destroy latency hiding — all warps stall together
  3. Correctness requires rigor — float32 non-associativity and integer overflow are real risks
  4. Explaining results matters more than numbers — understanding why 1.17x points to thread coarsening
  5. Hardware scaling validates kernel design — 17x T4 to H100 confirms no architecture bottlenecks

Files

File Description
matmul_fixed.cu CUDA source — naive + tiled kernels, host code, timing harness
portfolio_case_study.md This detailed case study

Tech Stack

CUDA C++ · C++17 · nvcc · cudaEvent timing · ptxas -v · Roofline Model · Tesla T4 (sm_75) · NVIDIA H100 (sm_90) · Tensara.org