Skip to content

Latest commit

 

History

History
88 lines (62 loc) · 3.27 KB

report-template.md

File metadata and controls

88 lines (62 loc) · 3.27 KB

AI Performance Engineering - Capstone Report

Fill this in with your own measured numbers from M0–M6. The point isn't the numbers themselves - it's that you can explain them with the roofline. This is the artifact you put on your CV / share.

Author: your name Date: YYYY-MM-DD Repo/notebook links: ...


1. The machines

CPU (M0-M4) GPU (M5-M6)
Model e.g. Xeon Gold 5220R (Cascade Lake) e.g. NVIDIA T4 (Colab)
Cores / SMs 12 physical 40 SMs
ISA / Features AVX2, AVX-512, bf16? CUDA cap, tensor cores
RAM / VRAM 39 GB 15 GB
How obtained native / droplet / Docker Colab T4

(paste the sysinfo / cell-1 output for each)

2. Roofline anchors (the ceilings)

Anchor CPU GPU
Peak compute fp32 (GFLOP/s)
Peak compute bf16 (GFLOP/s) tf32/fp16/bf16 →
Peak memory bandwidth (GB/s)
Ridge point (FLOP/byte) per precision

One-line takeaway: e.g. "GPU compute is ~30× the CPU, but memory bandwidth only ~20×, so memory-bound ops close the gap."

3. Where real ops land (M1 / M5)

Op CPU GFLOP/s GPU GFLOP/s Bound by Why
matmul_1024_4096 compute high arithmetic intensity
conv2d 3×3 compute
attention (SDPA) compute
maxpool / upsample memory AI ≈ 0.2 FLOP/byte

4. The optimization ladder (M2) – matmul, by hand

Variant GFLOP/s Speedup vs naive What changed
naive 1×
tiled (cache blocking) reuse data in cache
tiled + OpenMP all cores
oneDNN benchmark (M1.5) + SIMD + expert tuning (library ceiling)

Gap analysis: why is your hand kernel still Nx below the library? (SIMD width, blocking, packing, …) ISA note (M1.5): AVX-512 vs ONEDNN_MAX_CPU_ISA=avx2 – which was faster, and did you see throttling?

5. Precision (M3 / M6)

Precision CPU GFLOP/s GPU GFLOP/s Accuracy cost
fp32 reference
bf16 / fp16 relative error
int8 -- relative error

Takeaway: when did low precision help, when did it hurt, and why (hardware support)?

6. Custom kernel (M6, GPU)

  • Fused op: relu(x+y) / SiLU / … Correct vs PyTorch: yes/no.
  • eager: … ms – Triton: … ms (…×) – torch.compile: … ms (…×).
  • Insight: why the fused kernel wins (memory passes), and whether hand-tuning beat torch.compile.

7. LLM inference (M4 CPU / M6 GPU)

CPU (llama.cpp) GPU (transformers)
Model Qwen2.5 Qwen2.5-0.5B
tokens/s @ batch 1
tokens/s @ best batch (thread sweep) (batch sweep)
Quantization used Q4_K fp16 / 8-bit

Throughput vs latency: where did batching/threads stop helping, and why?

8. Conclusions

  • 3–5 bullets: the biggest surprises, the clearest roofline lesson, and one thing you'd optimize next.
  • Optional: a roofline plot (compute ceiling, BW ceiling, your ops as points) for CPU and GPU.

Method note: GPU numbers on a shared Colab T4 are indicative (throttled/shared). For publishable figures, re-run on a dedicated GPU and report variance across 3+ runs.