Skip to content

Troubleshooting

FindHao edited this page Aug 31, 2025 · 5 revisions

No CSV/log produced 🚫

  • Ensure CUDA_INJECTION64_PATH points to lib/cutracer.so
  • Set KERNEL_FILTERS to match the actual kernel name (mangled or unmangled)
  • Verify working directory has write permission

Histogram empty 📉

  • Ensure the kernel contains clock reads (e.g., Triton pl.scope adds them)
  • Check that CUTRACER_ANALYSIS=proton_instr_histogram is set

High overhead 🐢

  • Use opcode_only only; avoid reg_trace/mem_trace unless required
  • Narrow KERNEL_FILTERS and use INSTR_BEGIN/INSTR_END

CUDA graph / stream capture behavior 🕸️

  • Tool handles these paths; if missing outputs, validate stream capture status and synchronization
  • For captured graphs, data is flushed at cuGraphLaunch exit; ensure proper stream sync.

Version issues 🧩

  • nvcc/ptxas minimums are enforced by Makefile; check errors and adjust ARCH

IPC merge issues 🔗

  • Warp ID mismatch: ensure both runs target the same kernel launch; avoid filtering one side only.
  • Missing ipc values: cycles or instruction counts missing/zero; re-check both inputs.
  • Kernel hash ambiguity: pass --kernel-hash explicitly to the parser.

Clone this wiki locally