profiling.md

Profiling

Native (C/C++) and Python (relevant for me now, maybe extend later) (native maybe enough initially, Python support can maybe be added).

Also see debugging.

Most importantly Linux. Probably this is platform dependend. MacOSX has DTrace. (Windows has UIforETW. See the randomascii blog.) Linux also has eBPF now.

Common situations:

• Some process behaves strange, is much slower than usual, or totally hangs. I want to see, where does it hang.

Features I want:

• Be able to attach live to running process, without preparation.

• Multi-threading support. Although in many cases I'm mostly interested in the main thread (or one specific thread). Some more clever handling would be nice (although not sure how easy), e.g. if that (main) thread is hanging at a mutex, I would want to see which thread has the mutex currently, and what that thread is doing. But this can maybe also be configurable and simple, such that it just prints the threads which names match to some regexp.

Existing software:

• perf (C). Run perf record -p pid and then perf report --stdio. Maybe perf record -g dwarf -p pid.
• gperftools (C). Google.
• gperf (C). GNU. Needs to be compiled with such support.
• gdb -p pid (C). No real profiling, but helps at least to debug a permanent hang.
• Sysprof (C). System-wide, i.e. not quite what I want here.
• IgProf (C)
• DTrace (C).
• eBPF (C, native), like DTrace, since Linux 4.9.
• ftrace (C)
• LTTng (C)
• magic-trace (C)
• Pyflame (Python). deprecated.
• Py-Spy (Python). CPU. Fast.
• PyStack (Python). show stack, even including locals.
• austin (Python). CPU+memory. Fast.
• memray (Python). Memory. HN.
• Scalene (Python). CPU+GPU+memory. Fast.
• Pyinstrument (Python). CPU. Semi-fast (30% overhead).
• Tracing UI: Perfetto with Chromium Event JSON Format, Fuchsia Trace Format, Perfetto Protobuf (details)
• Tracing/profiling UI: Speedscope (GitHub)
• GDB scripting: GEF
• Tracing UI: magic-trace (GitHub, blog post)
• Tracy (C)
• optick (C)
• 0x.tools using eBPF (GitHub, HN)
• otel-profiling-agent using eBPF, support for many languages including Python
• Grafana Beyla eBPF-based, supports Python

Some useful StackOverflow questions:

• How can I profile C++ code running on Linux?
• How can I get perf to find symbols in my program
• Can I get the python call stack with the linux perf?

Other refs:

• Tristan Hume - All my favorite tracing tools: eBPF, QEMU, Perfetto, new ones I built and more