sched_process_free tracepoint format has changed starting on 6.16 kernel version #737
Description
How to reproduce the issue?
Run the ebpf-profiler in a Linux kernel 6.16+:
[root] $ ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls
INFO[0000] Starting OTEL profiling agent v0.0.0 (revision main-c4fdb9aa, build timestamp 1756239764)
INFO[0000] Interpreter tracers: perl,php,python,hotspot,ruby,v8,dotnet,go,labels
INFO[0001] Found offsets: task stack 0x20, pt_regs 0x3f48, tpbase 0x1678
INFO[0001] Supports generic eBPF map batch operations
INFO[0001] Supports LPM trie eBPF map batch operations
INFO[0001] eBPF tracer loaded
INFO[0001] Attached tracer program
ERRO[0001] Failed to start agent controller: failed to attach scheduler monitor: failed to configure tracepoint on tracer.hookPoint{group:"sched", name:"sched_process_free"}: cannot create bpf perf link: permission denied
Root cause
The format for the sched_process_free tracepoint changed in 6.16:
$ sudo cat /sys/kernel/debug/tracing/events/sched/sched_process_free/format
name: sched_process_free
ID: 306
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:__data_loc char[] comm; offset:8; size:4; signed:0;
field:pid_t pid; offset:12; size:4; signed:1;
field:int prio; offset:16; size:4; signed:1;
print fmt: "comm=%s pid=%d prio=%d", __get_str(comm), REC->pid, REC->prio
The pid offset moved from beign 24 to 12, that is because the task command field being a pointer instead of fixed length array:
- https://elixir.bootlin.com/linux/v6.15.11/source/include/trace/events/sched.h#L306
- https://elixir.bootlin.com/linux/v6.16/source/include/trace/events/sched.h#L306
Proposed fix
The following patch fixes the issue on the corresponding kernel, the final fix should be backwards compatible (CO-RE?).
diff --git i/support/ebpf/sched_monitor.ebpf.c w/support/ebpf/sched_monitor.ebpf.c
index 63164b1..0041f20 100644
--- i/support/ebpf/sched_monitor.ebpf.c
+++ w/support/ebpf/sched_monitor.ebpf.c
@@ -9,7 +9,7 @@
// See /sys/kernel/debug/tracing/events/sched/sched_process_free/format
// for struct layout.
struct sched_process_free_ctx {
- unsigned char skip[24];
+ unsigned char skip[12];
pid_t pid;
int prio;
};