Tracepoints only triggered by CPU 0

[JoyfulSheep]

Hello,

You will find at the bottom of this post a small code excerpt that seems to have changed its behavior on Ubuntu26.04 (differs from Debian13, Debian14, and Ubuntu24.04). On Debian or the previous Ubuntu distribution, this program logs all open syscalls. But on Ubuntu26.04, only the calls attributed to CPU#0 are logged. This has been tested by having a simple program (also in attachments just in case) opening files in a loop and starting it with taskset :
taskset -c 1,15 ./my_test_program -> no logs in /sys/kernel/tracing/trace_pipe
taskset -c 0 ./my_test_program -> calls are logged as expected

After reading the man page for perf_event_open, I saw that calling it with pid == -1 and cpu >= 0 (like what is done in cilium/ebpf/link/tracepoint) only "measures all processes/threads on the specified CPU" and not all CPUs. Modifying the code inside the cilium/ebpf/link package to loop over all possible CPUs when creating perf events does fix the issue, and every open call on the system is then logged.

My questions being :

• If the behavior observed on Ubuntu26.04 is the one that is expected according to the man page, why does it work on other systems ?
• Should tracepoint linking be updated in the cilium package to loop over CPUs ? And if not, how can I use a tracepoint to monitor events system-wide ?

If I misinterpreted anything, please do correct me.
Thanks a lot

Kernel : 7.0.0-38-generic
Cilium-ebpf : v0.22.0

C code for the ebpf program :

//go:build ignore
#include "vmlinux.h"

#include <limits.h>

#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char __license[] SEC("license") = "GPL";

struct open_args {
    __u8 unused[16];
    const char * filename;
    __u64 flags;
    umode_t mode;
};

SEC("tracepoint/syscalls/sys_enter_open")
int handle_open(struct open_args *ctx)
{
    bpf_printk("%s opened", ctx->filename);
    return 0;
}

Go code for linking :

package main

import (
	"fmt"
	"time"

	"github.com/cilium/ebpf/link"
)

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go@latest -tags linux bpf tracepoint.c -- -I .

func main() {
	objs := bpfObjects{}
	if err := loadBpfObjects(&objs, nil); err != nil {
		fmt.Printf("failed to load BPF objects: %v", err)
	}
	defer objs.Close()
	openTP, err := link.Tracepoint("syscalls", "sys_enter_open", objs.HandleOpen, nil)
	if err != nil {
		fmt.Printf("failed to open tracepoint: %v", err)
	}
	defer openTP.Close()
	for {
		time.Sleep(1 * time.Second)
	}
}

Test program :

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <syscall.h>

void main(void)
{
    int fd;

    printf("%d\n", getpid());
    while (true)
    {
        fd = syscall(SYS_open, "/tmp/test.txt", O_RDONLY);
        sleep(1);
        close(fd);
    }
}

[ti-mo]

@JoyfulSheep This is likely a kernel regression, and a pretty impactful one at that. There's a chance this has already been fixed upstream. Having vendor kernels in the mix when debugging this adds unnecessary noise, please try to diagnose this using upstream kernels. I'd recommend using vimto (see .github/workflows/ci.yml) using cilium/ci-kernels (note ebpf-go has a .vimto.toml that points it at that repo by default) and write a test using runtime.LockOSThread and unix.SchedSetaffinity to reproduce this from within a test. No need to create a subprocess out-of-band, a thread should do.

@mmat11 @florianl MBOI

[florianl]

@JoyfulSheep similar to @ti-mo I think you are running into a Kernel regression, rather than an issue with this package.

If I didn't mix up things, the described issue got fixed upstream with https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2c2b322acdcc78575b8d6afa64a085cf92e03c12.

[JoyfulSheep]

Isn't that the expected behavior though ? From the man page for perf_event_open :

The pid and cpu arguments allow specifying which process and CPU to monitor:
[...]
pid == -1 and cpu >= 0 :
This measures all processes/threads on the specified CPU.

So if link.Tracepoint is only calling perf_event_open once, with cpu == 0 and pid == -1 as arguments, I wouldn't be surprised that only events from CPU 0 are traced. Maybe the regression is in fact a fix ? I might be missing something.

When using strace on a tool like perf with system-wide collection ("-a" flag) like this for example :
strace -e trace=perf_event_open perf stat -e syscalls:sys_enter_open -a ./test
you can see as many calls to perf_event open as CPUs on the system

[mmat11]

I believe the man page refers to perf counters, bpf programs are not CPU filtered (see: iovisor/bcc#1490). I agree it's confusing

[JoyfulSheep]

Thanks for the clarification ! It is indeed a bit confusing. Turns out an issue has been opened on Canonical's side : https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2170225
Thanks for the responses, I will close this discussion.