Possible problem with experimental USDT / DTrace probes

[aapoalas]

Hello

It seems to me like the USDTs that you're generating might be turning up unsafe/unsound to trace in some circumstances. I just spent one and a half days wrestling with this in another project (and using a different crate than probe-rs to create the USDTs), and I noticed opendal's documentation shows similar (possible?) issues and figured I'd come in for a checkup.

Basically, the output readelf output given in the DtraceLayer's documentation looks like this:

stapsdt              0x00000039       NT_STAPSDT (SystemTap probe descriptors)
  Provider: opendal
  Name: create_dir_start
  Location: 0x00000000000f8f05, Base: 0x0000000000000000, Semaphore: 0x00000000003649f8
  Arguments: -8@%rax

The "Address" is the address of the nop instruction (the probe), and "Semaphore" is the address of the 16 byte value used to guard entry to the probe. But what about the "Base"? That is supposed to be the address of the _.stapsdt.base section, which is used "to detect prelink address adjustments" (see prelink address adjustments if you want to understand this fully; I do not): it being zero means that someone (the linker, I believe) decided that the section is unused, removed it, and replaced references to it with a zero. This then means that at tracing time prelink address adjustments cannot be detected and if such things have happened, then the probe and/or semaphore addresses will remain unadjusted. This (presumably; I'm not 100% sure on this) means that instead of changing the nop to an interrupt, some other location in the loaded executable gets turned into an interrupt unexpectedly. Similarly, I believe with more conviction, the semaphore increment and decrement will write to some other part of the program.

I was having crazy kind of segmentation faults in my own project during tracing, and at the end of the day I noticed that my linker (mold) was GC'ing the base section away and leaving me without base addresses. Changing to the standard linker left the base section in the executable, retaining my base addresses, and fixing the segmentation faults.

Now: even if you have these issues in opendal's DtraceLayer, it's not a bug in the library. You cannot reasonably do much about this (unless you want to introduce a spurious read of _.stapsdt.base or otherwise usage of its address into the code somewhere), but I figured it might be worth a warning since your documentation does show this pattern. At most you'll possibly want to check your readelf output and linker usage, change the documentation, and possibly warn users that they should make sure that their executables have base addresses set.

[yihong0618]

cc @Zheaoli you may have interest about this

[Xuanwo]

Wow, thanks a lot for raising this. I will take a look along with @Zheaoli who is the code owner of this layer.

[aapoalas]

Ah, there actually may also be something that you could do (or probe-rs) could do; currently in talks with mold linker developer on how to properly do this: rui314/mold#1510