Wrapping calls in the same compilation unit

[matthijskooijman]

A limitation of the --wrap method, is that it cannot intercept calls to a function inside the same compilation unit as where the original function is defined. This can be solved by moving to-be-intercepted functions into their own .cpp files, but that hurts the readability of the original code, requires changes to the original code when adding wrapped functions in the testsuite, and makes it a bit more fragile to breaking the wrapping when the original code is changed. So I'd like to see this limitation lifted. I've been digging into the ld sources to find a way around this, and this issue presents a couple of options for this. Below is a more generic writeup for handling wrapping, not limited to the PowerFake usage necessarily.

How ld does linking

In a gross oversimplification of the tremendously complex linking process, here's what ld does when it links an executable. I've mostly looked at linking elf object files into elf executables (specifically elf64-x86-64), but I think most of this will be similar for other targets as well (and the elf target is what we're using in almost all cases anyway).

Resolving symbols

• For each object file in turn, each global (exported) symbol in the file's symbol table is considered (elf_x86_64_relocate_section()) and entered into a global (in-memory) symbol table (that starts out empty). Note that the symbol table also contains undefined symbol entries, for symbols that are referenced but not defined.
• This starts by looking up any existing symbol by the same name in the global symbol table (_bfd_elf_merge_symbol()). If the name is not in the global table yet, it is simply added. If it is already there, the existing symbol and the new symbol are merged (_bfd_elf_merge_symbol() and _bfd_generic_link_add_one_symbol()). This merging handles this like a strong symbol replacing a weak symbol (or a weak symbol being discarded when there is already a strong symbol), or a strong symbol replacing a previously undefined symbol, raising an error when trying to merge two strong symbols, etc.
• In any case, the resulting symbol is associated with the symbol table entry in the current file. This resulting symbol may be the same symbol (typical for strong symbols, or weak symbols that are not defined yet), but it can also point to a different symbol (typical for an undefined symbol entry which was already defined in a previous object file).
• For example, consider processing a UNDEF foo() entry first (from a file that references but does not define foo()). This creates a new UNDEF foo() entry in the global symbol table, which is associated with the entry in the current file's symbol table. Then, in another file that actually defines foo(), a DEF foo() entry is processed. This looks in global table, finds the existing UNDEF entry, and merges it with the new DEF entry by overwriting the existing UNDEF entry with the new DEF entry. This also causes the (UNDEF) entry in the first file, to be associated with this new, merged, DEF symbol, so it can be found later.

Resolving relocations

• When you write a function call in the code, the compiler generates a dummy instruction, e.g. call 0x0. It then also leaves an instruction for the linker, saying "Put the address of the global symbol foo at this address (i.e. after the call)". This instruction is called a relocation.
• In the relocation, "the address of the global symbol foo" is not so explicit. Instead, a relocation refers to an entry in the file's symbol table.
• If the compilation unit does not define foo() itself, the symbol table contains an UNDEF foo() entry. After resolving symbols, as described above, the relocation is resolved by looking at the associated file symbol table entry, which has an associated global symbol table entry, which is now a DEF foo() entry that tells the linker where foo() is actually defined.
• If the compilation unit does define foo() itself, the symbol table contains a DEF foo() entry. In most cases, the associated global symbol table entry will be this same DEF foo() entry, so the linker resolves relocations for foo() to the definition in this file itself. There can be exceptions, e.g. when foo() is weakly defined, then the actual foo() to be used by the relocation can still be in a different file.

Implementing --wrap

To implement the --wrap option, the linker changes the second step in the resolving symbols procedure. When it "looks up any existing symbol by the same name in the global symbol table", and the name to look up was passed to --wrap, it actually does a lookup for __wrap_<name> instead. Similarly, when it has to look up __real_<name>, it looks up <name> instead. Simple, but quite powerful.

Except that it can only do this for UNDEF entries in the file's symbol table. Consider what would happen otherwise: There is a DEF foo() entry in one file. The global table lookup uses __wrap_foo instead, and finds an existing entry DEF __wrap_foo(), which is the wrapper to be used. Trying to merge these two entries will fail, since both are strong definitions. If you would instead discard the DEF foo() (as if it were weak) and associate the DEF __wrap_foo() entry with the file symbol table entry, then relocations (calls to foo()) in the same compilation unit would correctly resolve to __wrap_foo(). However, you would have discarded the original foo() entry, so you can no longer access it through __real__foo().

I guess the linker could have also (in addition to looking up __wrap_foo() and associating the result with the current entry) put the original DEF foo() entry in the global symbol table, without associating that entry with any symbol table entry in the current file (but putting it out there to be associated with other entries in other files later), but maybe this didn't seem relevant, or maybe this has a ton of unexpected side effects (the linker is horrendously complex after all, I'm just showing the supersimplified version of it here).

Diverting local calls

This analysis does suggest two possible ways you can still divert a call to a locally defined function to some other function:

• Make the locally defined function weak, so it can be overridden by something else (without using --wrap)
• Make the relocations point to an UNDEF foo() symbol that can be wrapped, and have a second DEF foo() symbol with the actual definition. This is essentially what you do when you move the definition of foo() into its own source file, but I think this could be done inside a single .o file as well.

In addition, Greg Carter shows that you can also use e.g. -Wl,--defsym,foo=__wrap_foo as a linker option to forcibly replace the foo function with the wrapper. I haven't full investigated how this works in the linker internals, but I believe that this approach loses access to the original symbol (unless you duplicate it under another name by modifying the object files, as suggested below), so I haven't investigated this option much further.

How to wrap local calls

So, how can you then actually achieve --wrap for local calls? The above suggests some ingredients, below I'll mix those into a couple of different (but similar) approaches.

A downside of all below approaches is that they require inspecting and modifying the object files in the build. A solution where you would not need to inspect the object files at all (or maybe just the object files that contain the wrappers, to get a list of wrapped functions) and handle everything by just adding compiler and/or linker flags would be ideal, but I haven't been able to figure out a way to allow this.

1. Weakening symbols, without --wrap

• Define the original symbols as weak, or use objcopy --weaken-symbol to do so after compilation (to prevent having to modify the original source files).
• Define the wrapper using the same name as the original, so it will replace the original.
• Do not use --wrap anymore.
• This is essentially what Peter Huewe proposes here, except they also suggest using --globalize-symbol to ensure the symbol is global, but I think that's only needed for non-exported (i.e. globals with the static keyword) symbols, and making those global might end up creating conflicts that were not previously present, so this must be done with care).
• Con: This looses access to the original function, so you cannot really "wrap" an existing symbol, only replace it.
• Con: Must know which symbol is the original and which is the wrapper (to not modify the wrapper).
• Con: If the original function has weak versions, the processed .o files can no longer be used to link the original executable, without adding the wrappers (since now all versions of the original name are weak, so a different version might be chosen than before making the one strong version weak).

2. Weakening symbols and adding a __real_ version, without --wrap

• Like above: Use objcopy --weaken-symbol to mark the original as weak
• Use objcopy --add-symbol to add a new symbol called __real_<name> with the address (i.e. pointing to the same bit of compiled code). This new symbol should ideally be a perfect copy (same section, size, visibility, flags, weakness, etc.), and the copy should be made in all compilation units that have the function (except where the wrapper is defined), so that if the existing symbol already has multiple copies (e.g. a weak and strong version), it will still resolve as without these changes.
• Like above: Define the wrapper using the same name as the original, so it will replace the original.
• Call __real_<name> from the wrapper to access the original.
• This is essentially what Javier Escalada proposes here, except they do not make a perfect copy of the symbol.
• Con: Wrapper is named the same as original, which can be confusing.
• Con: Build process must know where wrappers are defined (to not modify the wrapper).
• Con: Unsure if objcopy can create a proper identical copy, so might require building a custom command or script to parse and modify the elf file.
• Con: If the original function has weak versions, the processed .o files can no longer be used to link the original executable, without adding the wrappers (since now all versions of the original name are weak, so a different version might be chosen than before making the one strong version weak).

3. Splitting symbols, with --wrap

• For each definition of the original symbol, add an identical copy under the same name, and modify the original one to be an UNDEF instead. This result in two symbols by the same name, where relocations point to the UNDEF one and the DEF one points to the implementation, allowing to use --wrap as normal.
• Con: It does not seem objcopy can do this, so this probably requires building a custom command or script to parse and modify the elf file.
• Con: Having a duplicate name in the symbol table of an .o file might confuse tools? The base specification for the ELF format does not discuss uniqueness of names in the symbol table at all, it seems.
• Pro: Wrapper is named differently, making it potentially clearer.
• Pro: Build process can treat all object files equally, since wrappers can be distinguished from the original by their name.

4. Reimplementing --wrap

• For each definition of the original symbol, add an identical copy named __real_<name> and replace the original entry with an UNDEF __wrap_<name> (so that existing relocations now point to the wrapper).
• For undefined reference to the original symbol, also replace that with an UNDEF __wrap_<name> entry.
• Con: It does not seem objcopy can do this, so this probably requires building a custom command or script to parse and modify the elf file.
• Pro: Wrapper is named differently, making it potentially clearer.
• Pro: Build process can treat all object files equally, since wrappers can be distinguished from the original by their name.
• Pro: No duplicate names in the .o file symbol table.
• Pro: No dependency on the linker's --wrap implementation.
• Con: The processed .o files can no longer be used to link the original executable, without adding the wrappers.

I'm inclined to further investigate the last option (if we need to modify .o files with custom tooling anyway, might as well do the entire wrap thing ourselves, which might also simplify things because we no longer need to comply with the linker's requirements on __wrap_ naming). However, the fact that these files are no longer usable as part of a regular build is a bit annoying, and might make option 3 more suitable (if it works, I haven't tried it yet A quick manual edit using https://elfy.io suggests this indeed works). Option 1 is not feasible, since it does not allow calling the original function, and option 2 feels a bit fragile when it comes to existing weak functions.

[hedayat]

Hi Matthijs,
Thank you for looking into this project, and your interest in improving the project.
As you've provided detailed issues with multiple subjects, I should also provide detailed answer to both issues. But it'll take some time for me to answer to all points you've made in these issues.

First of all, about weak symbols: while I didn't experiment right now, but I really doubt that they will ever work for calls in the same translation unit. Actually, anything which would rely on ld will probably not work: because the compiler itself resolves calls in the same TU, not the linker. I'm not sure, but if optimizations are off, it might work, but it is very likely that it won't work when compiler optimizations are on. First because the symbols are not resolved by the linker at all, second because compiler might even make some calls inline.

Although, I might have tried before but I did not experiment with them right now before answering. So, some experiments will tell the correct answer. Another issue with weak symbols is that you cannot call original function. And, they are not good for third party libraries.

And, one of the design goals I had was to not require compiling separately for tests: corelib used in sample/ is expected to the actual library which is linked to the final executable, to be also linked with tests. If you want to recompile everything for tests, you probably would get much better results by using preprocessor tricks to replace calls to original functions with fake functions. But, creating a copy of the library and modifying it and linking with test code might be acceptable; although I've also tried to make dependency on external tools minimal.

So, to me also option 3 seems the best of all, although I'm still unsure if it'll work specially when optimizations are on (till now, I've tried to not force the user to compile with different settings, like not having optimizations. Although, LTO is currently not supported.

Unfortunately, interactions are not as simple as it seems in compiling/linking process. For example, you see that bind_fakes supports two methods of renaming symbols: 1. direct method using objcopy 2. a logical method using --defsym.
And now, the first option is the default; because while I expected the second one to always work, it doesnt! (ae28a5f). So, every solution should be tested extensively with multiple versions of toolchain on at least a few different distros.

(Unfortunately, I've not written when it didn't work :( )

[hedayat]

Just tried this method, and it simply stops working if you add -O2 to compiler cmdline in the build step.

Also, the other answer (option 1) also didn't work with -O2

[matthijskooijman]

As you've provided detailed issues with multiple subjects, I should also provide detailed answer to both issues. But it'll take some time for me to answer to all points you've made in these issues.

Yeah, take your time. For a large part, I'm writing all this down to structure my own thoughts and remember them for later, so don't feel pressured by my tendency to write (too) much :-p

Actually, anything which would rely on ld will probably not work: because the compiler itself resolves calls in the same TU, not the linker.

That's not what I've seen in my tests and review of the LD internals. E.g. consider:

matthijs@grubby:~$ cat foo.c
#include <stdio.h>

void foo(){
        printf("foo\n");
}

int main(){
        foo();
}
matthijs@grubby:~$ gcc -c foo.c
matthijs@grubby:~$ readelf -sr foo.o

Relocation section '.rela.text' at offset 0x298 contains 3 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000b  000500000002 R_X86_64_PC32     0000000000000000 .rodata - 4
000000000010  000c00000004 R_X86_64_PLT32    0000000000000000 puts - 4
000000000025  000a00000004 R_X86_64_PLT32    0000000000000000 foo - 4

Relocation section '.rela.eh_frame' at offset 0x2e0 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000020  000200000002 R_X86_64_PC32     0000000000000000 .text + 0
000000000040  000200000002 R_X86_64_PC32     0000000000000000 .text + 17

Symbol table '.symtab' contains 14 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS foo.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 
     5: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    8 
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT    9 
     9: 0000000000000000     0 SECTION LOCAL  DEFAULT    6 
    10: 0000000000000000    23 FUNC    GLOBAL DEFAULT    1 foo
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _GLOBAL_OFFSET_TABLE_
    12: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND puts
    13: 0000000000000017    25 FUNC    GLOBAL DEFAULT    1 main

Note that even though the call to foo() could be resolved by the compiler already, it is not, but a relocation is added so the call address can be replaced by the linker with the real address later (I believe there's some standards that require this. Also, for architectures that do not have relative calls (if any), this would be the only way to make calls work at all).

This also means that you can make the entry weak after compilation and replace it with some other function.

second because compiler might even make some calls inline.

This is true, you'd probably need to run with -fno-inline to prevent this (though moving functions into their own compilation unit would also prevent them being inlined, assuming no LTO, of course).

Another issue with weak symbols is that you cannot call original function.

This is indeed a problem, that can be worked around by adding another name for the original function. This is what option 2 above suggests.

And, they are not good for third party libraries.

What do you mean by that? That it requires modifying the source code to make declarations weak? Seems this is not strictly required, as suggested in option 1 above, you can make things weak aftewards using objcopy --weaken-symbol.

And, one of the design goals I had was to not require compiling separately for tests: corelib used in sample/ is expected to the actual library which is linked to the final executable, to be also linked with tests.

Yeah, I think this is also a relevant quality (though requiring -fno-inline would already break this goal a little bit, unless you want to forego inlining for all functions, which is probably a bit heavy for production code...).

Just tried this method, and it simply stops working if you add -O2 to compiler cmdline in the build step.

Yup, that causes the call to be inlined, and when that happens, you're done. Using -O2 -fno-inline makes it work again, I expect.

I wonder if maybe LTO could actually be used to our advantage here: If you can postpone all inlining until link time, and inlining only happens after the --wrap is resolved, then it might end up inlining the wrapper as expected. However, LTO-enabled (fat) .o files are a lot more complex, since they contain partly-compiled code to be optimized later, and I'm not quite sure how that interacts with --wrap and how any modifications of the symbol table interact with that...

[hedayat]

Yeah, you are correct: adding -fno-inline made it work. That's an improvement: assuming the most interesting functions to wrap are not inlined anyway, it means that these methods will at least allow you to intercept calls to a number of functions inside a TU; even if -fno-inline is not used.

The downside is that it needs even more work to be done during build process, and also it seems to not be able to replace the --wrap method. At least for third party libraries, which might be even shared ones. I feel --wrap method is much cleaner, so I prefer to keep it for the cases in which it works.

About LTO: unfortunately, the tools are not mature enough and well integrated, at least the last time I checked: the --wrap was not supported at all!

Well, I found related bugs:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88643

https://bugzilla.redhat.com/show_bug.cgi?id=1693831

[hedayat]

All in all, I also think I prefer solution 3 too!

[matthijskooijman]

About LTO: unfortunately, the tools are not mature enough and well integrated, at least the last time I checked: the --wrap was not supported at all!

Ah, that would completely block that path, it doesn't seem like this is easy to fix in gcc / ld either. But not supporting LTO is ok, especially since you can even enable LTO during compilation as long as you use fat objects (containing both partially compiled code for LTO and fully compiled code for normal linking) and then use LTO during the regular link and non-LTO for the test link.

All in all, I also think I prefer solution 3 too!

Cool! I'll probably see if I can hack something up for this, but not right now: I really need to get back to some actual progress on the underlying project, rather than goldplating PowerFake to maybe do unit testing on this project in the future :-p

[hedayat]

Good.

About LTO, I'm not sure if that is possible. Seems that handling even fat LTO files is special. IIRC, I've tried using a fat-LTO main library with non-LTO test executable with no success. But I'm not sure.

[matthijskooijman]

I previously tried applying option 3 manually on Linux, using ELF object files, which seemed to work as expected. Today, I tried the same on MinGW64, which uses COFF object files, but that did not work right away. Maybe some additional changes are needed, or some other approach is needed there. I might dive into the ld sources later to see what it does for COFF (I think MINGW64 just uses gcc / ld compiled for Windows).

What I did to do this manually is to use objcopy.exe --add-symbol foo=.text:0x54,global,function foo.o foo2.o to let objcopy do the heavy lifting of adding an entry to the symbol table (where 0x54 is the value of the original symbol) and then use a hex editor or elfy.io to update the other fields (copy the original foo fields into the new foo, copy the fields from some other undef symbol into the original foo). Note that in COFF, the symbol table starts with the symbol name (if it's short enough) and each entry is 18 bytes long (until the next name starts), so it is easy to spot in the hexdump without having to decode the rest of the file (see https://wiki.osdev.org/COFF#Example).

[matthijskooijman]

Looking more closely at the output of objdump -r foo.o, it seems that the COFF files generated by MINGW64 do not contain any relocation at all for the "internal" call at all, which probably explains why this trick does not work. This does surprise me a bit, since that would prevent weak symbols from working as well, but it seems that weak symbols are a bit weird in COFF anyway, and actually broken in MINGW64 currently anyway (so that also rules out the other options on MINGW).

[hedayat]

Thanks for the research! It'd be great if we can solve it for mingw too, but if it becomes a linux only feature it'll still be great.

[hedayat]

Well, I'm about to merge a new macro: HIDE_FUNCTION.

It is a very lightweight macro: at the end, it uses a link script which works similar to usign ld's --defsym: so it hides the original function and doesn't provide any means to call the original function.

Additionally, as you mentioned already, it doesn't work with mingw.

Also, in addition to function inlining, -foptimize-sibling-calls optimization (enabled with -O2 and above) should be also disabled otherwise function calls which happen at the end of another function will be optimized and will call the original function.

Despite all these limitations, if all those are acceptable, it lets you to capture calls inside the same TU.

It is far from ideal, but still simple enough and doesn't need a special tool to manipulate object files. Although I'll keep looking for something better.

Update: Merged with 96c7da9

[jenuc78]

Have you tried the weak attribute? In this way you mock a function without moving it to other file

[hedayat]

Have you tried the weak attribute? In this way you mock a function without moving it to other file

To be honest, I don't remember now. However, I wonder if it provides any benefits over the current HIDE_FUNCTION method, which also works for functions in the same file. And it is non-intrusive, while using weak attribute is an intrusive methods (although, it can still be done by creating a copy and modifying using objcopy, if it provides features which are not currently possible with HIDE_FUNCTION).

[jenuc78]

weak attribute is standard gcc, similar to ld wrap, is possible to create a macro and build twice, with and without weak attribute, when weak is enabled you can mock the original function, when weak is disabled you can call the real function

[hedayat]

Well, HIDE_FUNCTION is a helper macro too, and it uses standard ld script command EXTERN. I should conduct some tests to see if weak symbols has any advantages (e.g. work in more scenarios) than the LD script. If not, I prefer to stick with the latter which seems a more cleaner approach.

[jenuc78]

I don't think I understand very well how EXTERN/undefined ld option works in HIDE_FUNCTION. According to this link seems to work only for libraries and not for functions from same object files
https://stackoverflow.com/questions/42588983/what-does-the-gnu-ld-undefined-option-do

[hedayat]

Well, no. WRAP_FUNCTION is the main helper macro in PowerFake, which provides a flexible base for mocking calls to other object files/libraries. The HIDE_FUNCTION macro is specifically provided to enable capturing calls in the same object file, as noted in README:

Using HIDE_FUNCTION(), you'll lose access to the original function, but you can also capture calls in the same translation unit.

However, unlike WRAP_FUNCTION, you lose access to the original function.

The way it works is that it marks the symbol as undefined, forcing the linker to look for the symbol in other object files, so that it links our fake function instead of the original.

[jenuc78]

So if you want to mock a function (from same unit) but also call the original you need to have two builds, right?

[hedayat]

The flag only affects the test executable; so in the main binary you are fine. But in tests, yeah if you need some tests to be able to call the original function, yeah you'd need a separate test binary. (the original object file doesn't need to built twice, but you need two links: a test binary linked to the faked function, another one to link to the original function.

AFAIK, you need to do the same with weak symbols too.

[wafgo]

I wrote a tool because I was observing the same issue, that with --wrap you cannot wrap symbols in the same compilation unit.
For this I wrote a simple python script which is fixing it for me. This script is modifying the relocatable object files by adding/renaming symbols and switching symbol references in the relocation entries (utilizing the LIEF python module). Maybe this will also solve it for some of you:

Here the link:
https://github.com/wafgo/WrapMaster/tree/master

PS: Only ELF is supported and it is not very extensively tested yet, so problems may occur!

[hedayat]

@wafgo

Thanks for the link! I've already provided HIDE_FUNCTION() facility, but it makes the original function inaccessible. But as far as I can see, that your solution doesn't have this limitation.

Although, I've tried to not get involved in very low level modifications, it can be provided as an option.

[jenuc78]

I wrote a tool because I was observing the same issue, that with --wrap you cannot wrap symbols in the same compilation unit. For this I wrote a simple python script which is fixing it for me. This script is modifying the relocatable object files by adding/renaming symbols and switching symbol references in the relocation entries (utilizing the LIEF python module). Maybe this will also solve it for some of you:

Here the link: https://github.com/wafgo/WrapMaster/tree/master

PS: Only ELF is supported and it is not very extensively tested yet, so problems may occur!

would it be possible to modify the script so that functions from a CU be wrapped and be possible to call them from outside CU file?
this would be useful to test any parent function by calling it with _real prefix assuming that second functions level (child functions) are wrapped

[jenuc78]

starting from @wafgo python LIEF interface I have copied functions (only those for which I have created a wrapper) from compilation units to _orig, then I made the functions undefined in ELF object. In this way seems to work also with LD wrap. Wrap functions are still accessed via _wrap prefix but original functions are accessed via _orig prefix. I hope I did not missed an important detail :)

[hedayat]

Thanks for the report! I'll try to have a look to both WrapMaster and LIEF itself, and may provide using it as an option (might port WrapMaster functionality to C++ and using LIEF directly).