Linking side module with libc++ litters import export sections

[davits]

Our wasm module size is an issue for us and we are trying to break it down into smaller modules. We are trying to move forward with -sMAIN_MODULE=2 and -sSIDE_MODULE=2, for a tighter control and also because MAIN_MODULE=1 bloats the main module significantly.

Our side module has C++ standard library usages, which by default remain unresolved in side module and should be resolved by exports provided by main module. Since we link main module with mode 2, it is unrealistic to specify all necessary standard library functions one by one in the main module's exports, and undesirable. So we decided to create self sufficient side module(s) instead by linking libc++ with side modules.
The problem is that when we link with "-sSIDE_MODULE=2 -lc++" flags the same symbols which were unresolved are still appear in the imports section, and additionaly those appear in the exports section as well.

Beside being seemingly wrong it also bloats the executable (output from bloaty):

   FILE SIZE        VM SIZE
 --------------  --------------
  32.8%   226Ki   NAN%       0    Code
  31.3%   216Ki   NAN%       0    Import
  28.5%   196Ki   NAN%       0    Export
   6.6%  45.6Ki   NAN%       0    Data

It seems that emscripten treats libc++ symbols as special, and I am not sure if this is a bug or a feature :)
If it is a bug, and someone points to me where the boilerplate is, I am willing to work on it.

Example:

#include <emscripten.h>
#include <string>
EMSCRIPTEN_KEEPALIVE void foo(std::string str) {
  str += " from foo";
  printf("%s\n", str.c_str());
}

em++ example.cpp -sSIDE_MODULE=2 -lc++ -fwasm-exceptions -o example.wasm

Function "_ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKc" appears in both sections along with many others

[sbc100]

I don't believe its possible today to link libc++ into the side module like that. At least we for sure don't have any tests for that.

The easiest way to using -sMAIN_MODULE=2 and have it include exactly the symbols you need is to pass the side module as an argument when linking the main module. You might also want to combine that with -sAUTOLOAD_DYLIBS if you plan on loading at runtime using dlopen.

When you see symbols that are both in impots and exports section that normally means those symbols are weak, and I think C++ and templates make a lot of usage of weak symbols. Even vtables themselves I believe are defined as weak, and end up coming from whichever shared library is loaded first.

[sbc100]

Same with any header-defined inline function in a class. I believe they will all end up as weak symbols, which have this "import-and-export-at-the-same-time" characteristic in the final shared library.

What this is saying to the dynamic linker is "I need this symbol, and if nobody else has already defined it, here is a definition you can use"

[davits]

Thank you for the reply.

I've managed to get a small working dlopen example by passing "-sAUTOLOAD_DYLIBS=0 side.wasm" to the main module link step. The side module still has same functions in both import and export sections, so there still will be some bloat in the real project, which I will measure later.

Yes you are right, I did some tests and the unifying property of the symbols which end up in both import and export sections is that they are weak symbols. Vtables, templates, inlines have weak linkage, because they can and will be repeated across translation units, and unlike strong symbols in case of weak symbols linker just merges those, instead of screaming ODR violation.
However in case of runtime loading both strong and weak symbols can be present across multiple libraries (I am speaking in the context of Linux). And your last sentence is true for the strong symbols as well.

So for me at least it doesn't make sense that strong symbols honor SIDE_MODULE=2, but weak symbols do not. If to draw parallels, in my mind the SIDE_MODULE=2 is equialent to the '-fvisibilty=hidden', and visibility is orthogonal to the symbol linkage.

It looks like wasm(import + export) is equivalent to the dynamic symbol table entry with GOT+PLT, but import/export separation allows more nuanced fine tuning.

[sbc100]

I see, yes if -sSIDE_MODULE=2 does indeed impl -fvisibilty=hidden then you are correct and symbls should be internalized and then at that point loose their weak bindings.

I don't remember right now if that is true through.. I would need to check.

It could lead to some rather strange behaviour if you do this though, because it would mean you could end up with N copies of these weak / comdat groups, for each side module and one for the main program. The would certain, for example, break RTTI, but can also lead to all kind of ODR violations at runtime.

[davits]

Played with SIDE_MODULE=2 bit more. Found that global symbols are import/exported as well.
This behavior plus exporting weak symbols is not reflected in the documentation:

-sMAIN_MODULE=2 ... It is then your responsibility to make sure that code that side modules need is kept alive. ... There is also the corresponding -sSIDE_MODULE=2 for side modules.

My interpretation was that -sSIDE_MODULE=2 is a full manual mode as well.
I think regardless of the conclusion here the documentation needs to be updated.

There are problems on the other side of the fence as well. For example if you have a global variable which obviously has strong symbol and it is exported in side modules. Now assume you have somehow incorporated same variable into N side modules, in the end result there will be only one corresponding symbol from the first loaded side module, but there will be N constructor and N destructor calls for the same address. I met this scenario in real life, where second constructor call from a library was cleaning a constructed global state.

In overall the multi module development is littered with rakes and pitfalls, and I have fallen in my fair share of them :) to be dumbly confident that I can pull through. Given the right tools... There is -fvisibility, attribute((visibility(...))) and --version-script on linux side, and it would be nice to have the same/similar tools on the emscripten side.

Fun fact: I was trying to recreate dynamic_cast failure over the library boundary on Linux. It wasn't failing even with everything hidden from dynamic symbol table both from executable and shared library. It started failing as soon as I switched from default libstdc++ to libc++. I think that guys over the gcc side got fed up and just changed dynamic cast implementation to compare type_info(s) (effectively type names), rather then type_info pointers.
In the past I was able to solve dynamic_cast failure over the library boundary for llvm-17 by adding -fno-assume-unique-vtables flag, which also resorted to type_info comparison, but it seems that it was changed in consequent implementations.
The RTTI can be solved by implementing source of truth in the main module, delegating creation and casting to it, it is cumbersome but doable.

If -sSIDE_MODULE=2 behavior can't change due to backward compatibility. Can I assume the project would be open to new fully manual mode -sSIDE_MODULE=3, and a possibility for EXPORTED_FUNCTIONS to support simple glob pattern regexes, like --version-script does.

[sbc100]

I don't think its too late to change the -sSIDE_MODULE=2 behaviour.

Can you confirm I understand the problem correctly: You expect -sSIDE_MODULE=2 to internalize/hide all symbols by default, even those build with visibility=default, but currently all visibility=default symbols do seem to be exported. You would rather then only symbols explictly listed in -sEXPORTED_FUNCTIONS or EMSCRIPTEN_KEEPALIVE be exported?