-
-
Notifications
You must be signed in to change notification settings - Fork 14.4k
link modifier export-symbols: export all global symbols from selected uptream c static libraries
#150992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| let symbol = symbol.unwrap(); | ||
| out.push(( | ||
| String::from_utf8(symbol.name().to_vec()).unwrap(), | ||
| SymbolExportKind::Text, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do need to distinguish between text and data symbols. Also on Windows I believe and on macOS I know we currently use the pre-C mangling names in exported_symbols. I think the post mangling name is fine here too for Windows, but you did need to check. On macOS however it is definitively not ok. The symbol names in the object file are prefixed with _, but the linker doesn't accept this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After testing, the symbol does have a prefix _ on macos, but not on windows. I have removed the prefix symbol (I don’t know why it was added back in the export_symbols function?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
64-bit Windows symbols do not use special mangling, but 32-bit symbols are prefixed with an additional leading underscore.
So:
int foo() { return 42; }Would be foo symbol on 64-bit Windows, and _foo on 32-bit Windows.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And extern "stdcall" has extra mangling, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not only that, all calling conventions have their own mangling: https://learn.microsoft.com/en-us/cpp/error-messages/tool-errors/name-decoration?view=msvc-170
I don't remember whether linkers can export symbols using .DEF files if the symbol doesn't exactly match calling convention. IIRC, GNU ld and LLD (not so sure about this one) can handle it, at least in some cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OMG. This is an area I'm unfamiliar with, and I don't have a 32-bit Windows device to test on. I'm unsure how to handle these symbols in 32-bit Windows; perhaps the linker will handle them correctly? Because I can't seem to find any special code in rustc that handles these symbols.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can test it on 64-bit Windows or any other system by cross-compiling to a i686 Windows target.
Anyway, once you add the test to this repo CI will be able to test it on all systems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left a fixme, and I would like to provide the handling of 32-bit Windows as a future PR, is that ok?
|
I think this is a reasonable approach. This will need a test though. |
I've add a simple test. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
0d2e81c to
f4c91ab
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
def5d06 to
719e1b6
Compare
This comment has been minimized.
This comment has been minimized.
…jorn3,petrochenkov link modifier `export-symbols`: export all global symbols from selected uptream c static libraries In order to be able to export symbols from a specified upstream C static library, I redesigned a solution that, compared to a previous PR rust-lang#150335 I submitted, will not have any extra symbols leaking out. The following points should be noted: - This attribute will select and import the `Global` symbols of the first matching library it finds. - Developers should ensure that there are no libraries with the same name. - This modifier is only compatible with `static` linking kind - By default, upstream C static libraries will not export their `Global` symbols regardless of whether `LTO` optimization is enabled. However, after enabling this attribute, if the upstream C static library has `LTO` optimization enabled, the compiler will issue an error to inform the developer that the linked C library is invalid. The test code is the same as the PR rust-lang#150335. Here are the results: 1. `cargo +include-libs rustc --release -- -L. -lstatic:+export-symbols=c_add` (or you can use `#[link(name = "c_add", kind= "static", modifier = "+export-symbols")]` in the file) ```bash U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 0000000000014f60 T c_add U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 0000000000014f70 T c_sub w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000014ee0 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_DeleteException@GCC_3.0 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5 ``` 3. `cargo +nightly rustc --release -- -L ./` ```bash U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000011e10 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w gettid@GLIBC_2.30 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5 ``` r? @bjorn3
This comment has been minimized.
This comment has been minimized.
…chenkov link modifier `export-symbols`: export all global symbols from selected uptream c static libraries In order to be able to export symbols from a specified upstream C static library, I redesigned a solution that, compared to a previous PR #150335 I submitted, will not have any extra symbols leaking out. The following points should be noted: - This attribute will select and import the `Global` symbols of the first matching library it finds. - Developers should ensure that there are no libraries with the same name. - This modifier is only compatible with `static` linking kind - By default, upstream C static libraries will not export their `Global` symbols regardless of whether `LTO` optimization is enabled. However, after enabling this attribute, if the upstream C static library has `LTO` optimization enabled, the compiler will issue an error to inform the developer that the linked C library is invalid. The test code is the same as the PR #150335. Here are the results: 1. `cargo +include-libs rustc --release -- -L. -lstatic:+export-symbols=c_add` (or you can use `#[link(name = "c_add", kind= "static", modifier = "+export-symbols")]` in the file) ```bash U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 0000000000014f60 T c_add U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 0000000000014f70 T c_sub w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000014ee0 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_DeleteException@GCC_3.0 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5 ``` 3. `cargo +nightly rustc --release -- -L ./` ```bash U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000011e10 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w gettid@GLIBC_2.30 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5 ``` r? @bjorn3
This comment has been minimized.
This comment has been minimized.
|
💔 Test for d754d86 failed: CI. Failed job:
|
|
@bors r- |
|
@ZuseZ4 It seems that symbol export on macOS isn't so straightforward... I'm currently planning to limit it to Linux only, is that okay? |
50e4c55 to
323fd8e
Compare
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
|
Unknown argument "aarch64-apple". Did you mean to use |
…ed uptream c static libraries
|
@bors try jobs=aarch64-apple |
This comment has been minimized.
This comment has been minimized.
link modifier `export-symbols`: export all global symbols from selected uptream c static libraries try-job: aarch64-apple
This comment has been minimized.
This comment has been minimized.
|
@bors try cancel |
|
Try build cancelled. Cancelled workflows: |
323fd8e to
7c2000e
Compare
|
@bors try jobs=aarch64-apple |
This comment has been minimized.
This comment has been minimized.
link modifier `export-symbols`: export all global symbols from selected uptream c static libraries try-job: aarch64-apple
I think this already has some other limitations (export from dylib) due to which we can't use it for autodiff yet, so happy to land it in a limited form so we can push a follow-up in another PR. But let me ping @bjorn3 who revied it, maybe he knows a fix for it. |
Of course. However, we discussed this in the unsolved problem above. Previously, I simply tried removing the prefix, but it didn't pass the test. |
In order to be able to export symbols from a specified upstream C static library, I redesigned a solution that, compared to a previous PR #150335 I submitted, will not have any extra symbols leaking out.
The following points should be noted:
Globalsymbols of the first matching library it finds.staticlinking kindGlobalsymbols regardless of whetherLTOoptimization is enabled. However, after enabling this attribute, if the upstream C static library hasLTOoptimization enabled, the compiler will issue an error to inform the developer that the linked C library is invalid.The test code is the same as the PR #150335.
Here are the results:
cargo +include-libs rustc --release -- -L. -lstatic:+export-symbols=c_add(or you can use
#[link(name = "c_add", kind= "static", modifier = "+export-symbols")]in the file)U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 0000000000014f60 T c_add U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 0000000000014f70 T c_sub w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000014ee0 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_DeleteException@GCC_3.0 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5cargo +nightly rustc --release -- -L ./U abort@GLIBC_2.2.5 U bcmp@GLIBC_2.2.5 U calloc@GLIBC_2.2.5 U close@GLIBC_2.2.5 w __cxa_finalize@GLIBC_2.2.5 w __cxa_thread_atexit_impl@GLIBC_2.18 U dl_iterate_phdr@GLIBC_2.2.5 0000000000011e10 T downstream_add U __errno_location@GLIBC_2.2.5 U free@GLIBC_2.2.5 U fstat64@GLIBC_2.33 U getcwd@GLIBC_2.2.5 U getenv@GLIBC_2.2.5 w gettid@GLIBC_2.30 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U lseek64@GLIBC_2.2.5 U malloc@GLIBC_2.2.5 U memcpy@GLIBC_2.14 U memmove@GLIBC_2.2.5 U memset@GLIBC_2.2.5 U mmap64@GLIBC_2.2.5 U munmap@GLIBC_2.2.5 U open64@GLIBC_2.2.5 U posix_memalign@GLIBC_2.2.5 U pthread_key_create@GLIBC_2.34 U pthread_key_delete@GLIBC_2.34 U pthread_setspecific@GLIBC_2.34 U read@GLIBC_2.2.5 U readlink@GLIBC_2.2.5 U realloc@GLIBC_2.2.5 U realpath@GLIBC_2.3 U stat64@GLIBC_2.33 w statx@GLIBC_2.28 U strlen@GLIBC_2.2.5 U syscall@GLIBC_2.2.5 U __tls_get_addr@GLIBC_2.3 U _Unwind_Backtrace@GCC_3.3 U _Unwind_GetDataRelBase@GCC_3.0 U _Unwind_GetIP@GCC_3.0 U _Unwind_GetIPInfo@GCC_4.2.0 U _Unwind_GetLanguageSpecificData@GCC_3.0 U _Unwind_GetRegionStart@GCC_3.0 U _Unwind_GetTextRelBase@GCC_3.0 U _Unwind_RaiseException@GCC_3.0 U _Unwind_Resume@GCC_3.0 U _Unwind_SetGR@GCC_3.0 U _Unwind_SetIP@GCC_3.0 U write@GLIBC_2.2.5 U writev@GLIBC_2.2.5r? @bjorn3