Export LLVM symbols for autodiff on Apple

[sgasho]

Export LLVM symbols from rustc_driver on Apple when autodiff is enabled.

On Apple targets, Enzyme can hang if rustc and Enzyme each embed their own LLVM. Exporting LLVM symbols from rustc_driver lets Enzyme reuse

dependencies: rust-lang/enzyme#31
This PR can be merged after rust-lang/enzyme#31 has been merged and enzyme submodule has been updated.

I applied diffs on this PR to #151063 then produced artifacts at https://github.com/rust-lang/rust/actions/runs/21091289355

I confirmed that /path/to/rustc -Zautodiff=Enable -Clto=fat test.rs successfully finished when I used those artifacts↑

r? @ZuseZ4

[bjorn3]

This seems like it has the same goal as #150992 except implemented in a hackier way.

[sgasho]

Thanks for informing!
I took a look at #150992 and indeed that approach would be better.
I think I can wait until the PR to be merged

[sgasho]

After #150992 has been merged, I'll modify this PR or create a new PR to export LLVM symobls for autodiff on Apple target

I leave this PR as a draft for now

[ZuseZ4]

Also @Kobzol , just for awareness.
As a trade-off for using libloading we'll need to make llvm symbols visible on macos, since we can't have two llvm copies embedded into rustc and enzyme. Making the symbols public might have a perf impact.

[Kobzol]

Is there any other potential perf. impact other than making the toolchain larger?

[ZuseZ4]

I don't expect it to have any relevant runtime perf impact, but given the surface of LLVM and the complexity of our bootstrap (e.g. PGO) I also wouldn't be too surprised if there's a corner case where this affects a benchmark. On Linux I would have started a perf run, but since we don't have those for apple I'd just merge it when ready. I mainly wanted to make sure that you're aware and ok with this change.

[Kobzol]

Ok. Go ahead :)

[ZuseZ4]

Ok, now that the Enzyme update landed, can you create some new linux+apple artifacts based on the latest EnzymeAD/Enzyme + #150992 (as it seems almost ready to land)?
Once we verified that those still work, I'll update the submodule once more and we should be good to land the rest.

[bjorn3]

Why do we actually statically link LLVM on macOS in the first place? I know we do it on Windows to work around the limit around how much symbols can be exported from a single dll, but that is not an issue on macOS.

[ZuseZ4]

@Kobzol might know

[Kobzol]

Nope, no idea 😆

[nikic]

I think we currently just always use static linking, except the cases where we went out of the way to use dynamic linking for specific reasons (e.g. x86_64-unknown-linux-gnu so we can built libLLVM with LTO, PGO etc). It would probably make sense to invert this and always use dynamic linking unless there is a reason not to (Windows).

Of course, dynamically linking LLVM does increase startup overhead.