Support dynamic linking with Rust ABI and not C ABI

[adityashah1212]

Now I know the rust team has shot down the idea of having stable ABI, and I am in favor of it, but there is a way to support dynamic linking with Rust ABI without defining a stable ABI. It is lot more cumbersome, but definitely possible. What I am proposing is that each crate creates a single shared library and exports a symbol like below.

RUSTC_1.49::<Crate>_<version>

This symbol will then be required by the executable linking to this shared library at runtime.

But to achieve this we will have to solve a few questions first

• When including crate version, should we include full version (major.minor.build) or just major.minor?
• Does Cargo currently pass crate versions to rustc, and if not we will have to define a way to do this.
• Expose a mechanism in cargo to build a crate as dynamic library, or use a crate as dynamic library
• Provide a way in cargo to pick a prebuilt shared library, in a way that the above symbol is verified before linking and not during linking

This advantage of supporting this form of shared libraries, is that as long as the rust compiler is same and the whole ecosystem is controlled (like in embedded linux world), we can leverage the use of shared libraries and reduce disk space and memory usage. Both memory and disk space can be a big issue in embedded linux systems.

[nagisa]

You can already link to crate-type=dylib crates provided both the library and the artifact linking to were compiled with the same compiler. Hashes contained within symbols will also prevent linking to the wrong shared library at runtime.

Cargo also knows how to link to crate-type="dylib" crates, though much like for rlib crates it will compile its own copy and link against that.

Provided the crates are all reproducible, the shared objects cargo produces and links to should all be the same for every crate build and therefore it should be sufficient to just keep one of them.

[adityashah1212]

@nagisa In that case would it possible to pick a crate-type=dylib crate as a binary crate. The reason I ask is at my company adopting Rust would become a major issue, since building everything you need every time will be very time consuming
(I am talking in hours) and also resource consuming. It would be better if crates could be pulled in binary form from a controlled centralized location in their binary forms and directly linked.

Also would it be possible to link a crate that is not defined as crate-type=dylib dynamically from the application end?

I know in case of generics, we would still have to compile the source at call site, but that is minor thing compared to compiling all dependent crates every time.

[Lokathor]

not that it fully fixes the issue, but as a temporary measure have you considered something like sccache?

[burdges]

It'll become interesting if build reproducibility plays some role here. I've no objection since reproducible build becomes super important for many reasons, but it's clearly a dramatic step. :)

As raised on internals, we need where Self: Sized bounds for associated consts and types, so they exist only in monomorpized code but disappear from trait objects. In this way, we'd merge many type erased traits with their underlying trait, so then easy default type erased methods avoid orphan rules conflicts resulting from global impls of type erasing traits. This should make dyn Trait more usable in practice, which serves some of the same goals.

[guilhermewerner]

It would be nice to be able to link against pre-compiled dylib, without having to use the C ABI.

This becomes very important on large projects, where compiling multiple crates can be time consuming.

For example, a game engine, where whoever uses it to build their games would have to recompile the entire engine, when it could just be precompiled, at least in debug mode.

[kornelski]

cc rust-lang/rust#105586

[LunarLambda]

In theory, Rust could leverage ELF shared object symbol versioning and similar tricks to prevent version mismatches between binaries and libraries.

Unfortunately this mechanism is limited to 1) ELF targets and 2) Quite possibly GNU/Linux targets specifically.

Short of inventing a brand new Truly Unified Rusty Binary Format there's gonna be no way around establishing some kind of stable ABI for Rust before dylibs/dynamic linking before something that's seriously usable without going through the C ABI.

As for "crates could ship precompiled binaries" this has happened before (serde_derive) and was seen as extremely controversial and problematic due to multiple reasons. So it's unlikely that that's going to be a motivating use case for enabling "Rust-ABI" dylibs.

[Rudxain]

Would crABI (rust-lang/rust#111423) solve this?