Provide cross-compiling guidance when Apple SDK is missing

[madsmtm]

Emit more guided warnings when cross-compiling to Apple platforms from non-host macOS.

Note that in the common cross-compilation case (to macOS while linking with cc-like compiler), users won't actually hit this, since xcrun isn't invoked in that case. But I intend to change that in #131477, and then the error message here will start to matter a lot more.

The message now looks something like:

$ rustc +stage1 example.rs --target aarch64-apple-ios
warning: invoking `"xcrun" "--sdk" "iphoneos" "--show-sdk-path"` to find iPhoneOS.sdk failed: No such file or directory (os error 2)
  |
  = note: the SDK is needed by the linker to know where to find symbols in system libraries and for embedding the SDK version in the final object file
  = help: pass the path to the SDK using the SDKROOT environment variable
  = help: the SDK can be downloaded and extracted from https://developer.apple.com/download/all/?q=xcode (requires an Apple ID).
          
          The full Xcode bundle should contain the SDK in Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk.
  = warning: you will also need to use a linker capable of linking Mach-O files, consider using the bundled `lld` with `-Clinker=rust-lld`
  = warning: cross-compiling on iOS, tvOS, visionOS or watchOS (in particular the linking step) is ill supported on non-macOS hosts

Suggestions on how to improve it welcome!

Might also make sense to emit some of these help messages if the linker invocation itself fails? Unsure, but I'll work on that in a different PR.

(I originally considered not even attempting to invoke xcrun, but decided to continue doing that, since an xcrun-compatible interface can be present on other platforms, it's just unusual (e.g. Facebook developed xcbuild for a while, similar probably exist)).

Follow-up to #139010.
Part of #129432.
Related to rust-lang/rustup#1483 in that we now document the steps required for cross-compilation in the error message.

@rustbot label O-apple
r? wesleywiser since you just reviewed #139010
CC @BlackHoleFox @thomcc

[rustbot]

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

[madsmtm]

Converted to draft, because I want to think a little bit if we should make it a warning on host macOS too (to allow linking with e.g. zig cc on macOS while not having Xcode installed).

[wesleywiser]

As written, this looks fine to me so please ping me when you move it out of draft state (whichever way you decide) 🙂

[Mark-Simulacrum]

We might want to be a bit careful around recommending usage of the Apple SDK if our host triple isn't an apple triple - IIRC, the license terms don't permit usage of some parts of the SDK on non-apple hardware?

[madsmtm]

Ah, yeah, the Xcode SLA does say in section "2.2 A Permitted Uses and Restrictions":

Install a reasonable number of copies of the Apple Software [including the Apple SDKs] on Apple-branded computers that are owned or controlled by You to be used internally by You or Your Authorized Developers

And in section "2.7 Restrictions; No Other Permitted Uses":

The grants set forth in this Agreement do not permit You to, and You agree not to, install, use or run the Apple Software [including the Apple SDKs] or Apple Services on any non-Apple-branded computer or device, or to enable others to do so.

Btw, zig cc is clearly already violating that, since they bundle libSystem.tbd and headers from the macOS SDK, and thereby "enable others to use the Apple SDK on non-Apple-branded computers"...

I can change the wording to state that what the user is trying to do (cross link) is (probably, IANAL) not allowed according to Apple's terms? Though I still feel it important to provide these instructions in some shape or form.

---

Minutiae:

• codegen_ssa_xcrun_cross_about: technical description ("this is what an SDK is").
• codegen_ssa_xcrun_cross_env_var: also technical description ("this is how you pass SDKs to rustc").
• codegen_ssa_xcrun_cross_download_sdk: describes how the SDKs can be legally obtained.
  Note that I explicitly also chose to not mention that there are many ways of obtaining the SDKs without an Apple developer account.
  This way, I was hoping to defer the decision to the user, as they're obligated to read the Xcode SLA before using Xcode.
  But I could state explicitly here that doing this on non-Apple platforms is likely a violation of the SLA.
• codegen_ssa_xcrun_cross_linker_not_explicitly_set: technical recommendation ("here's an example of a linker you can easily and legally use on your platform").
• codegen_ssa_xcrun_cross_ill_supported_target: not sure, I'm really documenting the state of affairs of the linkers themselves (zig cc or lld), rustc's support is as good as for macOS. Should maybe just remove this.

[Mark-Simulacrum]

I think consulting with Foundation legal could be a good idea; I generally agree that as long as we're just pointing to Apple's pages on downloads etc. that's OK. Maybe adding some note around reading the terms would help.

I think the larger goal of this PR is not as clear to me though. It seems like, today, if I want to produce e.g. a dylib with Rust's standard library/core but no C code, I don't need the Apple SDK to do so -- lld is more or less happy to produce a Mach-O shared object (or non-shared object) from a pure Rust crate. If I want to compile C code for an Apple target, it's also possible that Apple SDKs are not required (e.g., because I'm not calling anything that requires Apple SDK headers).

But if we start requiring xcrun that seems to imply we think they are or should be required? Maybe this aspect of things relates more to the goals in #131477...?

[madsmtm]

It seems like, today, if I want to produce e.g. a dylib with Rust's standard library/core but no C code, I don't need the Apple SDK to do so -- lld is more or less happy to produce a Mach-O shared object (or non-shared object) from a pure Rust crate.

Hmm, perhaps I haven't been clear enough here. The Apple SDKs provides two things: headers and linker stubs.

lld is not able to produce neither a dylib nor binary without the linker stubs being available.

zig cc can in some cases because it bundles the linker stubs for system libraries (e.g. libSystem.tdb, the linker stub for libSystem.dylib, as well as libc.tdb etc.), but not system frameworks (CoreFoundation.framework/CoreFoundation.tdb etc.).

[Mark-Simulacrum]

lld is not able to produce neither a dylib nor binary without the linker stubs being available.

This seems untrue to me -- e.g., I produced this:

$ file libt.dylib
libt.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS|HAS_TLV_DESCRIPTORS>

Without linking any shared libraries on a x86_64-unknown-linux-gnu host with:

mkdir -p /tmp/sdk
SDKROOT=/tmp/sdk rustc +nightly t.rs --target=aarch64-apple-darwin --crate-type=dylib -Zunstable-options '-Clink-arg=-undefined dynamic_lookup' -Clinker=$HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld64.lld -Clinker-flavor=darwin-lld --verbose -Csave-temps

That fails with:

  = note: rust-lld: error: unknown argument '-flavor'
          rust-lld: error: unknown argument '-undefined dynamic_lookup'

But, if I edit that command to:

• Remove -flavor
• Fix -undefined ... to be two arguments, not one
• Remove -lSystem, -lc, -lm

That produces a libt.dylib that looks functional to me. Copying that to my macOS laptop, I can then link it with C and run it:

$ cat test.c
void rust_shared_foo();

int main() {
    rust_shared_foo();
    return 0;
}
$ clang test.c -lt -L. -o test
$ ./test
hello world from rust

Unless lld has bundled something (or maybe Rust has), I didn't use zig or similar here and AFAIK there's no linker stubs involved in producing this result.

[madsmtm]

Sorry for the delay in responding here. What you are saying is correct, by using -undefined dynamic_lookup you can delay basically the entire linking step to at runtime. This is a bad idea for several reasons, the primary one being performance, secondarily correctness (there are small things in Apple's frameworks which change behaviour depending on the SDK version that was linked with), which is why we don't explicitly offer it as an option nor do it by default in rustc.

Regardless, I have moved the "allow SDK to be missing" part to #131477, since it fits more clearly there, so this PR only contains the improved guidance when cross-compiling from non-macOS. And I have sent an email to the Foundation asking for legal help here.

@rustbot blocked (on Foundation legal)

[apiraino]

Hello, checking progress here.

Is this comment still actual? I think I don't see a discussion on Zulip with the Foundation about this.

Are there any updates?

[madsmtm]

I talked with some Rust Foundation people back in late August (email and a call), they were busy at the time with RustConf IIRC. I've sent a follow-up email now.

[geofft]

One thing about Zig's approach is they're only copying in the one library-stubs file libSystem.tbl and (for the most part) standard POSIX-y headers under the APSL. The SDK also contains a large number of other library stubs and headers for Apple-specific frameworks (e.g. Core Foundation, Cocoa, etc.); Zig isn't touching those. The entire source for libSystem is under the APSL, and so presumably this one stubs file is at worst under the APSL (and perhaps even uncopyrightable or fair use). Most of the Apple-specific frameworks are not. So, while the restrictive terms apply to the SDK as a whole, if all we need is libSystem, I think that makes things easier legally. I haven't found a record of why specifically Zig decided it was okay to vendor these files from the SDK, but that is what they do; they're not generating these files on their own in some other way.

I actually want to propose that rustc bundle a copy of libSystem.tbd (libc.tbd and libm.tbd are just symlinks to it) and automatically add it to the linker's library search path, maybe only if we're cross-compiling from a non-Mac system and SDKROOT is unset. If we do this, it would get us out-of-the-box macOS cross compilation support in the way Zig does. Combined with a cross-linker like lld, this appears to be the only file you need to successfully cross-build for macOS on Linux. (You don't even need the C headers because Rust binds to libc at the ABI level.) Specifically, if you throw libSystem.tbd (acquired in whatever way, e.g. from Zig's sources) in e.g. /tmp/libs, make two symlinks libc.tbd and libm.tbd to it, and create a .cargo/config.toml with something like the following, cargo build on Linux produces a working macOS binary for at least simple cases like the cargo init --bin hello-world:

[build]
target = "x86_64-apple-darwin"

[target.x86_64-apple-darwin]
linker = "clang-20"
rustflags = ["-C", "link-args=-fuse-ld=lld -target x86_64-apple-darwin", "-L", "/tmp/libs"]

[target.aarch64-apple-darwin]
linker = "clang-20"
rustflags = ["-C", "link-args=-fuse-ld=lld -target aarch64-apple-darwin", "-L", "/tmp/libs"]

(There might be better syntax for this, but also I am proposing we get to the point where there is no need for any user-provided syntax for this!)

So this would probably be a slightly different legal question. Instead of "Is it okay to suggest that users could download the whole SDK when they're likely on a system that you're not allowed to do that on," we're asking "is it okay for us to ship this one file which is derived from an APSL source but reaches us through a Mac, just like Zig has been doing for 4+ years".

(I do suspect you could regenerate this file from the public Apple OSS code drops if you tried hard enough, but that would likely turn into more ongoing maintenance work in addition to the up-front work, and I'd also worry about the risk of discrepancies with the real SDK.)

[madsmtm]

I actually want to propose that rustc bundle a copy of libSystem.tbd

I've been considering the same thing. A semi-alternative would be #146357.

One wrinkle is that this naturally forces rustc to keep it up-to-date with the latest SDK release (and its presence might affect the version and compatibility_version that's embedded in the final dylib, haven't tested that enough), not yet sure how to resolve that. We'd also probably need flags to control the SDK version used in the final binary (it has concrete effects, the best example I know of is that -[NSView wantsBestResolutionOpenGLSurface] changes behaviour depending on the SDK version).

Combined with a cross-linker like lld, this appears to be the only file you need to successfully cross-build for macOS on Linux.

Assuming you don't use macOS-specific frameworks, then yes.

You don't even need the C headers because Rust binds to libc at the ABI level.

See rust-lang/cc-rs#1585 for ideas on that.

So this would probably be a slightly different legal question. Instead of "Is it okay to suggest that users could download the whole SDK when they're likely on a system that you're not allowed to do that on," we're asking "is it okay for us to ship this one file which is derived from an APSL source but reaches us through a Mac, just like Zig has been doing for 4+ years".

Not just slightly, it's significantly different.

In the email I sent to the Rust Foundation I was trying to get clarification on how much of objc2 is allowed at the same time, that's probably why it stalled out TBH.

Feel free to send them an email yourself, that might help move things along.

[geofft]

Yeah, the use cases I have in mind are for cross-platform code anyway which generally don't need the Apple-specific frameworks. That said, a good handful of other useful frameworks appear to be APSL (SystemConfiguration, Security, etc.) or Apache (CoreFoundation), so the same argument could apply to including their .tbd files too, and including a few of those might be worth doing too for various conditionally-compiled Mac dependencies, though it's a little bit of scope creep. (I guess Zig only includes libSystem, and simply getting to parity with Zig seems like a good start IMO.)

I've been super out of the loop in Rust for a while, are you just contacting the Foundation's public contact address or do you have a contact specifically with legal? I'm definitely happy to email the public contact address with the question of "can we do what Zig does for this one file" and see how that goes.

Thanks for the link to that cc-rs issue! Seems like it would be helpful for -sys crates.

(it has concrete effects, the best example I know of is that -[NSView wantsBestResolutionOpenGLSurface] changes behaviour depending on the SDK version).

Do you actually see the behavior change, or are you just looking at the docs? Naively, if I compile

#import <AppKit/NSView.h>
#import <AppKit/NSOpenGLView.h>

int main(void) {
	NSOpenGLView *v = [[NSOpenGLView alloc] initWithFrame:NSMakeRect(1, 2, 3, 4) pixelFormat:[NSOpenGLView defaultPixelFormat]];
	printf("%d\n", v.wantsBestResolutionOpenGLSurface);
}

with cc -mmacos-version-min=10.13 -framework AppKit or 10.14 or 10.15 or even 10.0, I am always getting 1 (true). Even if I futz around with the compatibility version for AppKit in the binary or dig up an old AppKit.tbd file from the internet, that doesn't change the answer (and the compatibility version as of 10.13 was 45, the same as it is now). I am running on macOS 15.6.1.

In any case, Xcode only installs the single latest SDK and gives you -mmacos-version-min as your only knob, so I don't think we're losing any functionality that actual Xcode users writing in (Objective-)C would have available by just vendoring the single latest .tbd file and bumping it in new releases.

[madsmtm]

a good handful of other useful frameworks appear to be APSL (SystemConfiguration, Security, etc.) or Apache (CoreFoundation), so the same argument could apply to including their .tbd files too, and including a few of those might be worth doing too for various conditionally-compiled Mac dependencies, though it's a little bit of scope creep. (I guess Zig only includes libSystem, and simply getting to parity with Zig seems like a good start IMO.)

I don't think rustc / std needs to bundle those, ideally we'd instead have some sort of a mechanism to allow it to done by downstream crates (like my objc2 crates).

I've been super out of the loop in Rust for a while, are you just contacting the Foundation's public contact address or do you have a contact specifically with legal?

I wrote to them at contact@rustfoundation.org, and then I talked with @LawnGnome (adamharvey at former email domain).

It's on my todo-list to get a hold of some contacts internally Apple to help with these kinds of things too, so you might also just wanna wait for me to get around to doing that, idk.

with cc -mmacos-version-min=10.13 -framework AppKit or 10.14 or 10.15 or even 10.0

The thing that influences this particular case is actually the SDK part of the -platform_version linker flag, which is passed automatically by Clang by reading it from the current Xcode SDK.

I don't recall seeing cases that are influenced by the version/compatibility_version of the dylib, but I assume they exist somewhere.

I don't think we're losing any functionality that actual Xcode users writing in (Objective-)C would have available by just vendoring the single latest .tbd file and bumping it in new releases.

Prooobably not, though I'm still wary of overriding the versions from the SDK (e.g. if users are intentionally using an older Xcode to build their applications, it'd be surprising if we started making their app behave as-if it was built using a newer Xcode).

[geofft]

The thing that influences this particular case is actually the SDK part of the -platform_version linker flag, which is passed automatically by Clang by reading it from the current Xcode SDK.

Aha, thanks, I can see the difference with -Wl,-platform_version,macos,10.14,10.14 vs. -Wl,-platform_version,macos,10.15,10.15. Interestingly this produces more of a diff in the binary than just LC_BUILD_VERSION, but I can also see the difference from manually changing the LC_BUILD_VERSION alone and nothing else.

Prooobably not, though I'm still wary of overriding the versions from the SDK (e.g. if users are intentionally using an older Xcode to build their applications, it'd be surprising if we started making their app behave as-if it was built using a newer Xcode).

To be clear, I'm only proposing that we implicitly use a bundled libSystem.tbd if both we're not on macOS / are cross-compiling and SDKROOT isn't set. If you're on macOS and you're compiling software, there's little reason not to install the SDK the normal way, especially as you're going to need stuff like a linker anyway. (But we can make it available as an option.) If you're setting SDKROOT we should respect that. So this should only take effect in the case where you do not have an existing SDK available to override and have no obvious better (slash legal) way of getting one.

Also, if it is just LC_BUILD_VERSION and not something in the tbd file, then that's much easier to expose as an option if we really want to / it means that updates to the tbd file itself don't implicate behavior changes.

[rust-bors]

☔ The latest upstream changes (presumably #152156) made this pull request unmergeable. Please resolve the merge conflicts.