Enable `outline-atomics` by default on more AArch64 platforms #144938

tgross35 · 2025-08-05T02:29:48Z

The baseline Armv8.0 ISA doesn't have atomics instructions, but in
practice most hardware is at least Armv8.1-A (2014), which includes
single-instruction atomics as part of the LSE feature. As a performance
optimization for these cases, GCC and LLVM have the -moutline-atomics flag
to turn atomic operations into calls to symbols like __aarch64_cas1_acq.
These can do runtime feature detection and use the LSE instructions if
available, falling back to more portable load-exclusive/store-exclusive
loops.

Since the recent 3b50253 ("compiler-builtins: plumb LSE support
for aarch64 on linux") our builtins support this LSE optimization, and
since 6936bb9 ("Dynamically enable LSE for aarch64 rust provided
intrinsics"), std will set the flag as part of its startup code. The first
commit in this PR configures this to work on all platforms built with
outline-atomics, not just Linux.

Thus, enable outline-atomics by default on Android, FreeBSD, OpenBSD,
Windows, Fuchsia, and Apple platforms that don't have LSE in the baseline.
The feature is already enabled on Linux. Platform-specific details are
included in each commit message.

The current implementation can still be accessed by setting
-Ctarget-feature=-outline-atomics. Setting -Ctarget-feature=+lse or
a relevant CPU will use the single-instruction atomics without the call
overhead. https://rust.godbolt.org/z/dsdrzszoe

Link: https://learn.arm.com/learning-paths/servers-and-cloud-computing/lse/intro/
Original Clang outline-atomics benchmarks: https://reviews.llvm.org/D91157#2435844

try-job: aarch64-apple
try-job: aarch64-msvc-*
try-job: arm-android
try-job: dist-android
try-job: dist-aarch64-windows-gnullvm
try-job: dist-aarch64-apple
try-job: dist-aarch64-msvc
try-job: dist-various-*
try-job: dist-x86_64-freebsd
try-job: test-various

rustbot · 2025-08-05T02:29:52Z

r? @ChrisDenton

rustbot has assigned @ChrisDenton.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-08-05T02:29:55Z

These commits modify compiler targets.
(See the Target Tier Policy.)

tgross35 · 2025-08-05T02:30:02Z

Whoops

r? @ghost

[experiment] enable outline-atomics on more aarch64 platforms try-job: arm-android try-job: dist-android try-job: dist-x86_64-freebsd try-job: dist-aarch64-windows-gnullvm try-job: dist-aarch64-apple try-job: aarch64-msvc-1 try-job: aarch64-msvc-2 try-job: dist-aarch64-msvc

Enable outline-atomics by default on more aarch64 platforms try-job: aarch64-apple try-job: aarch64-msvc-1 try-job: aarch64-msvc-2 try-job: arm-android try-job: dist-android try-job: dist-aarch64-windows-gnullvm try-job: dist-aarch64-apple try-job: dist-aarch64-msvc try-job: dist-various try-job: dist-x86_64-freebsd

Enable `outline-atomics` by default on more AArch64 platforms try-job: aarch64-apple try-job: aarch64-msvc-* try-job: arm-android try-job: dist-android try-job: dist-aarch64-windows-gnullvm try-job: dist-aarch64-apple try-job: dist-aarch64-msvc try-job: dist-various-* try-job: dist-x86_64-freebsd try-job: test-various

rustbot · 2025-09-06T00:49:43Z

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

tgross35 · 2025-09-06T00:56:24Z

Cc target maintainers:

Apple @madsmtm
Android @maurer @mgeisler
FreeBSD @asomers
Fuchsia @erickt, @Nashenas88
OpenBSD @semarie
Windows @ChrisDenton @dpaoliello @wesleywiser

Zulip discussion: #t-compiler > outline-atomics on non-Linux targets

madsmtm

Looked through it as best I could, but this is a fair bit outside my comfort zone.

Is there a test or two I could run to verify whether this works on-device?

View changes since this review

library/std/src/sys/configure_builtins.rs

tgross35 · 2025-09-06T02:26:09Z

Is there a test or two I could run to verify whether this works on-device?

It's easy enough to check that outline atomics are used, just inspecting the code used by https://rust.godbolt.org/z/PWqeKP3W5. Verifying that feature detection works so LSE gets used is trickier though. The easiest thing is to call something that will use out-of-line atomics (like the example from that godbolt) and step through with a debugger, and ensure it is hitting the atomic instruction at this bit of code https://github.com/rust-lang/compiler-builtins/blob/a63d089c673aa9397d583c3cef506ad457c5f403/compiler-builtins/src/aarch64_linux.rs#L146-L148 rather than going to the fallback. Alternatively add an extern static or something that binds to the mangled name of HAVE_LSE_ATOMICS in that module, and just print to make sure it's nonzero.

(requires our compiler-builtins, would be similar but slightly different if using the C versions)

rust-bors · 2025-09-06T02:59:29Z

☀️ Try build successful (CI)
Build commit: 7041613 (704161326f1460ec409add8d318ad1976f8f3eb5, parent: 6c699a37235700ab749e3f14147fe41d49c056e8)

Build outline atomic symbols on all targets that have `outline-atomics` enabled, rather than only on Linux. Since this is no longer OS-specific, also rename the module.

Change the gating and link sections to enable this for any platforms that enable `outline-atomics`, rather than only Linux. Additionally, no longer run this if LSE is available, since in this case the outline versions will never be called.

Darwin and the `-sim` targets already have a baseline with LSE. Enable `outline-atomics` on all other Apple targets here.

Windows has a similar flag `/forceInterlockedFunctions`, which uses names such as `_InterlockedAdd64_rel`.

Per LLVM commit c5e7e64 ("[AArch64][Clang][Linux] Enable out-of-line atomics by default.") [1], Clang enables these on Android. Thus, do the same in Rust. [1]: llvm/llvm-project@c5e7e649d537067de

Clang does not currently have this enabled on FreeBSD, but there doesn't seem to be any specific reason not to. Thus, enable it here.

Clang has done this by default since LLVM commit 1a963d3 ("[Driver] Make -moutline-atomics default for aarch64-fuchsia targets"), [1], so do the same here. [1]: llvm/llvm-project@1a963d3

Clang has recently started doing this, as of LLVM commit 5d774ec8d183 ("[Driver] Enable outline atomics for OpenBSD/aarch64") [1]. Thus, do the same here. [1]: llvm/llvm-project@5d774ec

tgross35 · 2025-09-06T08:24:46Z

@bors2 try

Enable `outline-atomics` by default on more AArch64 platforms try-job: aarch64-apple try-job: aarch64-msvc-* try-job: arm-android try-job: dist-android try-job: dist-aarch64-windows-gnullvm try-job: dist-aarch64-apple try-job: dist-aarch64-msvc try-job: dist-various-* try-job: dist-x86_64-freebsd try-job: test-various

rust-bors · 2025-09-06T10:36:22Z

☀️ Try build successful (CI)
Build commit: daab644 (daab644ace186ab496d1c345a2ea59ae472a49c4, parent: 397f93362974d298b79e0e0cd43677014aa7b722)

semarie · 2025-09-06T12:48:22Z

@tgross35 thanks for the Cc for OpenBSD.

I checked with people involved in low level aarch64, and it should be fine for OpenBSD.

We are already using -moutline-atomics for some time in the kernel and we recently made it the default for our base clang (based on llvm-19). We also have a libcompiler-rt with runtime feature detection code necessary to enable LSE on systems that support it (which was upstreamed).

davidtwco

LGTM, up to you if you want to wait to hear back from the target maintainers individually before r=me

View changes since this review

Nashenas88 · 2025-09-10T05:38:48Z

I think this is good for fuchsia. I'm going to run it by some folks tomorrow morning.

Nashenas88 · 2025-09-10T17:30:05Z

No concerns from other folks on fuchsia. Thanks for working on this!

Nashenas88 · 2025-09-10T21:54:02Z

library/std/src/sys/configure_builtins.rs

+#[cfg_attr(
+    not(any(target_vendor = "apple", target_os = "windows")),
+    unsafe(link_section = ".init_array.90")
+)]
 static RUST_LSE_INIT: extern "C" fn() = {
    extern "C" fn init_lse() {
        use crate::arch;


This comment is actually for line 37 below. Shouldn't the comment be updated to refer to the renamed file?

-// This is provided by compiler-builtins::aarch64_linux. +// This is provided by compiler-builtins::aarch64_outline_atomics.

madsmtm

I verified that __rust_enable_lse is run on Apple platforms when having -lse,+outline-atomics.

It isn't actually used though, the code will always fall back to the slower, non LSE implementation (regardless of whether the current CPU actually supports it), since try_lse_op! is gated with .arch_extension lse, which doesn't seem to be enabled on Apple platforms?

And if I just remove it, I get:

error: ADR/ADRP relocations must be GOT relative
  |
note: instantiated into assembly here
 --> <inline asm>:6:14
  |
6 | adrp    x16, __ZN17compiler_builtins23aarch64_outline_atomics16HAVE_LSE_ATOMICS17h7f004a93068845c5E; nop; ldrb    w16, [x16, :lo12:__ZN17compiler_builtins...
  |              ^

error: unknown AArch64 fixup kind!
  |
note: instantiated into assembly here
 --> <inline asm>:6:14
  |
6 | adrp    x16, __ZN17compiler_builtins23aarch64_outline_atomics16HAVE_LSE_ATOMICS17h7f004a93068845c5E; nop; ldrb    w16, [x16, :lo12:__ZN17compiler_builtins...
  |              ^

That error seems to come from around here in AArch64MachObjectWriter.cpp, which seems to hint that the way the check is implemented isn't supported on Mach-O? Unsure.

View changes since this review

rustbot assigned ChrisDenton Aug 5, 2025

rustbot unassigned ChrisDenton Aug 5, 2025

tgross35 marked this pull request as draft August 5, 2025 02:30

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 5, 2025

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

tgross35 force-pushed the more-outline-atomics branch from a25e101 to e20add7 Compare September 5, 2025 19:39

tgross35 changed the title ~~[experiment] enable outline-atomics on more aarch64 platforms~~ Enable outline-atomics by default on more aarch64 platforms Sep 5, 2025

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

tgross35 marked this pull request as ready for review September 6, 2025 00:49

rustbot assigned davidtwco Sep 6, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 6, 2025

madsmtm reviewed Sep 6, 2025

View reviewed changes

library/std/src/sys/configure_builtins.rs Outdated Show resolved Hide resolved

library/std/src/sys/configure_builtins.rs Outdated Show resolved Hide resolved

library/std/src/sys/configure_builtins.rs Show resolved Hide resolved

tgross35 added 8 commits September 6, 2025 03:51

compiler-builtins: Change gating for outline atomic symbols

57f88d4

Build outline atomic symbols on all targets that have `outline-atomics` enabled, rather than only on Linux. Since this is no longer OS-specific, also rename the module.

Enable outline-atomics by default on AArch64 Apple platforms

dafcd25

Darwin and the `-sim` targets already have a baseline with LSE. Enable `outline-atomics` on all other Apple targets here.

Enable outline-atomics by default on AArch64 Windows platforms

f6d958f

Windows has a similar flag `/forceInterlockedFunctions`, which uses names such as `_InterlockedAdd64_rel`.

Enable outline-atomics by default on AArch64 Android

d49a46a

Per LLVM commit c5e7e64 ("[AArch64][Clang][Linux] Enable out-of-line atomics by default.") [1], Clang enables these on Android. Thus, do the same in Rust. [1]: llvm/llvm-project@c5e7e649d537067de

Enable outline-atomics by default on AArch64 FreeBSD

d8eb386

Clang does not currently have this enabled on FreeBSD, but there doesn't seem to be any specific reason not to. Thus, enable it here.

Enable outline-atomics by default on AArch64 Fuchsia

0f135f7

Clang has done this by default since LLVM commit 1a963d3 ("[Driver] Make -moutline-atomics default for aarch64-fuchsia targets"), [1], so do the same here. [1]: llvm/llvm-project@1a963d3

Enable outline-atomics by default on AArch64 OpenBSD

b18a6c6

Clang has recently started doing this, as of LLVM commit 5d774ec8d183 ("[Driver] Enable outline atomics for OpenBSD/aarch64") [1]. Thus, do the same here. [1]: llvm/llvm-project@5d774ec

tgross35 force-pushed the more-outline-atomics branch from 4c4f34f to b18a6c6 Compare September 6, 2025 08:19

This comment has been minimized.

Sign in to view

davidtwco approved these changes Sep 9, 2025

View reviewed changes

Nashenas88 reviewed Sep 10, 2025

View reviewed changes

madsmtm reviewed Sep 11, 2025

View reviewed changes

Enable outline-atomics by default on more AArch64 platforms #144938

Are you sure you want to change the base?

Enable outline-atomics by default on more AArch64 platforms #144938

Uh oh!

Conversation

tgross35 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Aug 5, 2025

Uh oh!

rustbot commented Aug 5, 2025

Uh oh!

tgross35 commented Aug 5, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment has been minimized.

This comment has been minimized.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment has been minimized.

This comment was marked as outdated.

This comment was marked as outdated.

This comment has been minimized.

rustbot commented Sep 6, 2025

Uh oh!

tgross35 commented Sep 6, 2025

Uh oh!

madsmtm left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgross35 commented Sep 6, 2025

Uh oh!

rust-bors bot commented Sep 6, 2025

Uh oh!

tgross35 commented Sep 6, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Sep 6, 2025

Uh oh!

semarie commented Sep 6, 2025

Uh oh!

davidtwco left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nashenas88 commented Sep 10, 2025

Uh oh!

Nashenas88 commented Sep 10, 2025

Uh oh!

Nashenas88 Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

madsmtm left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Enable `outline-atomics` by default on more AArch64 platforms #144938

Enable `outline-atomics` by default on more AArch64 platforms #144938

tgross35 commented Aug 5, 2025 •

edited

Loading

madsmtm left a comment •

edited by rustbot

Loading

davidtwco left a comment •

edited by rustbot

Loading

madsmtm left a comment •

edited by rustbot

Loading