Upgrade the pinned Rust toolchain to latest stable (1.98.1 → 1.99+) #1322

Activity

1. jwulf commented on Oct 1, 2026

   

   jwulf

   on Oct 1, 2026

   CollaboratorAuthor

   More actions

   Claimed — worktree rust-toolchain-upgrade, branch chore/rust-toolchain-upgrade.
2. jwulf commented on Oct 1, 2026

   

   jwulf

   on Oct 1, 2026

   CollaboratorAuthor

   More actions

   Investigation: root cause of the armv6 __aeabi_uread4 link failure

   Root cause (confirmed). Three facts combine to break the armv6 link:

   1. LLVM 23, which ships in Rust 1.99, now emits calls to __aeabi_uread4 / __aeabi_uwrite4 / __aeabi_uread8 / __aeabi_uwrite8 for unaligned accesses on strict-align ARM targets. Upstream tracking: Undefined symbol errors on __aeabi_uread* and __aeabi_uwrite* functions with opt-level = "z" and build-std rust-lang/rust#162259. The reference comes from the prebuilt libstd (std::backtrace_rs → object crate), not from our code. Only armv6 (arm-unknown-linux-gnueabihf) is strict-align. armv7, aarch64 and x86_64 link fine on 1.99.
   2. Rust fixed this in its own compiler_builtins (Add aeabi_u(read|write)(4|8) intrinsics. rust-lang/compiler-builtins#1317, backported to 1.99). The 1.99 libcompiler_builtins rlib for this target does define all four as weak symbols (checked with llvm-nm).
   3. cargo-zigbuild drops the libcompiler_builtins-*.rlib from the link on ARM (the filter_linker_arg function in 0.23.2) and relies on zig's compiler-rt instead, which doesn't provide these helpers. So the definitions Rust ships never reach the linker.

   Reproduction. I used a minimal binary (std + tikv-jemallocator, as in the gateway) with zig 0.13.0 and cargo-zigbuild 0.23.2, which is CI's exact pair:

   rustc	armv6	armv7 / aarch64 / x86_64
   1.98.1	✅	✅
   1.99.0	❌ undefined symbol: __aeabi_uread4 (with no, thin or fat LTO)	✅

   Upstream fix and why we can't take it yet. cargo-zigbuild 0.23.3 (commit 236ff1da) injects a weak C shim for these helpers. But 0.23.3 and 0.23.4 append that shim to every zig cc invocation that isn't -c, and that includes jemalloc's autoconf/C build. Results:

   • zig 0.13 + 0.23.4: aeabi_unaligned.c: FileNotFound and cannot specify -o when generating multiple output files
   • zig 0.16 + 0.23.4: the same, plus unsupported argument 'generic+v6+strict_align+vfp2-d32' to option '-mcpu='

   That's the regression already noted in the workflow's pin comment. It still applies in 0.23.4.

   Fix (PR to follow).

   • We keep zig 0.13 and cargo-zigbuild 0.23.2, and link the same weak shim ourselves, only at rustc's final link: CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS=-C link-arg=<abs>/server/build-support/aeabi_unaligned.c.
   • That variable is set by scripts/zig-arm-shim-env.sh, which both the publish workflow and scripts/cross-build.sh source.
   • The path must be absolute. I first tried a relative path in .cargo/config.toml; it failed for the engine-core cdylib, a path dependency outside the server workspace, which links from its own directory.
   • The shim compiles to byte loads/stores under zig's strict_align -mcpu, which I checked by disassembly, so it can't recurse into itself.
   • The definitions are weak, so a toolchain that provides real ones takes precedence.

   Verified locally on 1.99.0 with the shim:

   • A full gateway release build for armv6 succeeds (--features console, thin LTO / 16 CGUs, as in CI).
   • Clippy -D warnings is clean for engine-core (all features), read-model, processos (default and --no-default-features) and the server workspace.
   • Tests: engine-core nextest 942/942, server nextest 1122/1122, engine-wasm tests pass.

   Not yet done:

   • Running the armv6 binary under emulation. A Docker bind mount is blocked in my sandbox.
   • Running the darwin / Windows / processos release legs on 1.99. Normal CI doesn't run them, and the workflow_dispatch paths publish things: the c8ctl rolling binaries release + plugin release PR, and the processos rolling prerelease + S3 latest.

   Possible follow-up upstream: report the 0.23.3+ shim-injection regression to rust-cross/cargo-zigbuild. The shim should only be added at link steps, not to C configure/compile steps. Once that's fixed we can drop our shim and take the newer cargo-zigbuild.
3. jwulf commented on Oct 1, 2026

   

   jwulf

   on Oct 1, 2026

   CollaboratorAuthor

   More actions

   PR: #1323 (draft).
4. jwulf

   added a commit that references this issue on Oct 2, 2026

   chore: upgrade Rust 1.98.1 → 1.99.0; link ARM-RTABI shim for armv6 zi…

   ...

   a6dff95
5. jwulf

   closed this as completedin #1323on Oct 2, 2026
6. jwulf commented on Oct 2, 2026

   

   jwulf

   on Oct 2, 2026

   CollaboratorAuthor

   More actions

   Merged in #1323 (a6dff95). On the branch, a manual publish-c8ctl-binaries run (36939022256) built all 7 gateway binaries on rustc 1.99.0, including linux-armv6, and the upload/plugin-release job succeeded. The ProcessOS release builds weren't run on 1.99; the next tag will exercise them.
7. jwulf commented on Oct 2, 2026

   

   jwulf

   on Oct 2, 2026

   CollaboratorAuthor

   More actions

   Upstream follow-up: this cargo-zigbuild regression is already reported as rust-cross/cargo-zigbuild#478 and fixed by rust-cross/cargo-zigbuild#481, merged 2026-09-09 but not yet released (latest is v0.23.4). I narrowed it down: from 0.23.3 on, cargo-zigbuild treats -M/-MM dependency generation (jemalloc's Makefile) as a link, so it appends aeabi_unaligned.c, and zig then fails with cannot specify -o when generating multiple output files. Minimal repro: run the cargo-zigbuild zigcc wrapper with -MM -MT t.o -o t.d t.c; it succeeds on 0.23.2 and fails on 0.23.4. Once a release includes #481, we can bump cargo-zigbuild and drop server/build-support/aeabi_unaligned.c and scripts/zig-arm-shim-env.sh (the guard test goes with them).