fix(fex): make the arm64 FEX runtime run linux/amd64 containers (no SVE, no FEXServer) - #24
Conversation
Vendor a build-time patch decoupling the FEX interpreter from FEXServer, so the statically-linked binfmt interpreter runs standalone inside OCI container namespaces (where /arcbox/runtime/bin/FEXServer is not visible and cannot be forked): - ConnectToAndStartServer only attaches to an existing server, never forks - FEXInterpreter::main no longer aborts when SetupClient fails - RootFS default "" -> "/" so standalone FEX falls back to host root (the rootfs the server would otherwise supply) Excludes the fork's CMake/CI/static-PIE changes; those are handled separately in the build configuration.
- apply patches/fex/*.patch after clone, before configure (wires in the FEXServer-removal patch, which was otherwise inert) - configure with the validated minimal arg set: TUNE_CPU=apple-m1 (no-SVE baseline) plus -static-pie/lld for a self-contained binfmt interpreter - build only the FEX target and stage build/Bin/FEX directly; drop FEXServer and the ninja-install staging (FEX runs standalone after the patch) - neutralize fork name in the patch comment
FEX-2605 bundles fmt and xxhash as submodules (recursive clone already fetches them), so installing libfmt-dev/libxxhash-dev only created a shared-vs-static link conflict that the static-xxHash workaround then patched around. Drop both, drop the workaround, and install only the toolchain the FEX build actually needs: build-essential clang lld llvm cmake ninja-build nasm.
PeronGH
changed the title
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — Fixes the FEX64 binfmt interpreter so it runs standalone on Apple Silicon by disabling SVE codegen (-DTUNE_CPU=apple-m1) and removing the hard FEXServer dependency via a vendored source patch, while trimming CI dependencies to the minimal set.
- Drop FEXServer dependency — Vendored patch (
patches/fex/0001-run-without-fexserver.patch) makesConnectToAndStartServernever fork andSetupClientfailure non-fatal; defaultsRootFSto/. - Apply patches after clone — New
apply_patchesfunction and--patches-dirCLI arg wire patch application into the build pipeline (between clone and configure). - SVE-safe build flags —
-DTUNE_CPU=apple-m1,-static-pie(replaces-static), explicit clang/clang++ compilers,-DENABLE_CCACHE=False. Drops-DENABLE_LTO=True,-DENABLE_ASSERTIONS=False,-DCMAKE_DISABLE_FIND_PACKAGE_fmt=True. - Remove ninja install — Stages binaries directly from
build/Bin/instead ofinstall/usr/bin/; builds only theFEXtarget. - Minimal CI deps — Cuts apt packages from ~20 to 7 core ones; removes the static xxHash workaround (FEX-2605 bundles the submodule).
Big Pickle (free) (credentials for Anthropic not configured) ｜ 𝕏
There was a problem hiding this comment.
Pull request overview
This PR updates the FEX runtime build pipeline to produce a standalone, statically-linked interpreter that avoids SVE-related SIGILL on Apple Silicon and no longer relies on launching FEXServer from an absolute path.
Changes:
- Apply vendored
*.patchfiles after cloning FEX to remove/relax theFEXServerdependency and adjust default RootFS behavior. - Rework the FEX build to compile only the
FEXtarget (noninja installstaging) and stage the built binary directly from the build output directory. - Simplify the release workflow’s FEX build dependencies to avoid shared-vs-static linking conflicts.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/cmd/build_fex_runtime.rs |
Adds patch application, switches to building/staging only FEX, and updates CMake configuration for a static PIE build. |
patches/fex/0001-run-without-fexserver.patch |
Vendors upstream source changes to reduce/avoid FEXServer dependency and set a container-friendly RootFS default. |
.github/workflows/release.yml |
Trims FEX build dependencies and removes the previous static-xxHash workaround step. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — One commit since the prior pullfrog review: propagates read_dir entry errors in apply_patches instead of silently skipping them, and syncs the README to reflect the updated build (static-pie, no FEXServer, apple-m1 tune).
- Propagate patch-dir read errors —
apply_patchesnow uses.map(…).collect::<io::Result<_>>()to propagateread_direntry I/O errors, rather than silently filtering them out with.filter_map(|e| e.ok()). - Sync README with current build — Replaced references to
-static+ FEXServer staging with-static-pie/ no-FEXServer /TUNE_CPU=apple-m1documentation, and added an Apple Silicon load-bearing details paragraph.
Big Pickle (free) (credentials for Anthropic not configured) ｜ 𝕏
- apply_patches: propagate per-entry read_dir errors instead of silently skipping, so a missing patch fails the build deterministically - README: FEX is staged alone as static-pie (not FEX+FEXServer/-static); document the no-SVE tune and the FEXServer-removal patch
45f0815 to
2f7a6a6
Compare
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — One commit since the prior pullfrog review (force-pushed replacement of 45f0815): propagates read_dir entry errors in apply_patches instead of silently skipping, and syncs README to document -static-pie, no FEXServer, and apple-m1 tune.
- Propagate patch-dir read errors —
apply_patchesreplaced.filter_map(|e| e.ok())(silently drops unreadable entries) with.map(…).collect::<io::Result>()which propagatesread_dirI/O errors, failing the build deterministically on disk issues. - Sync README — Updated build docs from
-static+ FEX+FEXServer to-static-pie/ FEX-only / no-FEXServer / apple-m1 tune.
Big Pickle (free) (credentials for Anthropic not configured) ｜ 𝕏
|
Incremental re-review of PR #24 complete. One new commit (
|
Problem
The shipped FEX64 runtime SIGILLs on first use on Apple Silicon, so
linux/amd64containers don't run. Root cause, confirmed by running it under gdb in an arm64 container:cnth) in FEX's own.text— ungated code the compiler auto-vectorised because the build tuned for an SVE-capable host. Apple Silicon implements only streaming-mode SVE via SME, so any non-streaming SVE instruction is illegal → SIGILL. The binary was riddled with it (642 SVE instructions)./arcbox/runtime/bin/FEXServer), which isn't visible inside an OCI container mount namespace — so even without the SVE fault it couldn't run standalone.Fix
patches/fex/0001-run-without-fexserver.patch) so the interpreter runs standalone: never fork/require a FEXServer, and defaultRootFSto/. Patch application is wired into the build (apply_patches, after clone, before configure) — it was previously committed but never applied.TUNE_CPU=apple-m1(no-SVE baseline, stops the compiler emitting SVE) +-static-pie/lld for a self-contained binfmt interpreter. Build only theFEXtarget and stagebuild/Bin/FEXdirectly; drop FEXServer andninja installstaging.fmt/xxhashsubmodules, solibfmt-dev/libxxhash-devonly created a shared-vs-static link conflict that a static-xxHash workaround then patched around. Drop both and the workaround; install onlybuild-essential clang lld llvm cmake ninja-build nasm.Verification
Built this configuration's FEX and ran it in a stock arm64 Alpine container against a static x86_64 busybox:
Rebuilding without the SVE-emitting tune dropped FEX-code SVE from 642 instructions to 0 (only glibc's dead ifunc variants and FEX's gated
rdvlprobe remain) and the binary runs clean.