feat: add AppImage bundle target - #46
Open
s-celles wants to merge 2 commits into
Open
Conversation
An AppImage is mounted rather than installed, so a bundled Julia distribution never writes its tens of thousands of files to disk. That cost — the hydration problem — is what makes the other formats impractical on HPC filesystems, and is the motivation in issue PeaceFounder#42. The format is a runtime ELF followed by a squashfs image, and mksquashfs was already a dependency for Snap, so the work is the AppDir layout, the desktop and AppStream metadata, and obtaining the runtime. - `AppImage` config struct, `stage` and `bundle`, following the Snap shape. - `AppImagePack` with `pack`, `offset` and `unpack`. `unpack` reads the payload without running the runtime, which is the fallback where FUSE is unavailable. - `AppImageRuntime` resolving the runtime from an explicit path, the `appimage_runtime` preference, or `AppImageRuntime_jll` once that package is registered. There is no jll yet because Yggdrasil has neither squashfuse nor libfuse; that work proceeds separately and does not block this. - `appimage_depot` selects where a Julia payload keeps its depot. The mounted filesystem is read-only and transient, so `"app"` points USER_DATA at $XDG_DATA_HOME/<app>; `"julia"` keeps the stock depot via a startup file that sets up the load path without replacing DEPOT_PATH, since every set_depot_path_* in AppEnv begins with empty!(DEPOT_PATH). Two implementation details worth recording. `offset` derives the payload position from the ELF section header table rather than scanning for the squashfs magic: the runtime embeds squashfuse, whose own "hsqs" constant sits 750 KB before the real payload, so a scan reports a position inside the runtime. And `pack` has mksquashfs write into the output file after a reserved prefix instead of building the filesystem separately and concatenating, which would need a second full-size temporary copy — hundreds of megabytes, usually in /tmp. Verified against a real runtime end to end: the offset agrees with the runtime's own --appimage-offset, the payload carries a working distribution, and the application runs with its depot landing under XDG_DATA_HOME. Mounting itself is untested here, as this machine has /dev/fuse but no fusermount. Closes PeaceFounder#42 Assisted-by: AI
Building the AppImage runtime from source made it possible to check a claim I had only assumed: that squashfuse handles xz. It does not, in this configuration. The runtime bundles squashfuse built against zstd and zlib alone, and mksquashfs will happily produce an xz image that then fails to mount. Verified against the official upstream runtime, not just a locally built one: an xz payload gives "Failed to extract AppImage" with both, while zstd and gzip work with both. Accepting xz meant AppBundler could produce an AppImage that no user could run, with nothing failing until launch. It is now rejected at configuration time, alongside lz4. Assisted-by: AI
Member
|
An AppImage prototype is already in progress in #44, so there's a fair bit of overlap here. There are good ideas in this one though, and I'll pull them in where they fit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #42.
An AppImage is mounted rather than installed, so a bundled Julia distribution never writes its tens of thousands of files to disk. That cost — the hydration problem you raised — is what makes the other formats impractical on HPC filesystems.
You were right that
mksquashfsdoes most of the work. The format turns out to be nothing more than a runtime ELF followed by a squashfs image, so the remaining work was the AppDir layout, the desktop and AppStream metadata, and getting hold of the runtime.The depot question
You left this open in the issue, so it is a
appimage_depotpreference rather than a decision baked in:"app"(default) —AppRunpointsUSER_DATAat$XDG_DATA_HOME/<app>. The mounted filesystem is read-only and disappears when the application exits, so the depot has to live somewhere; this gives a persistent per-user one and never touches the host's~/.julia."julia"— what you floated in the issue: the stock depot, left as Julia found it, with the bundledshare/juliaappended so the shipped packages stay resolvable. This needs its ownstartup.jl, because everyset_depot_path_*in AppEnv begins withempty!(DEPOT_PATH), soAppEnv.init()cannot be used in that mode.The runtime
This is the one part that is not self-contained. The runtime is a ~900 KB static-PIE ELF, and there is no jll: it links
libsquashfuse,libsquashfuse_ll,libfuse3,libzstd,libzandlibmimalloc, and Yggdrasil hasmimalloc,ZstdandZlibbut neithersquashfusenorlibfuse. Packaging it properly is three recipes, and the artifact also sits awkwardly in Yggdrasil's model — it is keyed only by target CPU architecture and must run on any Linux, rather than following the glibc/musl platform split.So the source is pluggable: an explicit path or the
appimage_runtimepreference today,AppImageRuntime_jllautomatically once it exists. That Yggdrasil work is being prepared separately and does not block this. Until then the error message names both remedies, since this is the one step that fails for a reason outside the user's own project.Two details worth flagging in review
offsetreads the ELF section header table, not the squashfs magic. My first implementation scanned forhsqsand was wrong: the runtime embeds squashfuse, whose ownhsqsconstant sits at byte 194183 — some 750 KB before the real payload. The end-to-end test caught it because the runtime's--appimage-offsetreported 944632 and mine reported 194183. The unit test had passed only because its stub runtime was a blank buffer, so the stub now embeds a decoyhsqsand is a plausible ELF.packwrites into the output file after a reserved prefix.mksquashfs -offsetreserves room for the runtime, which then goes in over it —-offsetzeroes the skipped bytes rather than preserving them, so the order matters. Building the squashfs separately and concatenating would need a second full-size temporary copy: for a Julia distribution that is hundreds of megabytes, usually landing in/tmp, andmksquashfsreports exhaustion only asFATAL ERROR: Probably out of space on output filesystem. I hit exactly that while developing, so the docs also note the build-time space requirement.Verification
test/appimage.jl— 52 assertions with a stub runtime, so no network and no jll: AppDir layout (theIcon=key naming the icon with no path or extension,.DirIcon, the freedesktop copies), offset and round-trip throughunpack, compressor and depot validation, runtime resolution, and that the"julia"-modestartup.jlrenders to parseable Julia which calls the AppEnv entry points it depends on rather thanAppEnv.init().test/appimage_e2e.jl— opt-in (JULIA_RUN_APPIMAGE_E2E=true,APPIMAGE_RUNTIME=<path>), against a real runtime and theCmdAppexample:--appimage-offsetagrees withAppImagePack.offset;CmdApp's own environment dump shows the depot landing underXDG_DATA_HOME, which is what actually proves the depot decision works — it precompiles into that depot on first launch.test/bundle.jl(MSIX, DMG, Snap) still passes, and the docs build warning-free.Not verified here: mounting. This machine has
/dev/fusebut nofusermount, so the e2e test uses--appimage-extract-and-run, which goes through the same runtime andAppRunbut extracts first. The mount call itself is the one untested step — and since mounting is the whole point for HPC, that gap is worth someone confirming on a machine with FUSE before this is relied on.The FUSE requirement is documented, along with the no-FUSE fallback: the payload is an ordinary squashfs at a known offset, so
AppBundler.AppImagePack.unpack(orunsquashfs -o $(app --appimage-offset)) reads it without the runtime at all.Also included
bundle(::Function, ::MSIX, ::String)had an empty docstring that rendered blank in the manual; it now has one. ACHANGELOG.mdand ajustfileare added, per the conventions I work to — happy to drop either if you would rather they came separately.Assisted-by: AI