Use virtual filesystem (FUSE?) to bypass binfmt integration on execution

[TheAssassin]

I think it might be a good idea to implement an AILFS that will allow for bypassing the binfmt setup on execution.

Background

AppImageLauncher registers in the system using any available method (currently binfmt and MIME type) to catch all AppImage executions and perform actions before/during AppImage startup. The only action right now is desktop integration, but we already think about more features (e.g., security related questions, like checking signatures on startup or enforcing a sandbox on all AppImages, but we could also run wrapper scripts on distros like NixOS).

The problem is however that we need to bypass this integration to actually launch the AppImages. We cannot simply exec() the AppImages: that'd result in an infinite execution loop, as the kernel's binfmt would all AppImageLauncher again.

We can also not just invoke the linker /lib/ld-linux.so.2 (resp. lib<ARCH> for other supported ISAs), as the current and previous revision of the AppImage specification (types 1 and 2) violate the ELF specification, as the magic AppImage bytes are in regions that have special meanings. Some implementations of the linker don't like these bytes (they interpret them (normally optional), see an invalid value and refuse execution). These implementations can be found in most versions of qemu, some Linux distros, newer versions of Docker, etc. We aren't quite sure why the execution works "normally", all we can tell is we cannot call that linker directly on the AppImages, as we normally could with regular ELF files.

Therefore, the current solution for type 2 is to ship a custom, patched (as in: no magic bytes) runtime, that tends to be outdated and might lack features needed by the AppImages. Thoughts arised whether to extract the runtime of every AppImage, patching it and using it to run the AppImages, but that'd be a lot of file write overhead.

Even worse, the support for type 1 AppImages is implemented by copying these AppImages entirely, patching out the magic bytes and running them that way, cleaning up the temporary files after termination. This had to be done due to lack of a TARGET_APPIMAGE like feature in the type 1 runtime, and lack of interest in implementing that feature in the type 1 runtime to ship that, too.

Proposed solution

A virtual AppImageLauncher filesystem could solve these issues most elegantly, and for every AppImage type, even future ones. The filesystem would provide read-only access to AppImages, delivering the file contents transparently by reading them from the real files just-in-time. However, the magic byte areas could be nulled on the fly, without the need for any filesystem operation.

I've been thinking quite a while how "virtual files" could be implemented so that they'd act as transparent wrappers for the underlying real files but lacking the offending bits, but didn't find any easier solution. Today, I came up with the idea of implementing this in form of a FUSE filesystem. This filesystem shouldn't be too hard to implement, as it doesn't have to do anything fancy, only deliver a fake first few blocks of the file on read. We could simply copy the files on it to make it open a file descriptor to the files (to prevent them from getting lost and persist them even on rm operations on the real files), and close that file descriptor if all open FDs have been closed and the file has been rmed by some application.

It'd be technically interesting to me to write such a filesystem, as I never had a chance to get into FUSE, and it doesn't seem to be too complex. However, it'd elegantly solve a long time issue, and would allow us to drop these custom patched runtimes.

CC @azubieta @probonopd

[probonopd]

Feels like we are piling up way more complexity than needed here.

An AppImage is supposed to be self-executable (without the need to extract anything). In fact, an AppImage could implement an arbitrary filesystem today and still work. It would technically be a Type 0 image format, but it would work.

Wouldn't things be much easier if we used MIME types instead of binfmt to have AppImages opened with AppImageLauncher?

[azubieta]

binfmt has the advantage that we can catch all exec calls on AppImage. But this also becomes a drawback because we need all sort of hacks to avoid loop executions. And also tempt us to use it more and more with the risk of making AppImage depending on it. Which is one of the strongest points of the solution: not requiring anything to run an AppImage. At the early beginning of the development of this solution, I was against using binfmt, but I wasn't the one making decisions.

I guess that @TheAssassin has more arguments on it.

[TheAssassin]

@probonopd there is no "either/or", as MIME type doesn't cover all cases AppImageLauncher is interested in. You need to combine both.

[probonopd]

Which cases are not covered by MIME types? Can't we live with those cases not being caught?

But this also becomes a drawback because we need all sort of hacks to avoid loop executions.

Exactly...

I think it's always good to experiment and learn from the experiments, in this case we are learning that going the binfmt route has a lot of strings attached.

[TheAssassin]

No, we cannot live without. Period.

[probonopd]

Can you give some examples?

[TheAssassin]

#96 (comment), section background.

[probonopd]

That section describes the issues that binfmt brings, but I don't see a description of which cases binfmt does cover which MIME types do not cover.

I can think of one: launching an AppImage by executing it via the command line. Are there other cases?

[TheAssassin]

Primary use case: launching a non-integrated AppImage that is already executable, which is a must-have for this project.

Future use cases involve:

The only action right now is desktop integration, but we already think about more features (e.g., security related questions, like checking signatures on startup or enforcing a sandbox on all AppImages, but we could also run wrapper scripts on distros like NixOS).

The security related aspects have been added to the long term plan already.

[probonopd]

Primary use case: launching a non-integrated AppImage that is already executable, which is a must-have for this project.

What would make the AppImage "already executable" (other from the user, in which case it is deliberate)?

[TheAssassin]

https://github.com/TheAssassin/AppImageLauncher/wiki/Idea

[azubieta]

From my point of view, it's not required intercept every AppImage invocation. Just those that don't have the executable bit. Therefore only the MIME-TYPE integration will be required.

@TheAssassin and I already had this talk, so I'm just going to let my ideas below for the record (And because I usually forget it).

As the listed advantages of binfm_misc integration are the interception of exec on AppImages by the following means:

• Terminal (ignored by default in appimage_launcher_conde)
• desktop environments' launchers (to be in the launcher an AppImage must be integrated, integratedAppImages are also ignored by AIL)
• double click in file manager (which is the express intention of the user to execute an AppImage, therefore we should not get in the way)

[TheAssassin]

A core difference is our understanding of "integration". An integration is not "making executable", that's in fact "making executable". Integration refers to integrating the AppImage in the menu, which is not possible "on first run" for already executable AppImages (which are shipped by some people in .tar* archives for some reasons). You could only "monitor" the entire filesystem for files, and that doesn't scale, as we learned.

Furthermore, "integration" in terms of AppImageLauncher means "moving into a central directory". This avoids having AppImages in many directories, where they're hard to find. To accomplish this reliably, the executable bit must not get into one's way (like, "it's executable, so it's run directly").

Therefore, both binfmt and MIME type integration are combined to reliably intercept the first invocation of all new AppImages. Of course, there is some overhead, as binfmt intercepts all future invocations. However, as we now control the invocation, we can just silently move on to executing the AppImage normally. Also, AppImageLauncher doesn't spawn a QApp until it is decided whether an AppImage has been integrated or not to reduce the overhead (thanks @azubieta for implementing that).

As said, being "interpreter" enables us to perform more processes before actually invoking the AppImages. Security checks might be applied for instance (as in e.g., "is the signature still valid", etc.) in the future. See above for more information.

[TheAssassin]

Demo available here: https://github.com/TheAssassin/AppImageLauncherFS

Will try to build AppImages later for testing.

[TheAssassin]

Demo AppImages available for x86_64: https://github.com/TheAssassin/AppImageLauncherFS/releases/tag/continuous

[probonopd]

How do we use it?

[TheAssassin]

You "just" run the AppImage. It will be available immediately in /run/user/<uid>/appimagelauncherfs. For now, all AppImages in ~/Applications are available immediately. Additional paths can be registered by echo <path> > /run/user/<uid>/appimagelauncherfs/register. The files map can be seen using cat /run/user/<uid>/appimagelauncherfs/map.

[probonopd]

me@host:~$ cat /run/user/999/appimagelauncherfs/map is always empty for me, even when I copy AppImages to my (manually created) ~/Applications.

me@host:~$ '/home/me/Downloads/AppImageLauncherFS-3-44f14b5-x86_64.AppImage' 
FUSE version: 2.9
fusermount: entry for /run/user/999/appimagelauncherfs not found in /etc/mtab
rmdir: failed to remove '/run/user/999/appimagelauncherfs/': No such file or directory
mountpoint: /run/user/999/appimagelauncherfs/
Starting FUSE filesystem

Is there a way to increase verbosity?

[TheAssassin]

@probonopd please see #96 (comment). The project has been started just a couple of days ago. There's no inotify watching or so, manual "registration" is the most reliable way.

export DEBUG=1 increases verbosity by switching on several debug messages (mostly FUSE).

[probonopd]

OK, it seems like I have not entirely understood yet a) what this is supposed to do and b) which subset of this is already implemented. Looking forward to an informative README.md ;-)

[TheAssassin]

Feel free to pass by on IRC for a live demo.