Simplify packaging

[septatrix]

There are a few issues with the way virtme-ng-init is currently built, bundled and used. As a result many distros only ship virtme without the rust-based virtme-ng-init component.
The main problem is the submodule/git-fetch usage and the cargo invocation in setup.py. As a result offline compilation is made difficult, HTTP proxies fail to work, and simple control over cargo flags is not possible. These are all things common for package building and enterprise environments.

This situation could be improved by merging both repos, wrapping the init repo in a simple dummy python package and adding a dependency on that, or adding another search path for the init-ng binary outside of the package resources such that it could be build separately. Ideally this would also allow us to completely get rid of the setup.py file.

Please let me know your thoughts about these approaches. I myself have not yet completely explored the options so I am not yet sure which one would best suit packaging but they would all be an improvement over the status quo

[matttbe]

Hello,

Thank you for this bug report!

I already wanted to suggest merging both repos to help with the development. If it can also help with the packaging (if I understood correctly), why not doing that!

To merge the two repo, we can use git subtree add to include the history as well.

[SPYFF]

Right now virtme-ng has a dependency on a statically built busybox. I can imagine something like that on a statically built virtme-ng-init. That could be a separate package, one binary. Similarly to virtiofsd: if not found, fallback to 9p. However I have no idea how hard a static build of virtme-ng-inig would be, probably not that hard since its a relatively small program without complex dependencies.

A more extreme version would be a github CI wokflow, which produces reproducible builds of static virtme-ng-init binaries, and the python part would download & verify that binary if available. Sure, it would be much better to make available the init as a package on all distro, but maintainers might not find enough justification for that, since its only useful virtme-ng (and why not make part of it then, as you proposed).

[arighi]

Yeah, the virtme-ng-init submodule is bugging me. At some point I'd like to merge the code directly into virtme-ng. It'd be nice to keep the history somehow (like re-applying all the commits with some scripting maybe).

Then there's the packaging, which I'm not sure how to solve honestly. Having a separate package for virtme-ng-init seems complicated, since it'll be used only virtme-ng. I like the idea of having a dummy python module, maybe this can help? https://pypi.org/project/setuptools-rust/

[septatrix]

Yeah, the virtme-ng-init submodule is bugging me. At some point I'd like to merge the code directly into virtme-ng. It'd be nice to keep the history somehow (like re-applying all the commits with some scripting maybe).

With git merge --allow-unrelated-histories one can merge two repos while maintaining full history. It is a bit annoying if there are file conflicts but with git-filter-repo it is possible to move one into a subdirectory. I think order matters a bit but it is definitely doable.

Then there's the packaging, which I'm not sure how to solve honestly. Having a separate package for virtme-ng-init seems complicated, since it'll be used only virtme-ng. I like the idea of having a dummy python module, maybe this can help? https://pypi.org/project/setuptools-rust/

For distros I think it makes little difference if they are split or not. The question is likely rather which variant is more convenient for PyPi. It would be possible to have one be an optional feature of the other and install them like such virtme-ng[rust-init] but that would require opt-in. Merging them however would come at the disadvantage of a more annoying opt-out (or at least a more dynamic build process if we want to preserve stuff like BUILD_VIRTME_NG_INIT=1 - in general I am a fan of avoiding such moving parts of possible).

Regarding code changed: If the packages are split the pkg_resources.resource_filename would need to refer to a different package instead of __name__. I think that is already pretty much there is. Alternatively it could just look for the binary in the PATH.

One thing I will note is that distros might to want to invoke cargo/rustc itself instead of having that handled by setuptools to have better control over the build process. If the setuptools plugin properly respects the environment it might also work automatically but I am not sure about that.

[septatrix]

PS: We should also replace pkg_resources (deprecated) with importlib.resources (stdlib) while doing so

[marcosps]

@septatrix virtme-ng-init was merged into virtme-ng. Do you think we can close this issue?

[arighi]

@ribalda now that virtme-ng-init is merged into virtme-ng do you think it's possible to ship the virtme-ng-init binary in the deb? I guess we need to add some cargo dependencies, but in theory it should be possible to build everything offline without fetching any external component.

[SPYFF]

That would be nice; it looks like Debian lags behind two versions from the upstream vng version.

Going a bit further: with pyinstaller, it would be possible to create a fully static binary of vng. This would include the Python interpreter, vng, dependencies, and the vng-init binary, resulting in a single-file executable of approximately 50-60 MB. It could be directly downloaded from the GH Releases page.

If someone is lucky enough to daily drive a distribution with cutting-edge packages, such a release wouldn't offer much benefit. However, for those who prefer using a stable distribution (Ubuntu LTS and so on) but still want to use the latest vng, a "static binary" release would greatly simplify accessibility. Also, systems created with Yocto or similar, installing build dependencies would require re-generate the whole distro.

[matttbe]

do you think it's possible to ship the virtme-ng-init binary in the deb?

@arighi I didn't know this binary was not included in these packages. Because it is used by default (at least by devs), should we add a warning about that in the README?

Going a bit further: with pyinstaller, it would be possible to create a fully static binary of vng. This would include the Python interpreter, vng, dependencies, and the vng-init binary, resulting in a single-file executable of approximately 50-60 MB. It could be directly downloaded from the GH Releases page.

Is the installation via pip no enough, and maybe even better that a static binary?

[arighi]

Going a bit further: with pyinstaller, it would be possible to create a fully static binary of vng. This would include the Python interpreter, vng, dependencies, and the vng-init binary, resulting in a single-file executable of approximately 50-60 MB. It could be directly downloaded from the GH Releases page.

Hm, but we also depend on qemu... if you're suggesting to go with a containerized vng package (flatpack or similar) it can be quite big. But I agree that it might be useful for old distros.

[arighi]

@arighi I didn't know this binary was not included in these packages. Because it is used by default (at least by devs), should we add a warning about that in the README?

Yeah, I'm a bit conflicted on that, it's a distro choice to not ship the virtme-ng-init binary. Arch for example is shipping virtme-ng-init in the virtme-ng package. But I agree that it could be helpful to mention that in the README. Maybe in the troubleshooting section, something like "if you experience poor filesystem performance check if your distro is shipping virtme-ng-init" (of course rephrased in a more detailed and comprehensible way).

[ribalda]

@arighi I will try to look into vng this Friday. I will probably try without virtme-ng-init first (to sync up with upstream asap) and then try adding -init.

Sorry for the delay, but it has been pretty busy months in this side of the cable

[arighi]

@arighi I will try to look into vng this Friday. I will probably try without virtme-ng-init first (to sync up with upstream asap) and then try adding -init.

Looks like a good plan!

Sorry for the delay, but it has been pretty busy months in this side of the cable

No worries, take your time and thanks tons for your help!

[SPYFF]

Is the installation via pip no enough, and maybe even better that a static binary?

Yes, that's better. I quickly tried pipx install virtme-ng on some PCs around me (Ubuntu 24.04, 25.04, Debian Sid), and all installations were successful. With that, I admit that a static binary would be overkill, or at least the benefits of it are particular.

Hm, but we also depend on qemu... if you're suggesting to go with a containerized vng package (flatpack or similar) it can be quite big. But I agree that it might be useful for old distros.

To be honest, I totally forgot about QEMU :-D. PyInstaller only deals with Python and dependencies, so yes, probably vng would require an AppImage or Flatpak format. Given that pipx works very well, I think it's worth putting effort into it.