-
-
Notifications
You must be signed in to change notification settings - Fork 5
Build packages without installing them #214
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
| .to_str() | ||
| .ok_or(anyhow!("Failed to get string from filename"))? | ||
| .to_string(); | ||
| if !archive_name.ends_with(".pkg.tar.zst") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This relies on the extension being .zst (currently the case, but could potentially change with a different makepkg.conf). We could use a fancier check, possibly including the architecture... Or leave it as-is for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Compression can also be entirely disabled. Looking at the end of makepkg.conf you can see it implies potential support for .gz, .bz2, .lrz, .lzo, .z, .lz4, .lz. Not sure we have to support all of them but we could just accept .tar*.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, yes true.
I was confused and meant the extension of the repo database (repo.db.tar.gz & repo.files.tar.gz), sry.
Yeah, probably a wildcard is the easiest solution for now. In the future we could make this configureable.
| let build_flags = self.package_model.build_flags.get()?.split(";").join(" "); | ||
| // create new docker container for current build | ||
| let build_dir_base = "/var/cache/makepkg/pkg"; | ||
| let build_dir_base = "/build"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The actual build location doesn't really matter, but /var/cache/makepkg/pkg is a bit obscure when we just want some working directory. It's pretty common to use top-level directories like /build, /app, ... But we can obviously use something else if you prefer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.
| _name: String, | ||
| _build_dir_base: &str, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just temporary while we decide if we want this function at all in this state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PKGDEST is not required since we use
paru -G {name}
paru {build_flags} {name}
To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, we don't rely on any specific PKGDEST, we just look at the resulting archives in the current directory (like makepkg would do).
| let init_cmd = " | ||
| sudo pacman-key --init | ||
| sudo pacman-key --populate archlinux | ||
| paru -Syu | ||
| "; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Arbitrary split between a constant "initialization" part, and the build part that depends on the package. I could merge the two in a single one if desired.
|
|
||
| let build_id = self.build_model.id.get()?; | ||
| let container_name = format!("aurcache_build_{filtered_name}_{build_id}"); | ||
| let auto_remove = !cfg!(debug_assertions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Negating a constant results in the weird-looking !cfg!. Maybe it'd be better to do cfg!(not(debug_assertions)) here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, feel free to edit.
|
Unfortunately, while it can build packages just like before, it still fails with conflicting packages. I think it's an upstream bug, and I filed Morganamilo/paru#1376. |
|
Some of your questions: So the per-package arguments were required for packages that require pgp signing: I use Rustrover for backend and Intellij for frontend dev and use the podman/docker socket from my system to build packages. When you start the backend on your pc and start the frontend it should just work to try stuff out. (Remember, you can build flutter for native linux, so then it behaves just like a native app and works great.) When you need to try out docker specific stuff you can use the docker compose files in the root of the project |
|
The main idea why packages are built in docker containers is because of isolation (and concurrency), after a build the container gets deleted automatically. So there can't be any leftover build artifacts which might clutter the main aurcache container. Your idea, sounds good, but we have to be careful to delete all the leftover build artifacts. Since there is just this one folder it should be fine, but just as a note. |
|
I'll review this tomorrow too. |
Ah looks like Though it revealed a problem with the current PR: it assumes that This brings up a question: how to support split packages? I see a few options:
I'm still trying to understand the current situation. It pulls everything from makepkg's
I think it would be best to either entirely ignore the dependencies when registering archive files, or, better, to automatically properly register them as separate packages in AURCache. This second option is the best long-term but includes a large work:
In the meantime, ignoring dependency archives might be the best short-term way to avoid overlapping packages. Sorry for the wall of text - I realize this investigation sent me in all directions, and this would probably deserve dedicated issues. |
705f01d to
c3e0e59
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rest looks fine. Thanks again!
You're right we can bake the makepkg.conf into the image now. I'd just add it to the docker subfolder and ADD it in the Dockerfile.
I'd leave the custom build flags in for now until we are sure it is really not required.
The current situation is:
When a user builds a package which depends on other AUR packages those are getting built too and end up in the output folder. Aurcache picks them up and adds them to the repo. Since pacman doesn't allow different versions of the same package in one repo (and archlinux isn't really built for that) I thought this should be fine if they are just in the repo. All the archives which belong to one package are registered in a db table. (One could build a fancy tree out of that maybe) When a package is deleted it is checked if no other package requires this dependency package if not, it gets deleted too, if yes its kept.
I don't think an AUR package is able to request a specific version for a aur dependency, right?
Split packages are a pain. I'm aware of them but I didn't really care/had time to tackle them properly... But I guess the easiest would be to add just the base package, as you prefered. When user tries to add a split package, maybe give him a warning and add just the base package. Further steps might be to add a fancy list in the package overview of which split pakcage this pkgbuild consists.
Edit:
text wall is fine, thanks for your efforts!
| .to_str() | ||
| .ok_or(anyhow!("Failed to get string from filename"))? | ||
| .to_string(); | ||
| if !archive_name.ends_with(".pkg.tar.zst") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...
| let build_flags = self.package_model.build_flags.get()?.split(";").join(" "); | ||
| // create new docker container for current build | ||
| let build_dir_base = "/var/cache/makepkg/pkg"; | ||
| let build_dir_base = "/build"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.
| let init_cmd = " | ||
| sudo pacman-key --init | ||
| sudo pacman-key --populate archlinux | ||
| paru -Syu --noconfirm |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not required anymore since builder image gets auto built once a week.
|
|
||
| let build_id = self.build_model.id.get()?; | ||
| let container_name = format!("aurcache_build_{filtered_name}_{build_id}"); | ||
| let auto_remove = !cfg!(debug_assertions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, feel free to edit.
| _name: String, | ||
| _build_dir_base: &str, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PKGDEST is not required since we use
paru -G {name}
paru {build_flags} {name}
To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?
It's possible, but rarely used and rarely a good idea. In a PKGBUILD there's no difference between dependencies in the aur or in a repo (and indeed, a AURCache or similar repo would blur the distinction anyway). It's possible to add version constraints, mostly to prevent updating or installing the wrong version. It can't be used to fetch the newest supported version in a semver fashion.
Ah I didn't realize that - in that case overlapping packages are not as terrible. They just likely cause duplicated compute to build from multiple places.
One problem is that the "base package" is not always the name of an actual package. For example, here is the freeipa base package. Note the url includes pkgbase, not package. There is no But in any case it shouldn't be blocking for this PR, and could come later as an improved support for split packages.
This is the part that is changing in this PR: currently, I only pick up what is defined by this PKGBUILD, not any dependency that got built along the way. Though I suppose I could bring back |
c3e0e59 to
252eb3e
Compare
Ok, good to know! But I think we can ignore this edge case for now and leave it as is.
Yeah. But I think the duplicate compute is a price we easily want pay instead of all the dependencies are auto-added to the package list.
I just noticed, the link to the aur repo is broken within split packages... I'll create a issue for split packages and we handle this in another PR.
In my opinion for usability it would be better as it is now. I think a user doesn't really care about the packages dependencies. They just want to add the package they want to build, install it from the aurcache repo, and it should just work out of the box. |
|
In the current state, this PR now preserves the same behaviour as the current master branch (includes the archives of all dependencies and other split packages). It still relies on a custom makepkg.conf, but the path doesn't actually depend on the package name, so in the future we may be able to move this to the builder base image. The main issues remaining are:
An alternative I haven't investigated yet is the chroot-build feature of paru. It looks like it creates a chroot, build the package there, then add it to some "local repo" rather than installing it, which sounds very similar to what we want here. We could also consider replacing docker entirely with the chroot build, but it'd need more investigation to confirm we can run parallel chroot builds (we don't want to lose parallel builds). Also it may be a ton of changes, might not be worth it. Keeping docker gets us closer to possibly one day running the builder on kubernetes for a proper distributed build farm. |
|
Great!
Yes, we should create a new db migration. For this just create a new migration file (for example here and add it to the mod.rs. In every build_flags field you replace the -Syu with rust code...
Thanks! But I think just building and not installing the package should be a cleaner method as before. 👍
I've already seen that chroot stuff and tried it (just very quickly) out like one year ago. I couldn't get it to work then and thats why I sticked with the way it works now. Important thing is that we don't have to give the spawned docker containers more permissions than now for getting chroot to work.
Disagree with that, docker gives us major advantages in isolation. Eg. limiting cpu cores, memory, networking, multiple parallel builds... Most of this stuff would be much more pain without docker. |
|
Hey, @gyscos, can I help you somehow with this pr? Should we wait until the paru issue is resolved? |
|
Sorry, I've been busy these couple weeks. I may have a bit more time next week to come back to that. |
All right, no stress. Feel free to ping me anytime! |
|
I feel like I am running into issues for some of my packages due to paru trying to install the packages. This PR seems to be the right spot to report, as it seems to solve the issue. I maintain a meta-package repository, that is a fork of manjaro-meta. Build logThis issue arise from paru trying to install the packages and might be solved by this PR. |
|
Another note: I just figured, that I was already using My first try to solve this: Separating the conflicting package to another git repo However, building all my own meta-packages still failed. This time, paru somehow installed The make dependency of pkgname=(
'my-gstreamer'
'my-pipewire'
'my-pulse'
)to pkgname=(
'my-pipewire'
'my-gstreamer'
'my-pulse'
)solved this for me (Reason: However: Since I am assembling several meta-packages here that might have unsolveable cross-depencies in the future (no reordering will help here), I sinply turned off all dependency checks. First, paru's internal dependency checks must be forwarded to pacman via the With all of this, I was able to build my meta-package. You can find a strapped down example meta package, that produces the errors here: https://github.com/systemofapwne/pkg-my-meta-failing.git |
This is a draft PR/request for comment to switch from installing packages with
paru -Sto simply building them withparu -B.Note that this is currently entirely untested. I'm not entirely sure how to prepare an image for testing :^S
Here is the main idea: instead of installing a package in one step with
paru -S, download the PKGBUILD and sources withparu -G, then build that pkgbuild withparu -B, and finally move the resulting packages to the destination folder.To avoid moving/copying the package archives across filesystem/mount-point boundaries, I build the entire package in the mounted build dir, but only select files that look like archives from there (instead of picking everything).
Fixes #206
A few questions arise from these changes:
PKGDESTin themakepkg.conf. This now only contains the-j$(nproc)build flag. Should this instead be in the base builder image, removing the need for amakepkg.confentirely?-B --noconfirm --noprogressbar --color never. Any custom flags instead add a potential failure mode. Maybe we could remove custom build flags entirely? Do you have examples of how they are currently used?-Syu {package_name}, which was updating every package installed on the build node.paru -Bby itself doesn't change anything installed on the build node, so it doesn't update anything. I added an explicitparu -Syubefore that to match the current behaviour, but it's an open question: do we need to update everything everytime?