Skip to content

Conversation

@gyscos
Copy link
Contributor

@gyscos gyscos commented Jun 28, 2025

This is a draft PR/request for comment to switch from installing packages with paru -S to simply building them with paru -B.

Note that this is currently entirely untested. I'm not entirely sure how to prepare an image for testing :^S

Here is the main idea: instead of installing a package in one step with paru -S, download the PKGBUILD and sources with paru -G, then build that pkgbuild with paru -B, and finally move the resulting packages to the destination folder.

To avoid moving/copying the package archives across filesystem/mount-point boundaries, I build the entire package in the mounted build dir, but only select files that look like archives from there (instead of picking everything).

Fixes #206

A few questions arise from these changes:

  • We no longer need a custom PKGDEST in the makepkg.conf. This now only contains the -j$(nproc) build flag. Should this instead be in the base builder image, removing the need for a makepkg.conf entirely?
  • Building a package has much fewer turning knobs than installing a package. As such, there's really nothing that could be added to the -B --noconfirm --noprogressbar --color never. Any custom flags instead add a potential failure mode. Maybe we could remove custom build flags entirely? Do you have examples of how they are currently used?
  • The previous build flags included -Syu {package_name}, which was updating every package installed on the build node. paru -B by itself doesn't change anything installed on the build node, so it doesn't update anything. I added an explicit paru -Syu before that to match the current behaviour, but it's an open question: do we need to update everything everytime?

.to_str()
.ok_or(anyhow!("Failed to get string from filename"))?
.to_string();
if !archive_name.ends_with(".pkg.tar.zst") {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relies on the extension being .zst (currently the case, but could potentially change with a different makepkg.conf). We could use a fancier check, possibly including the architecture... Or leave it as-is for now.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compression can also be entirely disabled. Looking at the end of makepkg.conf you can see it implies potential support for .gz, .bz2, .lrz, .lzo, .z, .lz4, .lz. Not sure we have to support all of them but we could just accept .tar*.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes true.
I was confused and meant the extension of the repo database (repo.db.tar.gz & repo.files.tar.gz), sry.
Yeah, probably a wildcard is the easiest solution for now. In the future we could make this configureable.

let build_flags = self.package_model.build_flags.get()?.split(";").join(" ");
// create new docker container for current build
let build_dir_base = "/var/cache/makepkg/pkg";
let build_dir_base = "/build";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual build location doesn't really matter, but /var/cache/makepkg/pkg is a bit obscure when we just want some working directory. It's pretty common to use top-level directories like /build, /app, ... But we can obviously use something else if you prefer.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.

Comment on lines 2 to 3
_name: String,
_build_dir_base: &str,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just temporary while we decide if we want this function at all in this state.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PKGDEST is not required since we use

paru -G {name}
paru {build_flags} {name}

To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, we don't rely on any specific PKGDEST, we just look at the resulting archives in the current directory (like makepkg would do).

Comment on lines 183 to 187
let init_cmd = "
sudo pacman-key --init
sudo pacman-key --populate archlinux
paru -Syu
";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arbitrary split between a constant "initialization" part, and the build part that depends on the package. I could merge the two in a single one if desired.


let build_id = self.build_model.id.get()?;
let container_name = format!("aurcache_build_{filtered_name}_{build_id}");
let auto_remove = !cfg!(debug_assertions);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Negating a constant results in the weird-looking !cfg!. Maybe it'd be better to do cfg!(not(debug_assertions)) here.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, feel free to edit.

@gyscos
Copy link
Contributor Author

gyscos commented Jun 29, 2025

Unfortunately, while it can build packages just like before, it still fails with conflicting packages. I think it's an upstream bug, and I filed Morganamilo/paru#1376.

@Lukas-Heiligenbrunner
Copy link
Owner

Some of your questions:

So the per-package arguments were required for packages that require pgp signing:
#65
#64
This helped to make package builds at least a bit more configureable and more pkgs should be buildable

I use Rustrover for backend and Intellij for frontend dev and use the podman/docker socket from my system to build packages. When you start the backend on your pc and start the frontend it should just work to try stuff out. (Remember, you can build flutter for native linux, so then it behaves just like a native app and works great.) When you need to try out docker specific stuff you can use the docker compose files in the root of the project docker-compose.dindmode.dev.yaml and docker-compose.hostmode.dev.yaml depending on what build mode you want to try out.
You can build the buillder image locally with sth like: docker build -f docker/builder.Dockerfile .

@Lukas-Heiligenbrunner
Copy link
Owner

The main idea why packages are built in docker containers is because of isolation (and concurrency), after a build the container gets deleted automatically. So there can't be any leftover build artifacts which might clutter the main aurcache container. Your idea, sounds good, but we have to be careful to delete all the leftover build artifacts. Since there is just this one folder it should be fine, but just as a note.

@Lukas-Heiligenbrunner
Copy link
Owner

I'll review this tomorrow too.
And I don't think we need to -Syu update before each build since the image is auto-built once a week, so this should be fine.

@gyscos
Copy link
Contributor Author

gyscos commented Jun 29, 2025

So the per-package arguments were required for packages that require pgp signing:

Ah looks like --mflags --skippgpcheck is still available for paru -B. Though maybe something changed, but I'm now able to compile freeipa without this flag.

Though it revealed a problem with the current PR: it assumes that paru -G {package_name} will result in a {package_name} folder created; but the folder created is that of the base package. In the case of split packages (like freeipa-client-common), it can differ from the requested package name.

This brings up a question: how to support split packages? I see a few options:

  • Should aurcache only work with base packages (like freeipa), which will build (among others) the freeipa-client-common package archive?
    This would be my personal preference. We could have the UI indicate that when we add/install a split package, it will actually add the parent package for minimal surprise. Might also be nice to have
  • If instead we want to accept adding "sub-packages" like freeipa-client-common, then should we build the base package but only keep the requested package (and discard the rest)? One downside is that if we want all splits of a base package, we'd now need to build the entire base independently in each of the split packages.
  • If instead we keep everything we built, we essentially have a base package named like the split-package. It's not much better than the first "base packages only" solution, and instead risks having overlapping packages if we add two packages that share the same base. Overlapping packages sounds like something we want to avoid for obvious reasons.
    This is a bad option all around, just here for completeness.

I'm still trying to understand the current situation. It pulls everything from makepkg's PKGDEST. It looks like any dependency that's an AUR package itself (and needs to be built) will end up there, and gets registered as part of the main package target list of files. This means:

  • Any AUR dependency is now available for users, so they don't have to build anything themselves.
  • If different packages depend on the same AUR package, they will all try to build and include that dependency, resulting in overlapping packages or version mismatch/inconsistencies.

I think it would be best to either entirely ignore the dependencies when registering archive files, or, better, to automatically properly register them as separate packages in AURCache. This second option is the best long-term but includes a large work:

  • Compute the dependency tree when adding the package in AURCache itself, not in the builder.
  • Build the dependency package before building the downstream target (need to synchronize and order the builds).
  • Inject AURCache's repo in the builder's mirrorlist (I thought this was mentioned but I don't know if this is currently done?)

In the meantime, ignoring dependency archives might be the best short-term way to avoid overlapping packages.

Sorry for the wall of text - I realize this investigation sent me in all directions, and this would probably deserve dedicated issues.

@gyscos gyscos force-pushed the build_not_install branch 2 times, most recently from 705f01d to c3e0e59 Compare June 30, 2025 18:23
Copy link
Owner

@Lukas-Heiligenbrunner Lukas-Heiligenbrunner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest looks fine. Thanks again!

You're right we can bake the makepkg.conf into the image now. I'd just add it to the docker subfolder and ADD it in the Dockerfile.
I'd leave the custom build flags in for now until we are sure it is really not required.

The current situation is:
When a user builds a package which depends on other AUR packages those are getting built too and end up in the output folder. Aurcache picks them up and adds them to the repo. Since pacman doesn't allow different versions of the same package in one repo (and archlinux isn't really built for that) I thought this should be fine if they are just in the repo. All the archives which belong to one package are registered in a db table. (One could build a fancy tree out of that maybe) When a package is deleted it is checked if no other package requires this dependency package if not, it gets deleted too, if yes its kept.
I don't think an AUR package is able to request a specific version for a aur dependency, right?

Split packages are a pain. I'm aware of them but I didn't really care/had time to tackle them properly... But I guess the easiest would be to add just the base package, as you prefered. When user tries to add a split package, maybe give him a warning and add just the base package. Further steps might be to add a fancy list in the package overview of which split pakcage this pkgbuild consists.

Edit:
text wall is fine, thanks for your efforts!

.to_str()
.ok_or(anyhow!("Failed to get string from filename"))?
.to_string();
if !archive_name.ends_with(".pkg.tar.zst") {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...

let build_flags = self.package_model.build_flags.get()?.split(";").join(" ");
// create new docker container for current build
let build_dir_base = "/var/cache/makepkg/pkg";
let build_dir_base = "/build";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.

let init_cmd = "
sudo pacman-key --init
sudo pacman-key --populate archlinux
paru -Syu --noconfirm

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not required anymore since builder image gets auto built once a week.


let build_id = self.build_model.id.get()?;
let container_name = format!("aurcache_build_{filtered_name}_{build_id}");
let auto_remove = !cfg!(debug_assertions);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, feel free to edit.

Comment on lines 2 to 3
_name: String,
_build_dir_base: &str,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PKGDEST is not required since we use

paru -G {name}
paru {build_flags} {name}

To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?

@gyscos
Copy link
Contributor Author

gyscos commented Jun 30, 2025

I don't think an AUR package is able to request a specific version for a aur dependency, right?

It's possible, but rarely used and rarely a good idea. In a PKGBUILD there's no difference between dependencies in the aur or in a repo (and indeed, a AURCache or similar repo would blur the distinction anyway). It's possible to add version constraints, mostly to prevent updating or installing the wrong version. It can't be used to fetch the newest supported version in a semver fashion.

When a package is deleted it is checked if no other package requires this dependency package if not, it gets deleted too, if yes its kept.

Ah I didn't realize that - in that case overlapping packages are not as terrible. They just likely cause duplicated compute to build from multiple places.

But I guess the easiest would be to add just the base package, as you prefered

One problem is that the "base package" is not always the name of an actual package. For example, here is the freeipa base package. Note the url includes pkgbase, not package. There is no freeipa package, only freeipa-client-common (and a couple others). So currently freeipa doesn't even appear in the dropdown list. I didn't check yet but there may be an option to look for these as well, and mark them specially in the UI if a base package contains more than 1 package.

But in any case it shouldn't be blocking for this PR, and could come later as an improved support for split packages.

When a user builds a package which depends on other AUR packages those are getting built too and end up in the output folder. Aurcache picks them up and adds them to the repo.

This is the part that is changing in this PR: currently, I only pick up what is defined by this PKGBUILD, not any dependency that got built along the way. Though I suppose I could bring back PKGDEST and keep both - all built dependencies, and the built-but-not-installed target package.

@gyscos gyscos force-pushed the build_not_install branch from c3e0e59 to 252eb3e Compare June 30, 2025 20:52
@Lukas-Heiligenbrunner
Copy link
Owner

It's possible, but rarely used and rarely a good idea. In a PKGBUILD there's no difference between dependencies in the aur or in a repo (and indeed, a AURCache or similar repo would blur the distinction anyway). It's possible to add version constraints, mostly to prevent updating or installing the wrong version. It can't be used to fetch the newest supported version in a semver fashion.

Ok, good to know! But I think we can ignore this edge case for now and leave it as is.

Ah I didn't realize that - in that case overlapping packages are not as terrible. They just likely cause duplicated compute to build from multiple places.

Yeah. But I think the duplicate compute is a price we easily want pay instead of all the dependencies are auto-added to the package list.

One problem is that the "base package" is not always the name of an actual package. For example, here is the freeipa base package. Note the url includes pkgbase, not package. There is no freeipa package, only freeipa-client-common (and a couple others). So currently freeipa doesn't even appear in the dropdown list. I didn't check yet but there may be an option to look for these as well, and mark them specially in the UI if a base package contains more than 1 package.
But in any case it shouldn't be blocking for this PR, and could come later as an improved support for split packages.

I just noticed, the link to the aur repo is broken within split packages...
The current behaviour when building a split package is building and addint all the subpackages Here example log:
log.txt
Does this PR break this behaviour?

I'll create a issue for split packages and we handle this in another PR.

that got built along the way. Though I suppose I could bring back PKGDEST and keep both - all built dependencies, and the built-but-not-installed target package.

In my opinion for usability it would be better as it is now. I think a user doesn't really care about the packages dependencies. They just want to add the package they want to build, install it from the aurcache repo, and it should just work out of the box.

@gyscos
Copy link
Contributor Author

gyscos commented Jul 3, 2025

In the current state, this PR now preserves the same behaviour as the current master branch (includes the archives of all dependencies and other split packages). It still relies on a custom makepkg.conf, but the path doesn't actually depend on the package name, so in the future we may be able to move this to the builder base image.

The main issues remaining are:

An alternative I haven't investigated yet is the chroot-build feature of paru. It looks like it creates a chroot, build the package there, then add it to some "local repo" rather than installing it, which sounds very similar to what we want here.
We could run the chroot-build in the docker container, as an extra layer of isolation (and with the least changes from the current situation).
EDIT: looks like it won't be trivial to use chroot inside docker. Even when running privileged, some stuff aren't working.

We could also consider replacing docker entirely with the chroot build, but it'd need more investigation to confirm we can run parallel chroot builds (we don't want to lose parallel builds). Also it may be a ton of changes, might not be worth it. Keeping docker gets us closer to possibly one day running the builder on kubernetes for a proper distributed build farm.

@Lukas-Heiligenbrunner
Copy link
Owner

Lukas-Heiligenbrunner commented Jul 4, 2025

Great!

All existing build flags will need to be updated in a migration, since all the -Syu need to become -B. In addition, any additional custom flag like --skippgpcheck might need to become --mflags --skippgpcheck. This sounds like a painful migration, but it looks like at least we can write the migration in rust (and not just in SQL)?

Yes, we should create a new db migration. For this just create a new migration file (for example here and add it to the mod.rs. In every build_flags field you replace the -Syu with rust code...
I think we can ignore the --skippgpcheck since not so many people should use it. We just add a breaking change into the next releases changelogs.

Until paru -B --noconfirm complains about conflicts for the package it's not installing Morganamilo/paru#1376 is fixed there's really no benefit to this PR yet.

Thanks! But I think just building and not installing the package should be a cleaner method as before. 👍

An alternative I haven't investigated yet is the chroot-build feature of paru.

I've already seen that chroot stuff and tried it (just very quickly) out like one year ago. I couldn't get it to work then and thats why I sticked with the way it works now. Important thing is that we don't have to give the spawned docker containers more permissions than now for getting chroot to work.
Just saw your edit, yeah, I'm not sure if chrooting would give as major benefits anyway.

We could also consider replacing docker entirely with the chroot build.

Disagree with that, docker gives us major advantages in isolation. Eg. limiting cpu cores, memory, networking, multiple parallel builds... Most of this stuff would be much more pain without docker.
And yes, kubernets would be cool one day. :)

@Lukas-Heiligenbrunner
Copy link
Owner

Hey, @gyscos, can I help you somehow with this pr? Should we wait until the paru issue is resolved?

@gyscos
Copy link
Contributor Author

gyscos commented Jul 24, 2025

Sorry, I've been busy these couple weeks. I may have a bit more time next week to come back to that.

@Lukas-Heiligenbrunner
Copy link
Owner

Sorry, I've been busy these couple weeks. I may have a bit more time next week to come back to that.

All right, no stress. Feel free to ping me anytime!

@Lukas-Heiligenbrunner Lukas-Heiligenbrunner marked this pull request as draft November 5, 2025 18:24
@systemofapwne
Copy link

I feel like I am running into issues for some of my packages due to paru trying to install the packages. This PR seems to be the right spot to report, as it seems to solve the issue.

I maintain a meta-package repository, that is a fork of manjaro-meta.
It fails to install due to mutual exclusive packages like manjaro-pipewire conflicts with manjaro-pulse and vice versa.

Build log
Pulling image: ghcr.io/lukas-heiligenbrunner/aurcache-builder:latest:: Synchronizing package databases...
 core downloading...
 extra downloading...
 multilib downloading...
==> Appending keys from archlinux.gpg...

gpg: next trustdb check due at 2026-01-30
:: Generating .SRCINFO for ./tmp...
:: Generating .SRCINFO for ./tmp...
:: Resolving dependencies...

:: There are 2 providers available for resolvconf:
:: Repository core:
    1) systemd-resolvconf  
:: Repository extra:
    2) openresolv  
Enter a number (default=1): 
:: There are 2 providers available for jack:
:: Repository extra:
    1) jack2  2) pipewire-jack  
Enter a number (default=1): :: Calculating conflicts...
:: Calculating inner conflicts...

:: Inner conflicts found:
    manjaro-pipewire: manjaro-pulse (manjaro-pipewire)  
    manjaro-pulse: manjaro-pipewire (manjaro-pulse)  
    pipewire-pulse: pulseaudio (pipewire-pulse)  manjaro-pulse (pipewire-pulse)  
    pulseaudio: pipewire-pulse (pulseaudio)  

:: Conflicting packages will have to be confirmed manually
error: can not install conflicting packages with --noconfirm
failed to build packageDocker container wait error: 

This issue arise from paru trying to install the packages and might be solved by this PR.

@systemofapwne
Copy link

systemofapwne commented Dec 3, 2025

Another note: I just figured, that I was already using paru -B (as this PR introduces) since I used the new "intall from git" feature for my meta package. It still failed so you might want to consider adding this feature (or giving options to avoid build issues then).

My first try to solve this: Separating the conflicting package to another git repo
This is almost already working. At least for the manjaro-meta, it will work by simply removing the manjaro-pulse meta-package.

However, building all my own meta-packages still failed. This time, paru somehow installed jack2 on its own and then complained that jack2 conflicts with pipewire-jack (both provide jack). The reason was the following build order in my PKGBUILD and some funny makedepends issues.

- my-gstreamer --depends--> gst-libav      --makedepends-->  jack2 (provides jack)
- my-pipewire  --depends--> pipewire-jack (provides jack)

The make dependency of gst-libav caused the conflict. Switchin the order of

pkgname=(
  'my-gstreamer'
  'my-pipewire'
  'my-pulse'
)

to

pkgname=(
  'my-pipewire'
  'my-gstreamer'
  'my-pulse'
)

solved this for me (Reason: pipewire-jack now provides the required makedeps for gst-libav as it gets built before)

However: Since I am assembling several meta-packages here that might have unsolveable cross-depencies in the future (no reordering will help here), I sinply turned off all dependency checks.

First, paru's internal dependency checks must be forwarded to pacman via the --useask flag (solves the inner conflicts). Pacman then sees no issue for my-pipewire and my-pulse being mutually exclusive.
The conflict of jack2 vs pipewire-jack can then finally solved by adding the ultimate nodepends flag: -dd.

With all of this, I was able to build my meta-package.

You can find a strapped down example meta package, that produces the errors here: https://github.com/systemofapwne/pkg-my-meta-failing.git

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot build paru package (conflicts with paru-bin)

3 participants