Skip to content

Run CI build on Fedora 40 runners#716

Merged
achilleas-k merged 7 commits intoosbuild:mainfrom
achilleas-k:ci/fedora-40
Jun 7, 2024
Merged

Run CI build on Fedora 40 runners#716
achilleas-k merged 7 commits intoosbuild:mainfrom
achilleas-k:ci/fedora-40

Conversation

@achilleas-k
Copy link
Copy Markdown
Member

@achilleas-k achilleas-k commented May 29, 2024

Update the GitLab CI configurations (static and autogenerated) to use Fedora 40 runners.
Also, pull in a new version of osbuild to test the new optional metadata selection for depsolving [1] and to make sure everything is rebuilt.

[1] osbuild/osbuild#1775

@achilleas-k achilleas-k requested review from croissanne and thozza May 29, 2024 09:50
thozza
thozza previously approved these changes May 29, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@thozza thozza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Seeing so many changed lines in .gitlab-ci.yml made me wondering it we should simply define the fedora runners as variables at the top of the file and use those everywhere instead.

@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented May 29, 2024

LGTM.

Seeing so many changed lines in .gitlab-ci.yml made me wondering it we should simply define the fedora runners as variables at the top of the file and use those everywhere instead.

yeah, I think you might be right.
Though that file is generated from ./test/scripts/generate-gitlab-ci (and checked in tests). But the other generators (the dynamic ones) also need to be in sync so we could probably put it in a config somewhere anyway.

@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented May 29, 2024

Btw, found a bug: osbuild/osbuild#1795

bcl
bcl previously approved these changes May 30, 2024
View reviewed changes
Copy link
Copy Markdown

@bcl bcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented May 31, 2024

Btw, found a bug: osbuild/osbuild#1795

Updated osbuild commit ID to include this fix.

@achilleas-k achilleas-k dismissed stale reviews from bcl and thozza via 5e8b0ce May 31, 2024 12:51
thozza
thozza previously approved these changes May 31, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@thozza thozza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@achilleas-k achilleas-k mentioned this pull request May 31, 2024
Merged
@achilleas-k achilleas-k enabled auto-merge May 31, 2024 13:14
@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented May 31, 2024

podman run is failing on the runners (when trying to start ostree containers). Something to look into next week.
I think our F40 runners are a bit old though (since before F40 GA), so might just try updating them first.

@achilleas-k achilleas-k mentioned this pull request Jun 3, 2024
Merged
@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented Jun 3, 2024

Converting to DRAFT until the F40 runners get updated.

@achilleas-k achilleas-k marked this pull request as draft June 3, 2024 17:32
auto-merge was automatically disabled June 3, 2024 17:32

Pull request was converted to draft

@bcl bcl mentioned this pull request Jun 3, 2024
Closed
@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented Jun 5, 2024

Got new Fedora 40 runners thanks to @croissanne <3
Let's see if things get weird again.

@achilleas-k achilleas-k marked this pull request as ready for review June 5, 2024 15:30
@achilleas-k achilleas-k force-pushed the ci/fedora-40 branch 2 times, most recently from 6d7c4bd to e990729 Compare June 5, 2024 18:06
@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented Jun 5, 2024

Ok the weirdness hasn't gone away. I'm still getting permission errors when trying to run ostree containers:

📦 Starting container registry.gitlab.com/redhat/services/products/image-builder/ci/images/centos_9-x86_64-edge_container-empty:build-a8001dcacb938a15a1623c3cf1bde9ffe5abe13060ad246bf19b718064623db1 42000
❌ Command failed: ['podman', 'run', '-d', '--rm', '-p42000:8080', 'registry.gitlab.com/redhat/services/products/image-builder/ci/images/centos_9-x86_64-edge_container-empty:build-a8001dcacb938a15a1623c3cf1bde9ffe5abe13060ad246bf19b718064623db1']
Trying to pull registry.gitlab.com/redhat/services/products/image-builder/ci/images/centos_9-x86_64-edge_container-empty:build-a8001dcacb938a15a1623c3cf1bde9ffe5abe13060ad246bf19b718064623db1...
Getting image source signatures
Copying blob sha256:0e2f47f2266a843d0262757848e45968c4720183a7e99b3debc86686162c2c7a
Copying config sha256:09579bae409a9b8b084d0fbdb49b0f34aa4a294e88d950158590f2d52b91a368
Writing manifest to image destination
Error: crun: sd-bus call: Interactive authentication required.: Permission denied: OCI permission denied

While troubleshooting this, I decided to also do something I've been meaning to do for a while and switch to pulling ostree containers from the s3 bucket instead of putting them in the gitlab registry as well. I'm going to clean up the gitlab registry so we don't have the same containers in multiple places.

@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented Jun 5, 2024

Looking around some older issues, it seems it might be a configuration issue with rootless podman. I'm going to dig a bit more tomorrow but I'm not above just running it with sudo to get around this and move on.

@achilleas-k achilleas-k force-pushed the ci/fedora-40 branch 2 times, most recently from 9f02771 to 97639b3 Compare June 5, 2024 20:25
@achilleas-k
gitlab: run tests on Fedora 40 runners
ac71bf4 
Add repos to the Schutzfile for Fedora 40.
The repos for Fedora 39 can also be removed but let's keep them for a
bit.
@achilleas-k
schutzbot: update terraform commit ID
0141707 
Use the newest runners we have.
@achilleas-k
Schutzfile: update osbuild dependency
3b4f8af 
osbuild v120
@achilleas-k
test/generate-build-config: don't push containers to gitlab registry
c988cd2 
Container archives (and all image artifacts) are stored in S3 along with
the build info cache and manifests.  Let's stop pushing to the gitlab
registry.  We will use those archives instead.
@achilleas-k achilleas-k force-pushed the ci/fedora-40 branch 2 times, most recently from 6eaff78 to 639eaa0 Compare June 6, 2024 19:51
@achilleas-k
test/generate-ostree-build-config: download oci archive from s3
a0500fa 
Download the container archive from the s3 bucket instead of pulling it
from the gitlab registry.  After this, we will phase out use of the
gitlab registry and start removing all the containers we have there.

Each archive is downloaded to a temporary directory and run directly
with `podman run oci-archive:<path>`, which will also pull the archive
into the registry, then the temporary directory (along with the
downloaded archive) is discarded.
@achilleas-k
test/imgtestlib: remove old unused globals
b1eff23
@achilleas-k
test/generate-ostree-build-config: run podman as root
0adbdef 
Temporary workaround for podman permission issue.
@achilleas-k
Copy link
Copy Markdown
Member Author

achilleas-k commented Jun 6, 2024

Alright, we're green!!

bcl
bcl approved these changes Jun 6, 2024
View reviewed changes
Copy link
Copy Markdown

@bcl bcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, test failures are the integration tests only.

@achilleas-k achilleas-k added this pull request to the merge queue Jun 7, 2024
Merged via the queue into osbuild:main with commit d6575d0 Jun 7, 2024
@achilleas-k achilleas-k deleted the ci/fedora-40 branch June 7, 2024 11:28
@achilleas-k achilleas-k mentioned this pull request Jun 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thozza thozza thozza left review comments

@croissanne croissanne Awaiting requested review from croissanne

+1 more reviewer

@bcl bcl bcl approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@achilleas-k @bcl @thozza