Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 61 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ permissions:

jobs:
unit:
runs-on: ubuntu-24.04
strategy:
matrix:
os: [ubuntu-24.04, ubuntu-24.04-arm]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
Expand All @@ -26,9 +29,65 @@ jobs:
- name: Tests
run: python3 -m pytest env/tests -q

# The same suite on macOS: proves the control-node Python is macOS-clean
# (the qemu backend's ps fallback, no /proc, no /dev/kvm). No VMs here:
# GitHub macOS runners cannot use Hypervisor.framework.
unit-macos:
runs-on: macos-15
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
python3 -m venv "$HOME/ci-venv"
"$HOME/ci-venv/bin/pip" install --quiet pytest pyyaml
- name: Tests
run: |
"$HOME/ci-venv/bin/python" -m pytest env/tests -q

# Native aarch64 image build (no emulation): proves --distarch bakes an
# arm64 channel image with the provenance label.
arm-image:
runs-on: ubuntu-24.04-arm
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Build the aarch64 channel image
run: bash images/build.sh --dist 45 --tag freeipa-ci --ipa-from-distro --distarch aarch64
- name: Check the image architecture
run: |
arch="$(podman image inspect freeipa-ci/full:45 --format '{{.Architecture}}')"
[ "$arch" = arm64 ] || { echo "::error::architecture is $arch, expected arm64"; exit 1; }

# The published images are multi-arch manifests: proves the arm64 member
# of the tag the presets reference resolves and is pulled as arm64. No VM
# here (the GitHub arm runners have no KVM): the pull and the arch check
# are what this side of the flow needs.
containerdisk-arm:
runs-on: ubuntu-24.04-arm
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Pull the published image for the host arch
run: |
sudo apt-get -o Acquire::Retries=3 update
sudo apt-get install -y --no-install-recommends podman python3-pytest qemu-utils
python3 - <<'PY'
import sys
sys.path.insert(0, 'env')
from freeipa_env import containerdisk
disk = containerdisk.resolve(
'containerdisk:ghcr.io/freeipa/ci-freeipa-vm-images/'
'fedora-45:latest')
print(f'resolved: {disk}')
PY
arch="$(podman image inspect \
ghcr.io/freeipa/ci-freeipa-vm-images/fedora-45:latest \
--format '{{.Architecture}}')"
[ "$arch" = arm64 ] || { echo "::error::architecture is $arch, expected arm64"; exit 1; }

# Boots a VM from the published Fedora 45 image and installs a FreeIPA
# server on it with this checkout's freeipa-env: the qemu backend, the
# containerdisk pull, guest_packages and the integration run, end to end.
qemu:
runs-on: ubuntu-24.04
timeout-minutes: 60
Expand Down
66 changes: 63 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ The backends are in `freeipa_env/vmbackend.py`.
- **`ssh`**: a pool of VMs that already exist. Each entry is `user@host[:port]` in `vm.hosts`. `terminate` does nothing. Use this as a simple reference.
- **`command`**: runs your scripts. `vm.provision_cmd <count>` prints the VM handles. Print one `user@host[:port]` per line, or a JSON list. `vm.deprovision_cmd <handle>` removes a VM. This is the general integration point for a cloud API. Wrap any cloud in two scripts. See `examples/nested/`.
- **`openstack`**: an example that uses the `openstack` CLI. It runs `server create`, `show`, and `delete`, and it uses floating IPs. Use it as a template for other clouds.
- **`qemu`**: local VMs under qemu and KVM on the control node. There is no cloud API.
- **`qemu`**: local VMs under qemu on the control node: KVM on Linux, Apple hvf on macOS. There is no cloud API.
- **`krun`**: the same local VMs, as libkrun microVMs instead of qemu. See [The krun backend](#the-krun-backend).

#### The qemu backend
Expand All @@ -246,7 +246,14 @@ The backends are in `freeipa_env/vmbackend.py`.
- Each VM has a private NIC on `192.168.100.0/24`. `vm1` gets `.10`, `vm2` gets `.11`, and so on. The VMs have each other in `/etc/hosts`, so they can reach each other by name.
- `vm.vms` sets `name`, `image`, `cpus`, `memory`, and `disk` for each VM. `vm.count` must equal the length of `vm.vms`. With `inner: external`, `memory` comes from each host's `resources` entry instead, and a `memory` key in the vm block is an error.
- `terminate` stops the qemu process and removes the overlay. The `console.log` file stays in `vm.state_dir`.
- You need `/dev/kvm` and the host packages below.
- `vm.dns` (optional) sets the guest's nameservers, bypassing the slirp DNS forwarder (`10.0.2.3`, which uses the host's resolvers — on networks where those do not answer the guest, e.g. corporate macOS hosts, point it at public resolvers).
- `vm.image` is a file, a `containerdisk:` reference, or an `http(s)://` URL. A URL is downloaded once to `~/.cache/freeipa-ci/images/<basename>` (no checksum verification: for reproducibility, compare the file with the published `CHECKSUM` before the first `up`).
- `vm.arch` (default: the host arch, with `arm64` normalized to `aarch64`) picks the guest architecture: `qemu-system-<arch>` is then used. `aarch64` guests get the `virt` machine and a UEFI firmware; x86_64 guests keep `q35` and SeaBIOS.
- `vm.accel` (default: auto) picks the accelerator: `hvf` on macOS, `kvm` when `/dev/kvm` is writable, else `tcg`. Explicit values: `hvf`, `kvm`, `tcg`.
- `vm.cpu` (default: `host` under kvm/hvf, `max` under tcg) and `vm.machine` (informational: the backend derives it from the arch) can override the defaults.
- `vm.firmware` (aarch64 only) overrides the UEFI firmware auto-discovery, which looks next to the qemu binary (`<brew-prefix>/share/qemu/edk2-aarch64-code.fd`, the Homebrew layout) and in the usual distro paths (`/usr/share/edk2/aarch64/QEMU_EFI.fd`, ...).
- You need `/dev/kvm` only when the accelerator is `kvm` (an x86_64 Linux host, or an explicit `accel: kvm`); hvf on macOS and tcg need nothing. The host packages are below.
- USB passthrough and `gpu: virgl` need a Linux host: their host-side paths (`/sys/bus/usb`, `/dev/dri/...`) do not exist on macOS.

#### The krun backend

Expand All @@ -273,14 +280,16 @@ vm:
image: containerdisk:ghcr.io/OWNER/fedora-cloud-base:45-1.4
```

- `up` runs `podman pull`, copies the disk out with `podman create` and `podman cp`, and checks that it is a qcow2 file. It needs podman only: no root, no `bootc`, no conversion. This is the difference from the bootc disk above, which each guest would have to build.
- `up` runs `podman pull --platform linux/<arch>`, copies the disk out with `podman create` and `podman cp`, and checks that it is a qcow2 file. The arch is the one the VM will boot (`vm.arch`, default the host's), so a multi-arch tag yields its matching member; a tag without that member fails with a clear error instead of booting a disk of the wrong architecture. Inside a container (where podman is never run) skopeo selects the member with `--override-arch`/`--override-os`. It needs podman only: no root, no `bootc`, no conversion. This is the difference from the bootc disk above, which each guest would have to build.
- The disk is cached in `~/.cache/freeipa-ci/images/containerdisk/`, named after the image id. A tag that moves is picked up on the next `up`. An unchanged image is not copied again.
- `images/vm/publish.sh DISK.qcow2 REF [--push]` builds the image. It labels the image with the disk's SHA-256. Run `podman login` first; the script does not handle credentials. The Fedora cloud image that `qemu-prep.sh` downloads is the usual input.
- On Testing Farm, `qemu-prep.sh` does not download an image named `containerdisk:`, and the qemu plan installs podman. A guest then pulls from your registry, not from `download.fedoraproject.org`.
- The Fedora 45 qemu presets (`nested-qemu`, `nested-qemu-external`, `nested-qemu-mixed`, `qemu-copr-customized_ds_config_install`) use `containerdisk:ghcr.io/freeipa/ci-freeipa-vm-images/fedora-45:latest`, which [ci-freeipa-vm-images](https://github.com/freeipa/ci-freeipa-vm-images) builds, updates and tests. That is an updated image, so it may be newer than the file images. They need podman for the pull. The Fedora 44 image of the mixed presets is still a file until the registry has it.

I tested it with a local image: `publish.sh` built a containerdisk from the Fedora 44 cloud image, and the qemu backend booted a VM from it in 17 s. I did not pull from a real registry, and I did not run it on Testing Farm.

- The published tags are multi-arch manifests (x86_64 and aarch64): the same preset ref works on both host architectures. The aarch64 members are the pristine upstream images until the arm lanes can update and test them (the GitHub arm runners have no KVM).

#### GPU: virgl

Set `vm.gpu: virgl` to give a VM virtio-gpu with OpenGL. The control node's GPU renders the output.
Expand Down Expand Up @@ -316,6 +325,9 @@ These packages are required:
sudo dnf install qemu-system-x86-core qemu-img genisoimage rsync openssh-clients
```

On macOS (Apple Silicon): `brew install qemu xorriso` (qemu includes
`qemu-img`; add `podman` only for `containerdisk:` images).

You can use `xorriso` or `mkisofs` instead of `genisoimage`.

For `vm.tpm`, also install `swtpm` and `edk2-ovmf`.
Expand Down Expand Up @@ -449,6 +461,52 @@ The nested provider runs on the **control node**. The control node is a CI machi

A queue of `provider: nested` presets runs under the `freeipa-env queue` supervisor. It runs on a control-node `--runner`. Each job provisions its own VMs.

## Running on macOS (Apple Silicon)

Both lanes work on macOS; the control node stays this machine, no lima.

**VM lane (qemu, Apple hvf).** `brew install qemu xorriso podman`, then run
the smoke preset (the same one the x86_64 lane runs; the image tag is a
multi-arch manifest, so `up` pulls its aarch64 member):

```sh
freeipa-env up env/presets/qemu-smoke.yaml
freeipa-env run env/presets/qemu-smoke.yaml
freeipa-env down env/presets/qemu-smoke.yaml
```

The VM runs under Apple's Hypervisor.framework (`hvf`, near-native speed)
and the UEFI firmware bundled with brew's qemu is found automatically.
GitHub macOS runners cannot use Hypervisor.framework, so there is no macOS
VM job in CI; local runs are the proof. If `hvf` is unavailable,
`vm.accel: tcg` falls back to emulation (slow). USB passthrough and
`gpu: virgl` need a Linux host.

**Container lane (podman machine).** `brew install podman`, then:

```sh
podman machine init --cpus 4 --memory 8192 # new machine
podman machine set -m 8192 # existing one; restarts it
bash images/build.sh --dist 45 --tag freeipa-distro --ipa-from-distro
```

The machine builds aarch64 images natively (a 3500 MiB master container
needs more than the 4 GiB default machine). Any podman preset then runs
unchanged with `image: freeipa-distro/full:45` — the whole validated podman
flow runs inside the machine.

The podman machine shares only your home directory with its VMs (virtiofs):
keep the `--workdir` of podman-lane runs under `$HOME` (e.g.
`~/macos-smoke`). A workdir in `/tmp` fails at the first volume mount
(`statfs ... no such file or directory`). The qemu lane has no such limit.

**aarch64 image builds.** `images/build.sh --distarch aarch64` passes
`--platform linux/aarch64` to the container tool: native on arm hosts
(including a macOS podman machine); cross-builds from x86_64 need
qemu-user emulation and are slow. The tag scheme is arch-less — aarch64
builds on a shared host use a distinct `--tag`. Multi-arch *containerdisks*
(the VM images) come from [ci-freeipa-vm-images](https://github.com/freeipa/ci-freeipa-vm-images).

## Artifacts

`run` writes these files:
Expand Down Expand Up @@ -560,6 +618,8 @@ If the workdir has a `results.yaml` for each stage, the report adds a `Stages` t

- `presets/base-xmlrpc.yaml`: the Azure **xmlrpc** job. One master, DNS and KRA, and `test_xmlrpc` without `test_dns_plugin.py`.
- `presets/external-smoke.yaml`: a template for external provider smoke tests.
- `presets/qemu-smoke.yaml`: one qemu VM from the published Fedora 45 image with the distro FreeIPA packages, and ipatests' single-master install. The image tag is a multi-arch manifest, so the same preset runs on x86_64 and aarch64 hosts (Apple Silicon macOS under hvf, aarch64 Linux under KVM).
- `presets/nested-qemu-external.yaml`: master + client on their own qemu VMs, `inner: external`; the same multi-arch image tag as `qemu-smoke`.
- `presets/azure/`: one preset for each Azure CI job. These presets recreate `ipatests/azure/azure_definitions/{base,gating}-fedora.yml`.

| Preset (`presets/azure/`) | Azure job | Hosts | Suite |
Expand Down
78 changes: 60 additions & 18 deletions env/freeipa_env/containerdisk.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import gzip
import json
import os
import platform
import re
import shutil
import subprocess
Expand Down Expand Up @@ -49,12 +50,14 @@ def _podman(args):
return p.stdout.strip()


def _skopeo_pull(ref):
def _skopeo_pull(ref, arch):
"""(image id, layer path, temp dir) of `ref`, pulled with skopeo.

Used instead of podman on a control node that is itself a container
(see container.py): skopeo never touches a container store. The caller
must remove the temp directory.
must remove the temp directory. `arch` selects the member of a
multi-arch manifest list (the Go name: arm64, not aarch64) and pins the
OS to linux, or the host's (darwin) would select nothing.
"""
if not shutil.which('skopeo'):
raise ContainerDiskError('skopeo not found (needed for a '
Expand All @@ -71,9 +74,11 @@ def _skopeo_pull(ref):
with open(auth, 'w') as f:
f.write('{}\n')
env.setdefault('REGISTRY_AUTH_FILE', auth)
p = subprocess.run(['skopeo', 'copy', f'docker://{ref}',
f'dir:{tmp}'], capture_output=True, text=True,
timeout=3600, check=False, env=env)
p = subprocess.run(
['skopeo', 'copy', '--override-arch', _goarch(arch),
'--override-os', 'linux', f'docker://{ref}', f'dir:{tmp}'],
capture_output=True, text=True, timeout=3600, check=False,
env=env)
if p.returncode:
raise ContainerDiskError(f'skopeo copy {ref} failed: '
f'{(p.stderr or p.stdout).strip()[-800:]}')
Expand Down Expand Up @@ -115,16 +120,45 @@ def _qemu_img():
return None


def _resolve_podman(ref):
_podman(['pull', '-q', ref])
def _goarch(arch):
"""The Go architecture name registries use (x86_64/amd64, aarch64/
arm64)."""
return {'x86_64': 'amd64', 'aarch64': 'arm64'}.get(arch, arch)


def _assert_arch(ref, want):
"""Fail when the pulled image's architecture is not `want`.

A missing member of a manifest list makes the platform-asked pull fail
on its own, but a *single-arch* image is returned unchanged (no
error), so a linux/amd64-only tag asked for arm64 would otherwise hand
the qemu backend an unbootable disk.
"""
arch = _podman(['image', 'inspect', '--format', '{{.Architecture}}',
ref])
if arch != _goarch(want):
raise ContainerDiskError(
f'{ref}: the image is {arch}, not {_goarch(want)} (no '
f'{arch} member in the tag)')


def _resolve_podman(ref, arch):
# a containers-storage: ref names an image already in the local store:
# there is nothing to pull (podman 6 cannot, podman 4 need not), and
# the store is addressed without the transport prefix
local = ref.startswith('containers-storage:')
store_ref = ref[len('containers-storage:'):] if local else ref
if not local:
_podman(['pull', '-q', '--platform', f'linux/{arch}', ref])
_assert_arch(store_ref, arch)
image_id = _podman(['image', 'inspect', '--format',
'{{.Id}}', ref])[:12]
'{{.Id}}', store_ref])[:12]
dest = os.path.join(
CACHE, f'{re.sub(r"[^A-Za-z0-9._-]+", "_", ref)}-{image_id}.qcow2')
if os.path.isfile(dest):
return dest
os.makedirs(CACHE, exist_ok=True)
cid = _podman(['create', ref, DISK_IN_IMAGE]) # scratch: no command runs
cid = _podman(['create', store_ref, DISK_IN_IMAGE]) # scratch: no cmd runs
try:
_podman(['cp', f'{cid}:{DISK_IN_IMAGE}', dest + '.partial'])
except ContainerDiskError as e:
Expand All @@ -134,8 +168,8 @@ def _resolve_podman(ref):
return dest


def _resolve_skopeo(ref):
image_id, layer, tmp = _skopeo_pull(ref)
def _resolve_skopeo(ref, arch):
image_id, layer, tmp = _skopeo_pull(ref, arch)
try:
dest = os.path.join(
CACHE, f'{re.sub(r"[^A-Za-z0-9._-]+", "_", ref)}-'
Expand All @@ -149,21 +183,29 @@ def _resolve_skopeo(ref):
shutil.rmtree(tmp, ignore_errors=True)


def resolve(image):
"""The local qcow2 for `image`: a plain path is returned as it is."""
def resolve(image, arch=None):
"""The local qcow2 for `image`: a plain path is returned as it is.

`arch` (a qemu arch such as x86_64 or aarch64) selects the member of a
multi-arch manifest list; the default is the host platform. A tag that
has no such member fails with a clear error instead of handing the
backend a disk of the wrong architecture, which would not boot.
"""
if not image.startswith(PREFIX):
return image
if arch is None:
arch = platform.machine()
if arch == 'arm64':
arch = 'aarch64'
if container_mode.in_container():
container_mode.prepare_tools() # qemu-img, on the wrapper PATH
ref = image[len(PREFIX):]
if ref.startswith(('containers-storage:', 'oci-archive:')):
dest = _resolve_podman(ref) # local stores: only podman has them
if os.path.isfile(dest):
return dest
dest = _resolve_podman(ref, arch) # local stores: only podman has them
elif container_mode.in_container():
dest = _resolve_skopeo(ref)
dest = _resolve_skopeo(ref, arch)
else:
dest = _resolve_podman(ref)
dest = _resolve_podman(ref, arch)
if os.path.isfile(dest):
return dest
tool = _qemu_img()
Expand Down
6 changes: 6 additions & 0 deletions env/freeipa_env/krunproc.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,14 @@ def _alive(pid):
try:
with open(f'/proc/{pid}/cmdline', 'rb') as f:
return bool(f.read()) # a zombie has an empty cmdline
except OSError:
pass # no /proc (macOS): fall through to ps
try:
p = subprocess.run(['ps', '-p', str(pid), '-o', 'stat='],
capture_output=True, text=True)
except OSError:
return False
return p.returncode == 0 and not p.stdout.strip().startswith('Z')


def hub(sock_path, pidfiles):
Expand Down
Loading
Loading