This repository contains the FreeIPA CI tools. It has these parts:
freeipa-env: a tool that creates or attaches FreeIPA test environments. It reads one YAML file and runs the install, test, and uninstall steps.- The image pipeline, the preset catalog, and the queue catalog.
- A tmt test that runs a job batch on a Testing Farm guest.
The FreeIPA source tree does not contain this code. It contains only a pinned tmt plan. That plan runs the test from this repository against the checkout under test. See Repository split. The design is in the FreeIPA repository, in doc/designs/ci_modernization.md.
Use these guides for specific tasks:
- To test a FreeIPA worktree with an agent, read ci-freeipa-worktree.
- To test the packages that a Fedora release ships, read Podman images: the packages Fedora ships and ci-freeipa-distro-packages. You do not need a checkout, a build, or a COPR repository.
- Repository split
- Commands
- Where the IPA packages come from
- Env file
- Nested provider
- Artifacts
- Log analysis
- Presets
- Migrated PRCI definitions
- Run queues
- Runner transports
- Fedora dist-git lane
- Podman images: the packages Fedora ships
- Podman images: build the IPA RPMs
- Blend a preset
- Preset validation
- Notes
CI code and the branch under test have separate versions. The only CI file in a FreeIPA checkout is the pin: the plan ci/tmt/plans/freeipa-env.fmf.
The pin does three things:
- It finds the test in this repository (
tmt/tests/freeipa-env/). It uses a git URL and a pinned ref. - Its
preparestep installs the BuildRequires of the branch'sfreeipa.spec.in. Build dependencies belong to the code that is built.
To use newer CI tools, change the ref: line in the plan. You do not change this repository when a branch changes its build dependencies.
To run the test from a FreeIPA checkout, use this command. The plan's discover.url must be a git URL that tmt can reach. A local path works for local runs.
tmt run plan --name /ci/tmt/plans/freeipa-env -e FREEIPA_SRC=$PWD
FREEIPA_SRC gives the path of the checkout. The test builds the SRPM from it on the guest.
Testing Farm sets FREEIPA_REPO_URL and FREEIPA_REPO_REF instead. The guest clones the branch under test. Testing Farm copies the fmf tree as plain files. The copy has no .git directory and no submodules.
To submit a Testing Farm batch from here, use this command. The repository under test defaults to the origin and branch of that checkout.
freeipa-env queue run QUEUE --runner testing-farm --freeipa-src ~/freeipa
A contributor can ask for tests in the FreeIPA tree itself. A commit adds .ci-freeipa.yaml at the root of the tree, the way PRCI takes a link .freeipa-pr-ci.yaml. The file lists presets, or a queue, to run:
build: true # default: build FreeIPA from this checkout
presets:
- presets/qemu-smoke.yaml
- preset: presets/qemu-smoke.yaml
tests: [test_integration/test_installation.py::TestInstallMaster]
queue: queues/qemu-copr.yaml # or a queuefreeipa-env request .ci-freeipa.yaml --freeipa-src ~/freeipa [--workdir DIR] [--dry-run]
- A path is a file of the checkout when it has one, so a contributor can add a preset in the same commit. Otherwise it is a file of this repository: a preset is relative to
env/, a queue to the repository root. A name that is in neither is an error. An unknown key is an error. - With
build: truethe command runsmake srpmson the checkout in a Fedora container. The container installs the packages of the checkout'sci/tmt/plans/freeipa-env.fmf. Then each preset that does not name its own build gets that SRPM:build.srpmfor a podman preset,vm.srpmsfor a nested one. An external preset tests its own IPA. testsreplacesrun.testsof that preset.- The presets run one after the other:
up,run,summary,down. The exit status is 0 when every preset passed. A table of the results goes to$GITHUB_STEP_SUMMARYwhen it is set. --runner testing-farm --tf-repo-url URL --tf-ref COMMITsubmits the request to Testing Farm instead (queue run --runner testing-farmwith a queue made from the presets). It needsTESTING_FARM_API_TOKENand a commit the guest can clone. Only presets of this repository are accepted: a preset of the checkout,testsandbuild: falseare errors. A qemu preset is accepted: the request asks for a KVM guest,tf-runner.shbuilds the RPMs once in podman (images/build.sh --srpm DIR --build-only --rpms-out DIR), and the preset runs as a copy withvm.rpmsset (tmt/tests/freeipa-env/adapt-preset.py). That needs the qemu packages in the pin plan.--dry-runprints the request that would be sent. A dry run with a dummy token printed the right plan, repository, commit andFREEIPA_JOBS; I did not submit a real request.- A local request gives the VMs of a nested preset the RPMs too: it builds them once in podman, so the VMs do not compile FreeIPA.
The FreeIPA tree has a workflow that runs this on a contributor's fork. See doc/designs/ci_modernization.md there. The tests of this repository cover the request file and the adaptation of the presets. A request with qemu-smoke and build: false passed on a local machine, and make srpms of a FreeIPA checkout ran in the container.
freeipa-env up ENV.YAML Create or attach the environment.
freeipa-env run ENV.YAML Run the test workflow: install, tests, uninstall.
freeipa-env down ENV.YAML Collect journal and logs, then remove the environment.
freeipa-env show ENV.YAML Show the state, image labels, and installed IPA, kuipa, ahdapa, and akamu builds.
freeipa-env logs ENV.YAML Sort and retrieve the collected logs.
freeipa-env logs ENV.YAML --timing Time per installer service and step, rated red/amber/green, with the oddities; stored in a SQLite database.
freeipa-env logs ENV.YAML --test T --file NAME Show the last lines of a collected file, for one test and host.
freeipa-env run ENV.YAML --tests X Run these tests instead of the run.tests in the env file.
freeipa-env run ENV.YAML --detach Start the tests and return. The run continues after this command ends.
freeipa-env status ENV.YAML Show the state and output of a detached run.
freeipa-env wait ENV.YAML Wait for a detached run, get the report, and print a summary.
freeipa-env summary ENV.YAML|QUEUE.YAML Show the result of each test in the last run, with the cause of each failure. For a queue, each job in turn and then the totals.
freeipa-env overlay ENV.YAML SRC:/DST Copy files into every container. Each copy is checked with a hash.
freeipa-env reset ENV.YAML Create fresh containers from the same spec. Keep the old logs.
freeipa-env list List the environments on this host.
freeipa-env vms [PATTERN] [--stop [--yes]] List running qemu and krun VMs and who owns each; stop them.
freeipa-env prune [--yes] [--workdirs] Remove the environments that have nothing running.
freeipa-env report [ENV.YAML] [--format md] Make a self-contained results.html (or a Markdown results.md) from the artifacts.
freeipa-env resolve ENV.YAML Podman provider: find the image references in the local image store. Read only.
freeipa-env ensure ENV.YAML Podman provider: build the missing channel images for the env. Use Fedora packages, or build.srpm.
freeipa-env migrate PRCI.YAML Make presets from a PRCI definition.
freeipa-env queue run QUEUE.YAML --runner USER@HOST [...] Run a queue.
freeipa-env queue generate PRCI.YAML -o QUEUE.YAML Make a queue in PRCI order.
freeipa-env blend BASE PROVIDER --name NAME [--tests T ...] Make a preset or queue from a base and a provider.
freeipa-env check [PRESETS ...] Check presets. No containers are started.
--workdir DIR: the state and log directory. The default is$XDG_STATE_HOME/freeipa-env/<env-name>, which is~/.local/state/freeipa-env/<env-name>. It is never the current directory.--tool podman: use podman.--seccomp FILE: use a seccomp file. Podman only.--strict: with the external provider, fail when a port cannot be reached.--ssh-key FILE: with the external provider, use this root SSH key.
A backup or restore test class can take 30 minutes or more. A foreground run depends on the calling process. If a wrapper timeout ends the command, or a laptop sleeps, the command stops. The tests can keep running in the master.
Use run --detach for long runs:
statusshows the state and the last lines of output.waitgetsnosetests.xmland prints a summary for each test. It also gives the cause of each failure. Use--timeoutto set a limit.- With the podman provider the detached run lives in the master container. With the external and nested providers it is a background copy of the same
runcommand on this host: its output and exit status are in<workdir>/detached/, andstatusandwaitread them. After any provider'srunorwait, the report is copied from the controller to<workdir>/nosetests.xmland summarized. A report left by an earlier run is removed first, so a run that writes none is not mistaken for it. summarylooks fornosetests.xmlin the workdir, in itslogs, and underlogs/collected/<host>/, so it also works afterdown. It says which file it read when that is not the workdir's own.- For a nested external env,
downcollects into<workdir>/inner/logsand copies that into<workdir>/logs.summary,logsandreportreadlogs; if it has nocollectedtree, for example after a failed copy, they readinner/logsinstead. run --tests Xchooses tests without an edit to the env file.
Several classes in one run share the containers. If an uninstall leaves state behind, the next class gets that state. That is a real result, not noise. For an independent result, use reset. It creates fresh containers and keeps the old logs as logs.<timestamp>.
up chooses a free podman subnet when the env file names none. If a named subnet is in use, up fails and gives the name of the network that uses it.
up warns when the dns_forwarder does not answer from the master. An install with DNS fails its forwarder check when the resolver is down.
overlay SRC:/DST copies files into every container. It checks each file with SHA-256. It writes the copies to <workdir>/overlay.json. After that, show and the run summary say that the run is not a packaged run. Sources must be final files, not *.in templates.
list shows every environment on the host. It reads the podman networks, so it also shows orphans whose workdir is gone. prune removes the stopped environments. It is a dry run unless you add --yes.
vms lists the running qemu and krun VMs, one line each: id, qemu pid, host name, IP and the environment that owns it. A VM that no environment's nested-state.json names is shown as (orphan); this happens when a workdir is removed or its state file is lost. Give one or more glob patterns to pick VMs by environment name, workdir path or VM id, or orphan for the unowned ones. --stop stops the VMs listed (all of them when you give no pattern). It is a dry run unless you add --yes. For example, vms 'ci-*' --stop --yes. It stops the qemu process only: disks stay, and down still collects logs and removes them for a VM an environment owns. It reads the default qemu state_dir; use --qemu-dir for another.
up on a nested environment that already has VMs recorded in its nested-state.json is refused: a second up would overwrite that file and leave the first VMs running with nobody to stop them. Run down first.
Stopping a VM leaves its directory (a disk image, 1 to 2 GB) under the qemu state_dir. vms --stale lists the directories of VMs that are not running and that no environment names, with their sizes; with --purge it deletes them (a dry run unless --yes). A directory changed in the last five minutes is skipped, because up may still be filling it. vms --stop --purge --yes also deletes the directories of the VMs it stops.
What the IPA under test is, and how it gets there, depends on the provider. Only the podman provider uses a container image. The sections on channels, resolve, ensure, images/build.sh, --srpm, build: and --ipa-from-* describe those images. They do not apply to the other providers.
| Provider | The IPA runs in | The IPA packages come from |
|---|---|---|
podman |
containers on this host | a freeipa-ci/full:<channel> image. images/build.sh bakes it with dnf from the distribution or a COPR, or from RPMs you give it, or from an SRPM it compiles. The preset's image: names a channel. |
external |
hosts you provide | what is already installed on them. freeipa-env installs and builds nothing. |
nested, inner: podman |
containers inside a VM | the same channel image, made on the VM. |
nested, inner: external |
the VMs themselves | the vm: settings, installed by dnf in each VM: guest_packages, copr, rpms and srpms. No image is built. |
A queue runner follows each preset. A podman job needs a channel image on the runner. A nested-qemu job needs none: see Running the jobs in qemu VMs.
name: base-xmlrpc
provider: podman # or: external (pre-created), nested (get a VM through an API)
domain: ipa.test
dist: 44
image: freeipa-current # build channel. The provider finds the image. Per-host image: overrides it.
hosts:
- {role: master, name: master1}
- {role: client, name: client1}
resources: # defaults match the Azure variables-fedora.yml file
master: {memory: 1800m, memory-swap: 2500m}
client: {memory: 512m, memory-swap: 1024m}
run:
mode: base # or: integration
setup_dns: true
setup_kra: true
forwarder: 8.8.8.8 # set it here, or use --auto-forwarders
tests: [test_xmlrpc]
ignore: [test_xmlrpc/test_dns_plugin.py]External systems already exist. To use one, set provider: external. You can also give a host an address:.
Each host needs name, role, and address. It can also have user, port, image, and os.
upchecks reachability, SSH, the hostname, the distribution, and the IPA state. It does not change the system.runworks the same as before.downcollects logs. It does not delete anything.
Use --ssh-key to set the key. The default is ~/.ssh/id_rsa.
Use provider: nested when the test VMs exist only through an API. A nested provider has two parts:
- An outer VM backend gets a VM.
- An inner provider runs the env in the VM. The default inner provider is
podman.
The inner environment uses the same spec. Its provider is inner, and the vm: block is removed. The hosts: become containers in the VM. The full podman flow then runs in the VM.
The backend gets a runner when it is needed. It releases the runner at down. This is the supervisor's pre-allocated runner, generalized.
name: nested-example
provider: nested
image: freeipa-current
inner: podman # inner provider. The default is podman.
vm:
backend: command # ssh | command | openstack | qemu | krun
count: 1 # number of VMs. The podman inner provider uses the first one.
provision_cmd: examples/nested/get-vm.sh # $1 is the count. Print user@host[:port], one per line.
deprovision_cmd: examples/nested/drop-vm.sh # $1 is the VM id or target.
hosts:
- {role: master, name: master1}
- {role: replica, name: replica1}
run: {mode: integration, tests: [test_integration/test_simple_replication.py]}The backends are in freeipa_env/vmbackend.py.
ssh: a pool of VMs that already exist. Each entry isuser@host[:port]invm.hosts.terminatedoes nothing. Use this as a simple reference.command: runs your scripts.vm.provision_cmd <count>prints the VM handles. Print oneuser@host[:port]per line, or a JSON list.vm.deprovision_cmd <handle>removes a VM. This is the general integration point for a cloud API. Wrap any cloud in two scripts. Seeexamples/nested/.openstack: an example that uses theopenstackCLI. It runsserver create,show, anddelete, and it uses floating IPs. Use it as a template for other clouds.qemu: local VMs under qemu on the control node: KVM on Linux, Apple hvf on macOS. There is no cloud API.krun: the same local VMs, as libkrun microVMs instead of qemu. See The krun backend.
- Each VM is a qcow2 overlay on
vm.image. Use a cloud base image, such as Fedora Cloud Base. vm.disk(default40G) is the size of the overlay. It must be at least the image's virtual size: a smaller overlay cuts off the end of the image, and the VM then fails to boot in a way that looks like a network problem.upchecks this withqemu-img infoand refuses a smaller disk.- cloud-init adds the control node's SSH key for root. SSH uses a forwarded localhost port.
- Each VM has a private NIC on
192.168.100.0/24.vm1gets.10,vm2gets.11, and so on. The VMs have each other in/etc/hosts, so they can reach each other by name. vm.vmssetsname,image,cpus,memory, anddiskfor each VM.vm.countmust equal the length ofvm.vms. Withinner: external,memorycomes from each host'sresourcesentry instead, and amemorykey in the vm block is an error.terminatestops the qemu process and removes the overlay. Theconsole.logfile stays invm.state_dir.vm.dns(optional) sets the guest's nameservers, bypassing the slirp DNS forwarder (10.0.2.3, which uses the host's resolvers — on networks where those do not answer the guest, e.g. corporate macOS hosts, point it at public resolvers).vm.imageis a file, acontainerdisk:reference, or anhttp(s)://URL. A URL is downloaded once to~/.cache/freeipa-ci/images/<basename>(no checksum verification: for reproducibility, compare the file with the publishedCHECKSUMbefore the firstup).vm.arch(default: the host arch, witharm64normalized toaarch64) picks the guest architecture:qemu-system-<arch>is then used.aarch64guests get thevirtmachine and a UEFI firmware; x86_64 guests keepq35and SeaBIOS.vm.accel(default: auto) picks the accelerator:hvfon macOS,kvmwhen/dev/kvmis writable, elsetcg. Explicit values:hvf,kvm,tcg.vm.cpu(default:hostunder kvm/hvf,maxunder tcg) andvm.machine(informational: the backend derives it from the arch) can override the defaults.vm.firmware(aarch64 only) overrides the UEFI firmware auto-discovery, which looks next to the qemu binary (<brew-prefix>/share/qemu/edk2-aarch64-code.fd, the Homebrew layout) and in the usual distro paths (/usr/share/edk2/aarch64/QEMU_EFI.fd, ...).- You need
/dev/kvmonly when the accelerator iskvm(an x86_64 Linux host, or an explicitaccel: kvm); hvf on macOS and tcg need nothing. The host packages are below. - USB passthrough and
gpu: virglneed a Linux host: their host-side paths (/sys/bus/usb,/dev/dri/...) do not exist on macOS.
vm.backend: krun is a drop-in for qemu: the same cloud images, ssh over a forwarded localhost port, private NIC on 192.168.100.0/24, and freeipa-env vms lists and stops the VMs. It is Fedora-only for now, and inner: external is the only inner provider I tested. The example is env/presets/krun-smoke.yaml.
- Each VM is a
krunproc.py vmmprocess that drives libkrun throughctypes. The ssh NIC is served bypasst. The private NIC is a datagram hub (krunproc.py hub), one perup, that exits after the last VM stops. - The config is the qemu backend's, except that
gpu,render_node,usbandqemuare errors. Three keys are new, with the Fedora Cloud layout as the default:vm.root_device(/dev/vda3),vm.root_fstype(btrfs) andvm.root_options(subvol=root). Other image layouts must set them; there is no auto-detection. - The guest runs the kernel built into libkrunfw (6.12). It has no loadable modules, no iso9660, NFS or autofs, no iptables-legacy (nftables works), and no nested KVM. Tests that need NFS or automount fail on this backend.
vm.cpusis at most 16. The cloud-init seed is a vfat image, because the kernel cannot read an ISO.- A guest reboot ends the VM; there is no restart.
- The private NIC has no virtio checksum or TSO offload. The hub passes frames on unchanged and nothing completes a partial checksum, so with offloads on, the receiving guest dropped TCP segments (ping still worked). I checked a 50 MB transfer between two VMs, with no checksum errors.
- libkrun 1.19.7 guests die at boot, with no output, when the RAM size is between about 3285 and 3328 MiB. The backend raises such a size to 3336 MiB. I found the edges by trying sizes, so they may be off by a few MiB.
vm.state_dirmust be short: the VM directory path is at most 83 bytes, because unix socket paths are limited to 107 bytes.uprefuses a longer one.terminatestops the VM and removes the overlay.console.log,vmm.log,passt.logandhub.logstay in the VM directory.- Testing Farm has no krun plan:
tmt/plans/qemu.fmfis qemu only. - The packages (Fedora):
sudo dnf install libkrun passt qemu-img dosfstools mtools. You also need/dev/kvm.
vm.image (also per VM in vm.vms) can be a file or containerdisk:REF. REF is a container image reference that podman can pull, such as ghcr.io/OWNER/fedora-cloud-base:45-1.4. The image is a containerdisk: a FROM scratch image that holds one file, /disk/disk.qcow2.
vm:
image: containerdisk:ghcr.io/OWNER/fedora-cloud-base:45-1.4
uprunspodman pull --platform linux/<arch>, copies the disk out withpodman createandpodman cp, and checks that it is a qcow2 file. The arch is the one the VM will boot (vm.arch, default the host's), so a multi-arch tag yields its matching member; a tag without that member fails with a clear error instead of booting a disk of the wrong architecture. Inside a container (where podman is never run) skopeo selects the member with--override-arch/--override-os. It needs podman only: no root, nobootc, no conversion. This is the difference from the bootc disk above, which each guest would have to build.- The disk is cached in
~/.cache/freeipa-ci/images/containerdisk/, named after the image id. A tag that moves is picked up on the nextup. An unchanged image is not copied again. images/vm/publish.sh DISK.qcow2 REF [--push]builds the image. It labels the image with the disk's SHA-256. Runpodman loginfirst; the script does not handle credentials. The Fedora cloud image thatqemu-prep.shdownloads is the usual input.- On Testing Farm,
qemu-prep.shdoes not download an image namedcontainerdisk:, and the qemu plan installs podman. A guest then pulls from your registry, not fromdownload.fedoraproject.org. - The Fedora 45 qemu presets (
nested-qemu,nested-qemu-external,nested-qemu-mixed,qemu-copr-customized_ds_config_install) usecontainerdisk:ghcr.io/freeipa/ci-freeipa-vm-images/fedora-45:latest, which ci-freeipa-vm-images builds, updates and tests. That is an updated image, so it may be newer than the file images. They need podman for the pull. The Fedora 44 image of the mixed presets is still a file until the registry has it.
I tested it with a local image: publish.sh built a containerdisk from the Fedora 44 cloud image, and the qemu backend booted a VM from it in 17 s. I did not pull from a real registry, and I did not run it on Testing Farm.
- The published tags are multi-arch manifests (x86_64 and aarch64): the same preset ref works on both host architectures. The aarch64 members are the pristine upstream images until the arm lanes can update and test them (the GitHub arm runners have no KVM).
Set vm.gpu: virgl to give a VM virtio-gpu with OpenGL. The control node's GPU renders the output.
vm.render_nodesets the render node. The default is/dev/dri/renderD128. The user must be able to read and write it.- You need the packages
qemu-device-display-virtio-gpu-glandqemu-ui-egl-headless. - To check the guest, run
EGL_PLATFORM=surfaceless eglinfo. The output must show thevirtio_gpudriver and avirglrenderer.
Each VM with virgl also sends its screen over VNC and websocket, on the loopback address. up prints the port.
To see the screen in a browser, install noVNC (the novnc package). Open vnc.html?host=127.0.0.1&port=<ws>. To view it from another machine, make a tunnel: ssh -L <ws>:127.0.0.1:<ws>.
Set vm.usb: [VVVV:PPPP, ...] in vm.vms[i] to give host USB devices to that VM. Each entry claims one free host device with that vendor and product ID. One device can go to one VM only.
- The devices attach at boot only. There is no hot-plug.
- While the VM runs, the device is not available on the host.
- The user must be able to read and write
/dev/bus/usb/BBB/DDDfor the device. If not,upfails and prints that path. A udev rule is the usual fix.
Set vm.tpm: true in vm.vms[i] to give that VM a TPM 2.0 (tpm-crb, emulated by swtpm). Check it in the guest with systemd-analyze has-tpm2, which must say yes. That needs UEFI: on a BIOS boot it says partial, with tpm-tis too, so a VM with tpm boots with /usr/share/edk2/ovmf/OVMF_CODE.fd.
- The TPM state lives in the VM directory and is discarded at
down. - The
krunbackend does not support it.
These packages are required:
sudo dnf install qemu-system-x86-core qemu-img genisoimage rsync openssh-clients
On macOS (Apple Silicon): brew install qemu xorriso (qemu includes
qemu-img; add podman only for containerdisk: images).
You can use xorriso or mkisofs instead of genisoimage.
For vm.tpm, also install swtpm and edk2-ovmf.
For vm.gpu: virgl, also install:
sudo dnf install qemu-device-display-virtio-gpu-gl qemu-ui-egl-headless virglrenderer
To see the VM screen in a browser, also install novnc.
The guest GL packages (mesa-dri-drivers, egl-utils) go inside the VM, not on the host.
The user must be able to read /dev/kvm. For virgl, the user must also read and write render_node.
Set vm.guest_packages in the nested spec. Each entry is a set name, a plain package name, or -name to remove that set's or package's packages from the list. up installs the packages with dnf on every VM, before the inner up.
gpu: installsmesa-dri-driversandegl-utils. Use it for the virgleglinfocheck.ipa-distro: installs the FreeIPA server, client, and Python packages from the distribution, plusfirewalld, which the FreeIPA test suite expects. It does not use COPR or a local build. It also installspytest-html==4.2.0with pip.ipa-distro-el: the same for CentOS Stream and RHEL clones, where the packages are namedipa-*instead offreeipa-*. Checked against Stream 9 and 10.
vm:
guest_packages: [gpu, ipa-distro]
gpu: virgl- On Fedora,
dnfinstalls from the Fedora repositories. - On CentOS Stream, RHEL, Rocky Linux, and AlmaLinux, the install first enables EPEL and CRB. If CRB is not available, it uses
powertools. - Plain names also work. For example:
guest_packages: [tmux]. - Subtract with
-name. For example,guest_packages: [ipa-distro, -firewalld, -python3-pip]keeps theipa-distroset but drops those two packages. The-entry works in any position. The pip step of a set is not removed. - VM settings are applied in this order, on every VM:
selinux,copr,guest_packages,rpms,srpms,script. Each is set undervm:and can be overridden for one VM in thevm.vmsentry named after its host (short or full name). The table isSETTINGSinenv/freeipa_env/guest_settings.py; a new setting is one row. vm.selinux: permissive(orenforcing) sets the SELinux mode, now and after a reboot.permissivealso turns ondontauditlogging, so/var/log/audit/audit.logrecords every denial a confined service would hit.vm.copr: [OWNER/PROJECT]enables those COPR repos withdnf copr enable, before the packages are installed.vm.rpms: [PATH, ...]copies local RPM files (globs allowed) to the VM, then runsdnf installanddnf reinstallon them. The reinstall is there because dnf skips a file whose version is already installed, which would keep the old build.vm.srpms: [PATH, ...]copies local source RPMs to the VM and rebuilds them there: for each one, in name order,dnf builddep,rpmbuild --rebuild, then install of the result. Later SRPMs build against the earlier ones. The built RPMs stay in/root/rpmbuild/RPMS.vm.script: [LINE, ...](or one multi-line string) runs asbash -ecafter the packages are installed.- If a package name is wrong,
upfails and prints thednfoutput. - Only Fedora 45 is tested. The EL clones are not tested.
- There are no guest desktop packages in any set yet.
up runs these steps:
- Provision the VMs.
- Wait for SSH.
- Copy the ci-freeipa tree and the inner spec to the VM.
- Run the inner
freeipa-env upon the VM. - Save the VM handles to
<workdir>/nested-state.json. A laterdownuses this file to find the VMs.
down runs these steps:
- Run the inner
freeipa-env down. This collects the logs. - Copy the remote workdir back. Then
logsand the xunit output work on the control node. - Release the VMs.
- Clear the state.
run starts the inner freeipa-env run on the first VM.
resolve does not work for nested environments. The VM does not exist yet. The image is found on the VM during up. The VM must have the freeipa-ci image. Or set vm.image_ref so that the VM runs podman pull.
With inner: external, nothing runs in containers. Each host runs directly on a VM named after the host. For the qemu backend, a vm.vms entry with the host's name sets that VM's cpus and other options.
These rules apply:
vm.countmust equal the number of hosts.
up does these steps:
- Provisions the VMs.
- Maps each host to the forwarded SSH port (
addressandport) and to the private address (ip) of its VM. - Writes the inner external spec.
- Attaches with
ExternalProviderfrom the control node.
run and show use the same provider. The ci-freeipa tree is not copied to the VMs. The example is env/presets/nested-qemu-external.yaml.
The framework runs on the master and SSHes to every host as root. up generates that key on the master at /root/.ipa/framework_key and authorizes its public key on each host. It also writes /etc/hosts entries for every host on every host.
Set framework_key in the env file to choose the key. type is passed to ssh-keygen -t unchanged, so any type the master's OpenSSH supports works (ed25519, rsa, ecdsa, ecdsa-sk, ...):
framework_key: {type: rsa, bits: 3072} # optional bits: passed as ssh-keygen -b- Without
framework_key, the default is ed25519. Withfips: true, the default is rsa. - The key is created only if
/root/.ipa/framework_keyis absent. Changingtypeon an existing master does not regenerate it. Remove the file on the master first. - A type the host does not support fails in
ssh-keygenon the master, duringup.
For a bootc host, set vm.bootc: true. up then runs bootc usroverlay on each VM. It runs this before guest_packages, so dnf can install into /usr.
The overlay is removed at reboot. After a reboot, run up again.
The example is env/presets/nested-qemu-bootc.yaml. It uses guest_packages: [ipa-distro] and one master1 VM. FreeIPA runs directly on the VM, without containers.
ci-freeipa does not build the base disk. The disk was built with these steps. The steps run in a guest VM, because the conversion needs root.
podman pull quay.io/fedora/fedora-bootc:44
truncate -s 20G /root/output/disk.raw
podman run --rm --privileged --pid=host -v /dev:/dev \
-v /var/lib/containers:/var/lib/containers -v /root/output:/output \
--security-opt label=type:unconfined_t quay.io/fedora/fedora-bootc:44 \
bootc install to-disk --via-loopback --generic-image --filesystem ext4 \
--root-ssh-authorized-keys /output/authorized_keys /output/disk.raw
qemu-img convert -O qcow2 /root/output/disk.raw /root/output/disk.qcow2Copy the qcow2 file to ~/.cache/freeipa-ci/bootc/fedora-bootc-44.qcow2.
Test status:
- Tested: the bootc VM starts.
usroverlayanddnf installwork. Theipa-distropackages install. The freeipa-server version is 4.13.4. - Not tested:
ipa-server-installon the bootc VM. FreeIPA does not support the bootc server side yet. This is the failure to investigate.
The nested provider runs on the control node. The control node is a CI machine. It has the cloud API credentials and SSH access.
A queue of provider: nested presets runs under the freeipa-env queue supervisor. It runs on a control-node --runner. Each job provisions its own VMs.
Both lanes work on macOS; the control node stays this machine, no lima.
VM lane (qemu, Apple hvf). brew install qemu xorriso podman, then run
the smoke preset (the same one the x86_64 lane runs; the image tag is a
multi-arch manifest, so up pulls its aarch64 member):
freeipa-env up env/presets/qemu-smoke.yaml
freeipa-env run env/presets/qemu-smoke.yaml
freeipa-env down env/presets/qemu-smoke.yamlThe VM runs under Apple's Hypervisor.framework (hvf, near-native speed)
and the UEFI firmware bundled with brew's qemu is found automatically.
GitHub macOS runners cannot use Hypervisor.framework, so there is no macOS
VM job in CI; local runs are the proof. If hvf is unavailable,
vm.accel: tcg falls back to emulation (slow). USB passthrough and
gpu: virgl need a Linux host.
Container lane (podman machine). brew install podman, then:
podman machine init --cpus 4 --memory 8192 # new machine
podman machine set -m 8192 # existing one; restarts it
bash images/build.sh --dist 45 --tag freeipa-distro --ipa-from-distroThe machine builds aarch64 images natively (a 3500 MiB master container
needs more than the 4 GiB default machine). Any podman preset then runs
unchanged with image: freeipa-distro/full:45 — the whole validated podman
flow runs inside the machine.
The podman machine shares only your home directory with its VMs (virtiofs):
keep the --workdir of podman-lane runs under $HOME (e.g.
~/macos-smoke). A workdir in /tmp fails at the first volume mount
(statfs ... no such file or directory). The qemu lane has no such limit.
aarch64 image builds. images/build.sh --distarch aarch64 passes
--platform linux/aarch64 to the container tool: native on arm hosts
(including a macOS podman machine); cross-builds from x86_64 need
qemu-user emulation and are slow. The tag scheme is arch-less — aarch64
builds on a shared host use a distinct --tag. Multi-arch containerdisks
(the VM images) come from ci-freeipa-vm-images.
run writes these files:
<workdir>/nosetests.xml: JUnit XML fromipa-run-tests --with-xunit.<workdir>/logs/: the per-host journal, the/var/log/ipa*tarballs, and the install and test run logs.IPA_TESTS_LOGSDIR(inside the container): per-test logs from the framework (--logfile-dir).downandcollect_logsalso copy this tree to<workdir>/logs/collected/<master>/ipa-env/logs. Thenlogscan show it. See theper-testcategory below.
freeipa-env logs turns the collected files into a triage view.
- With no
--category, it prints a summary for each category. The summary shows the number of sources and the number of interesting lines. - With
--category NAME, it prints the matching lines and the tail of each source file. Use--category allfor every category. You can repeat the option. - Daemon categories (
ds,httpd,ca,kdc,samba,dns,ipa,system) also read the per-test VM trees. Each test's uninstall removes the daemon logs from the VM, so the per-test copies are the only ones left. - Identical matching lines are reported once per host, keyed on the full line text, timestamp included. The summary counts and
--jsonuse the same rule. A repeated line shows asno new matching lines.--alldumps the unfiltered file and does not dedupe. - Given a queue file (for example one from
blend, withjobs:),logslists its jobs. Each job shows thelogscommand and the workdir of its env file. Runlogson one of those env files. - Filtered scans are cached in
<workdir>/.scan-cache.sqlite, keyed by file, pattern and size/mtime. A repeatlogsrun on an unchanged workdir takes well under a second. The first run after a newrun/downrescans the changed files.freeipa-env cache show ENVlists the cache per category;cache clear ENV --category NAME(repeatable) or--alldrops it.ENVmay be a queue file, and then the command applies to every job's workdir.--patternscans show ascustom:.
freeipa-env logs ENV.YAML # category summary
freeipa-env logs ENV.YAML --category tests # failed tests, grouped by module
freeipa-env logs ENV.YAML --category ds --category httpd --category kra # several at once
freeipa-env logs ENV.YAML --category tests --json # summary in JSON
freeipa-env logs --workdir DIR --category system --pattern 'OOM|segfault'
Each category has a default line filter. You can replace it with --pattern REX.
| Category | Source |
|---|---|
tests |
nosetests.xml (JUnit). Totals, failed and errored tests with messages and traceback tails, grouped by module. |
per-test |
The framework's per-test log tree. One directory for each test ID. It has the install and uninstall logs and the journal for each host. The xunit status is added. |
run |
logs/run.log. The console output of install, test, and uninstall. FAILED, ERROR, and summary lines. |
install |
ipaserver-install.log. ERROR and WARNING lines. |
uninstall |
ipaserver-uninstall.log. ERROR and WARNING lines. |
ds |
The 389-ds errors log. - ERR - and - CRIT - lines. |
httpd |
error_log. IPA framework errors and Python tracebacks. Debug messages are removed. |
kra, ca |
The pki instance logs (pki-tomcat/{kra,ca}) and the spawn logs. ca also reads the ipacta and akamu journal-namespace logs. |
oauth2 |
The ahdapa journal-namespace log. |
kdc |
krb5kdc.log. error, crit, and fatal lines. |
samba |
/var/log/samba. AD trust. |
dns |
The BIND runtime logs (named.run). |
ipa |
The per-operation logs in /var/log/ipa/. ERROR and CRITICAL lines. |
system |
The boot journal. Failed units, OOM events, segfaults, and watchdogs. |
Useful options:
--pattern REX: replace the line filter of a category.--lines N: the number of matched lines for each file. The default is 30.--tail N: the number of tail lines. Use 0 to turn it off.--all: print each whole file, with no filter.--host NAME: show one host only.--json: print the summary for scripts.--refresh: collect the logs again from an environment that still runs, before you analyze them. If the collection fails, the existing files are kept.
logs extracts the daemon tarballs into <workdir>/logs/extracted/<host>/. It extracts only the needed files. It checks the mtime and size of each file. Repeated calls are fast.
ipaserver/install/service.py logs, for each service it configures, the estimate it prints (Estimated time: 3 minutes), every step ([3/46]), the time of every step (step duration: dirsrv __create_instance 4.52 sec) and, in newer versions, of the service (service duration:). logs --timing reads these lines from every installer log it finds (ipaserver, ipareplica and ipaclient install and uninstall logs, for each host and per-test snapshot) and prints, per installer run:
- each service: its estimate, its actual time, the ratio, and a rating
- the slowest steps (
--top N, default 10) - the oddities: logged warnings and errors (identical ones counted once), steps that failed (
[error]), retries and timeouts, stalls (no log line for--gapseconds, default 30), and a run that did not end
The logs are appended to, so each Logging to ... line starts a new run. The framework keeps a snapshot of the same log per test; an invocation that appears in several files is reported once.
The ratings are Green, Amber and Red. A service is green up to its estimate, amber up to 1.5 times it, red beyond; a service with no estimate is not rated. A step is amber from --amber-step seconds (default 10) and red from --red-step (default 60); where its service gave an estimate, those limits are raised to half and all of that estimate. A failed run is red. Warnings and errors count at most amber in a run that succeeded, and a stall is listed but not counted, since the step that stalled is already rated.
Every run is also stored in a SQLite database, <state dir>/freeipa-env/timing.sqlite (--db FILE to choose, --no-db to skip). Tables: runs (environment, host, source, times, outcome, overall rating), services, steps and oddities, joined by run_id. Running it again on the same logs replaces the earlier rows, so the database holds one row per run. For example, the slowest steps of every environment:
sqlite3 ~/.local/state/freeipa-env/timing.sqlite \
"select env, host, service, method, sec from steps join runs on runs.id = run_id order by sec desc limit 10"
--json prints the same analysis as JSON. --host limits it to one host.
freeipa-env report makes one file, results.html. The default location is <workdir>/results.html. Share this file or paste it into a ticket.
--format md writes results.md instead: the same status, stages, tests (with the failure text) and matched log lines per category, as Markdown for a terminal, a PR or an issue. It has no per-test log blocks, and it shows at most 50 matched lines per file where the HTML shows 200. The default location is <workdir>/logs/results.md.
The report uses the same code as logs. It works offline. It needs no environment file for a workdir that is already collected. It has no JavaScript and no external files. All CSS is inside the file. Sections collapse with the HTML <details> element.
Each failed, errored, or skipped test row has its failure message and traceback tail.
freeipa-env report --workdir DIR # writes <workdir>/results.html
freeipa-env report --workdir DIR -o /tmp/r.html # choose the output file
freeipa-env report --workdir DIR --format md # writes <workdir>/logs/results.md
freeipa-env logs ENV.YAML --html # the same report, from logs
freeipa-env tf-logs <request-id> --html # download a TF request, then make the report
The report is less detailed than the report.html from PRCI. PRCI uses pytest-html inside the image. This report has these parts:
- Per-test status and duration from the xunit file.
- The shared log categories.
- The per-test log trees. Each test has its own collapsed block, with its xunit status.
It does not show the captured stdout of each test. The image in this repository does not run pytest-html.
If the workdir has a results.yaml for each stage, the report adds a Stages table. The table shows the result, duration, and note of each stage. The Testing Farm transport uploads this file.
presets/base-xmlrpc.yaml: the Azure xmlrpc job. One master, DNS and KRA, andtest_xmlrpcwithouttest_dns_plugin.py.presets/external-smoke.yaml: a template for external provider smoke tests.presets/qemu-smoke.yaml: one qemu VM from the published Fedora 45 image with the distro FreeIPA packages, and ipatests' single-master install. The image tag is a multi-arch manifest, so the same preset runs on x86_64 and aarch64 hosts (Apple Silicon macOS under hvf, aarch64 Linux under KVM).presets/nested-qemu-external.yaml: master + client on their own qemu VMs,inner: external; the same multi-arch image tag asqemu-smoke.presets/azure/: one preset for each Azure CI job. These presets recreateipatests/azure/azure_definitions/{base,gating}-fedora.yml.
Preset (presets/azure/) |
Azure job | Hosts | Suite |
|---|---|---|---|
base.yaml |
base | master | Unit tests and test_xmlrpc/test_dns_plugin.py (-k 'not test_dns_soa') |
kerberos-flags.yaml |
kerberos_flags | master, client | test_kerberos_flags.py |
forced-client-reenrollment.yaml |
forced_client_reenrollment | master, replica, client | test_forced_client_reenrollment.py |
external-ca-install.yaml |
external_ca_ExternalCAInstall | master | test_external_ca.py::TestExternalCAInstall |
simple-replication.yaml |
simple_replication | master, replica | test_simple_replication.py |
service-permissions.yaml |
service_permissions | master | test_service_permissions.py |
sudo.yaml |
sudo | master, client | test_sudo.py. Deselects TestSudo_Functional. |
external-ca.yaml |
external_ca_ExternalCA | master, replica, client | test_external_ca.py::TestExternalCA |
topologies-testconfig.yaml |
topologies_and_testconfig | master | test_topologies.py, test_testconfig.py. No installs. |
external-ca-constraints.yaml |
external_ca_ExternalCAConstraints | master, client | test_external_ca.py::TestExternalCAConstraints |
membermanager.yaml |
membermanager | master | test_membermanager.py |
netgroup.yaml |
netgroup | master | test_netgroup.py |
external-ca-profile-scenarios.yaml |
external_ca_ExternalCAProfileScenarios | master | test_external_ca.py::TestExternalCAProfileScenarios |
caless-to-ca-full.yaml |
ServerReplicaCALessToCAFull | master, replica | test_caless.py::TestServerReplicaCALessToCAFull |
external-ca-self-external-self.yaml |
external_ca_SelfExternalSelf | master | test_external_ca.py::TestSelfExternalSelf |
Run modes:
- Base pipeline jobs use
mode: base. The steps are install, test, and uninstall. The script isrun-base-tests.sh. - Gating jobs use
mode: integration. The framework does the install and uninstall for each class. It reads the multihost config.
Each host gets the same memory limits as in the Azure definitions.
freeipa-env migrate ipatests/prci_definitions/<def>.yaml converts a PRCI definition. It writes one preset for each job under presets/prci/<def>/. It also writes a MIGRATED.md report.
Run migrate again after you change a PRCI definition. Options:
-o DIR: the output directory.--jobs NAME: keep only the jobs whose name contains NAME.
presets/prci/ has all 11 PRCI definitions. It has 1301 presets in total.
| Directory | PRCI file | Presets |
|---|---|---|
gating/ |
gating.yaml |
31 |
nightly-latest/ |
nightly_latest.yaml |
185 |
nightly-latest-389ds/ |
nightly_latest_389ds.yaml |
48 |
nightly-latest-pki/ |
nightly_latest_pki.yaml |
91 |
nightly-latest-selinux/ |
nightly_latest_selinux.yaml |
185 |
nightly-latest-sssd/ |
nightly_latest_sssd.yaml |
20 |
nightly-latest-testing/ |
nightly_latest_testing.yaml |
185 |
nightly-latest-testing-selinux/ |
nightly_latest_testing_selinux.yaml |
185 |
nightly-previous/ |
nightly_previous.yaml |
185 |
nightly-rawhide/ |
nightly_rawhide.yaml |
185 |
temp-commit/ |
temp_commit.yaml |
1 |
- Topology to hosts.
ipaservergives one master, in base mode.master[_Nrepl]_Mclientgives master, replica, and client hosts. These use integration mode when there is more than one IPA host. The AD tokensadandadroot_adchild_adtreeadd external AD hosts. Their roles aread,ad_subdomain, andad_treedomain. - Topology memory to limits. PRCI gives a total for each topology.
migratedivides it evenly between the roles. It rounds to 100 MiB. The minimum is 512 MiB. - test_suite to
run.tests. The space-separated list is kept. - class Build. Skipped. The
freeipa-ciimage pipeline replaces it. - ipa_ipa_trust topologies. Skipped. These need two IPA domains. The preset model supports one IPA domain and AD domains.
- RunWebuiTests. Uses
mode: base. A note is added. PRCI installs a browser and selenium. Thefreeipa-ci/fullimage does not have them. These presets fail as they are.
PRCI creates its own AD domain controllers. This repository does not. Each AD host is an address placeholder from RFC 5737, with a placeholder FQDN. Edit both before you run freeipa-env up. The footer of each preset lists every line marked EDIT ME.
up writes the AD FQDNs into /etc/hosts on the IPA containers. The tests set up the trust. They use ad_admin_name and ad_admin_password. The defaults are Administrator and Secret123.
A Samba AD DC works if SSH can reach it for log collection. Windows-only DCs need the user value changed.
With the nested provider, an ad host with os: samba becomes a Samba AD DC on its own VM. up installs samba-dc, provisions the domain ad_domain (default ad.test), keeps the DC on the private network only, serves time with chrony, and creates the users and groups that the trust tests expect. The IPA hosts get guest_packages; the DC does not.
ad_domain: ad.test
hosts:
- {role: master, name: master1}
- {role: ad, name: dc1, os: samba}The generated ipa-test-config.yaml has the DC in its own AD domain with host_type: samba. The ipatests harness then drives the DC with samba-tool instead of PowerShell. That needs an ipatests that has SambaADHost: the FreeIPA commit "ipatests: AD host operations are methods, with Windows and Samba AD backends". Install it with vm.rpms or vm.srpms; the distribution package does not have it yet.
The preset is env/presets/qemu-samba-ad.yaml. I ran TestCertsInIDOverrides (the trust, an AD user, ID overrides and an sssd D-Bus lookup) against it; both tests passed with the changed ipatests copied onto the IPA VM. Not covered yet: the child and tree domains, the cannotchangepwduser and expiredaduser properties, UPN suffix users, and AD CS. The Samba DNS server has one global forwarder, not one per zone.
A queue runs an ordered list of presets. It uses a pool of pre-allocated runners. A runner is a host with SSH, podman, and systemd. The lab VMs are runners. Queues replace the per-PR job scheduling of PRCI.
Queue files are in queues/:
azure.yaml: the 16 Azure CI jobs. Azure ran them in parallel. The order in the queue is the usual order.- One file for each PRCI definition. The names match the preset directories, for example
gating.yamlandnightly-latest.yaml. Each file keeps the job order of its PRCI definition. Make them withfreeipa-env queue generate. Run it again after a PRCI definition changes.
A queue file has a jobs: list. Each entry is one of these:
- A preset path, relative to
env/. - A directory path that ends with
/. It expands to all presets in that directory, sorted. - A dict with
presetordir, and an optionalnote.
This section and the three that follow it describe the podman provider, the only one that runs from a prebuilt image. See Where the IPA packages come from for the others.
A preset does not name a build. It names a channel:
image: freeipa-current: the newest build of the current release.image: freeipa-next: the next release. PRCI calls itfedora-rawhide.image: freeipa-previous: the previous release.
PRCI uses one build generation for each definition file. Every job in the file uses that build. The channels do the same thing here.
images/build.sh --channel NAME adds the channel tag, freeipa-ci/full:<channel>. It also adds the fixed provenance tag, freeipa-ci/full:<dist>-<sha>.
The provider finds the image for a channel on each host. This is a provider task.
- The podman provider maps the channel to a podman tag. It checks the tag in the local image store during
up. freeipa-env resolve ENV.YAMLdoes only this step. It prints<ref> <concrete> <image_id>.- The external provider ignores the channel. External hosts run their own IPA.
A preset can also name an explicit image reference. The reference is used as it is. Use this to pin a build.
freeipa-env queue run queues/gating.yaml \
--runner root@192.168.122.215 --runner root@192.168.122.216 \
[--jobs NAME ...] [--limit N] [--dry-run] [--keep-on-failure] \
[--job-timeout SECS] [--outdir DIR] [--no-bootstrap] [--ssh-key FILE]
For each run, the supervisor does these steps:
- Checks every preset. It parses each file and checks that it exists.
- Checks each runner for SSH and podman. Copies the local ci-freeipa tree to
--remote-ci. The default is/root/freeipa-ci/ci. - Asks each runner to find the channel image for each podman preset. A nested-qemu preset needs none. This runs
freeipa-env resolveon the runner. If an image is missing, the run stops before any job starts. The results are saved per runner insummary.md. - Schedules the jobs. Each runner takes the next job from the front of the list. It runs one job at a time. Each job runs
up, thenrun, thendown. Each job has a remote timeout. The default is 4 hours. Each job uses the workdir/root/jobs/<job-key>. - Writes
summary.tsv,summary.md, and a transcript for each job, in--outdir. The default is<state dir>/freeipa-env/queues/<queue>, next to the environment workdirs, so a run leaves nothing in the repository. The exit code is 0 only when every job passes.
- With one runner, the jobs run in the queue order.
- With N runners, each runner gets its own part of the queue. Each part keeps the queue order. The jobs leave the queue first in, first out.
--keep-on-failure skips down for a failed job. The environment stays on the runner. Use freeipa-env logs --refresh to look at it. Run down yourself before you reuse the runner.
A runner is a host that runs queued jobs. It needs podman, systemd, and a copy of the ci-freeipa tree. Each --runner SPEC names one transport.
| Spec | Transport | How the jobs run |
|---|---|---|
user@host[:port] (default) |
SSH | A runner that you set up. The ci-freeipa tree is copied to it. up, run, and down run on the runner. |
local |
Local | The control node is the runner. No SSH is used. The same steps run as local processes. You need podman on the control node. |
testing-farm |
TF | No SSH and no host you set up. The jobs of one runner run as one Testing Farm request. The guest is set up once. For podman jobs it builds the SRPM and the channel images once, then runs the presets in order. A nested-qemu job builds no image. |
All three transports use the same job steps: up, run, and down. Each job has a timeout. down has no timeout and must finish. The output files (summary.tsv, summary.md, and the transcripts) have the same form for each transport.
The remote and local job paths start with ~/. The remote shell expands them to the home directory of the SSH user. A non-root user keeps the tree out of /. The defaults --remote-ci ~/freeipa-ci/ci and --jobs-dir ~/freeipa-jobs work for any user. The local runner expands ~/ to the home directory of the local user.
Use --runner testing-farm to send the jobs to the public Testing Farm API. Each runner's jobs are one request.
Testing Farm does these things:
- It creates a new Fedora guest. You do not need a host or root access on your machines.
- It clones the FreeIPA repository under test.
- It runs the pinned plan from that checkout. The plan finds this repository's test at the pinned ref.
For podman jobs, the guest builds the SRPM and each channel image once. Then it runs the presets in order, and it reuses the images. A nested-qemu job builds no image: see Running the jobs in qemu VMs.
A preset passes only when these two things are true:
- The request completes.
- The preset's own
results.yamlshowspassed.
The request fails when the run step of any preset fails.
You need an API token. Pass --tf-token TOKEN, or set the TESTING_FARM_API_TOKEN environment variable.
Other options:
--tf-repo-url: the repository that the guest clones. The default is theremote.origin.urlof--freeipa-src.--tf-ref: the ref to check out. The default is the current branch of--freeipa-src.--tf-url: the API base URL.--tf-arch: the architecture. The default isx86_64.--tf-compose: the compose. The default isFedora-44.--tf-plan: the plan in the FreeIPA repository. The default is/ci/tmt/plans/freeipa-env.--tf-variable KEY=VALUE: an extra request variable. You can repeat this option.
Testing Farm clones over the network. Use a commit that is pushed to a public repository.
You do not supply an SRPM. The tmt plan prepare step installs these packages on the guest:
git,podman, and the autotools toolchain.- Every BuildRequires of the branch's
freeipa.spec.in.
This follows the synta pattern.
Testing Farm copies the fmf tree as plain files. So the tmt test clones the repository first. It runs git clone --recursive and then git checkout, using --tf-repo-url and --tf-ref. Then it builds the SRPM:
autoreconf -i./configurewith the flags that match the spec.make srpms
For each preset, the guest bakes the channel image from the SRPM with images/build.sh --srpm. It bakes each channel once. A channel that is already baked is reused.
Then the guest runs each preset in order. Each preset runs up, run, and down with its own --job-timeout. The request fails if any run step fails.
To use a prebuilt SRPM, pass --srpm <URL>. The URL must be HTTP or HTTPS, and the guest must be able to fetch it.
The tmt test uses result: custom. Each preset reports one result for each stage:
srpm-buildimage-buildenv-uptest-runenv-down
It also reports a parent result for the preset. Each result has its own console log. The batch has a parent result. It is fail when the run step of any preset fails.
The artifacts of each preset go to .../data/artifacts/<preset-key>/. These are the run console, nosetests.xml, and the collected per-host logs. The results.yaml file goes to .../data/<preset-key>/. You can download and check each preset separately.
freeipa-env tf-logs <request-id> downloads the artifacts of a finished request. It puts them in a local workdir. The layout is the same as for a local or SSH job. Then it runs logs. You do not need an environment file.
A batched request has one workdir for each preset. The path is <workdir>/<preset-key>/. Each one has its own logs/, stages/, and results.yaml. --html writes one results.html for each preset. A request with one preset has one flat workdir.
The request ID is printed in the queue run transcript. You can also get it from the TF API. --category, --pattern, --json, and --host work as they do for logs. --workdir DIR sets where the artifacts go. The default is ./tf-<request-id>.env. If you run the command again on the same directory, it reuses the files that are already downloaded.
freeipa-env tf-logs <request-id>
freeipa-env tf-logs <request-id> --category tests --category run --lines 20
freeipa-env tf-logs <request-id> --html # download, then make results.html
tf-logs also downloads the results.yaml of each stage. The report then shows the Stages table.
You do not need a token to download a finished request. The request state (GET /v0.1/requests/<id>) and the artifact store are public. You need a token only to submit a request with queue run --runner testing-farm.
freeipa-env queue run queues/gating.yaml \
--runner testing-farm \
--tf-token $TESTING_FARM_API_TOKEN \
--tf-repo-url https://github.com/abbra/freeipa.git \
--tf-ref modrnize-ci \
--job-timeout 21600 # one build and several runs. The default is 4 hours for each preset.
The supervisor checks the request until it ends. The batch deadline is --job-timeout. The default is 14400 seconds (4 hours) for each preset, plus a buffer for the build. If the deadline passes, the supervisor cancels the request.
The full build (SRPM, binary RPMs, and the image) runs once. Every preset in the request uses the result.
--dry-run with the testing-farm runner prints the exact batch request JSON. It does not send it. You still need a token.
If you do not pass --freeipa-src, --tf-repo-url, --srpm, or --copr, the tests use the packages that Fedora ships. Nothing is built. This repository is not a FreeIPA repository, but the request names it. The request uses its plan /tmt/plans/distro (tmt/plans/distro.fmf).
prepareinstalls onlypodmanandpython3-pyyaml.- The test comes from the same ref, so you do not change a pin.
- The request uses the origin and current branch of this checkout. You must push the branch first.
The guest builds the channel images for its own release. The compose chooses the Fedora version:
freeipa-env queue run queues/azure.yaml \
--runner testing-farm --tf-compose Fedora-45
To run that plan from another clone of this repository, use --tf-repo-url and --tf-ref. Also pass --ipa-from-distro. If you pass only --tf-repo-url, the tool expects a FreeIPA repository to build.
Add --copr OWNER/PROJECT to the same command. You can repeat this option. The same plan runs. The guest installs the IPA packages from the COPR and from the distribution together. The COPR must have builds for that Fedora version.
A job whose preset is a nested qemu environment (provider: nested, vm.backend: qemu) runs in qemu VMs started on the Testing Farm guest, not in podman containers. Submit it with the qemu plan:
freeipa-env queue run queues/qemu-copr.yaml --runner testing-farm \
--tf-plan /tmt/plans/qemu --tf-compose Fedora-45
- The request gets a
hardwaresection on its own, from the presets:virtualization.is-supported(the VMs need KVM) and memory for the VMs of the biggest job plus 4 GiB. A podman job adds nothing.--dry-runprints the request body. - The plan
/tmt/plans/qemuinstalls qemu,xorrisoandrsyncon the guest.tests/freeipa-env/qemu-prep.shthen checks/dev/kvm, makes an ssh key if the guest has none, and downloads the cloud images the presets name. It checks each image against the SHA-256 in Fedora'sCHECKSUMfile. - The runner bakes no channel image for these jobs. The IPA comes from the preset's
vm.coprandvm.guest_packages, so the compose only picks the guest OS that runs the VMs. The preset must be in this repository: the guest clones it. queues/qemu-copr.yamlis the same test thatpresets/qemu-copr-customized_ds_config_install.yamlruns locally withfreeipa-env upandrun, so a local and a Testing Farm result can be compared.
This lane runs podman jobs: it makes the channel image from the scratch build.
Fedora dist-git CI (Packit) runs a Koji scratch build for each dist-git PR. It runs the scratch build before the tests. It passes the task ID to the test request as KOJI_TASK_ID.
The test uses that build. The file tmt/tests/freeipa-env/koji-rpms.py reads the buildArch subtask results of the task. It downloads the binary RPMs from kojipkgs. It downloads x86_64 and noarch RPMs. It leaves out debuginfo and debugsource.
The channel image is made from these RPMs with build.sh --rpms. No SRPM is needed. No build image is needed. No rpmbuild runs.
The scratch build targets the PR's Fedora dist. So this lane makes the base and channel images for the dist of the guest. It uses images/build.sh --dist. base/Dockerfile uses ARG DIST. A rawhide guest uses the rolling rawhide toolbox tag.
Use this fallback for manual runs, or for a CI system without Packit. Set FREEIPA_DISTGIT=1. The test then builds the SRPM from the dist-git checkout in the plan. A dist-git repository has the spec, the patches, and a sources file with the tarball hashes. It does not have configure.ac.
tmt/tests/freeipa-env/distgit-srpm.sh does these steps:
- Reads the plan tree at
$TMT_TREE. The CI copies this tree to the guest. - Downloads each entry of
sourcesfrom the Fedora lookaside. - Runs
rpmbuild -bs.
Then the channel image is made from the SRPM with --builddep. The build image adds any BuildRequires that the PR's spec adds. The image is for the guest's dist.
Override settings:
FREEIPA_DISTGIT_DIR: the tree path. The default is$TMT_TREE.FREEIPA_DISTGIT_SPEC: the spec file name.FREEIPA_DISTGIT_PKG: the package name in the lookaside. The default is theName:of the spec.FREEIPA_LOOKASIDE_URL: the base URL of the lookaside.
The dist-git plan uses the same pin as the upstream plan. Its prepare step is smaller. The binary build runs in the build image, so the guest needs no BuildRequires.
discover:
how: fmf
url: https://github.com/abbra/ci-freeipa.git
ref: <pinned tag>
test: "^/tmt/tests/"
prepare:
how: install
package: [podman, curl, git, rpm-build]
execute:
how: tmt
environment:
# fallback source when the request has no KOJI_TASK_ID
FREEIPA_DISTGIT: "1"
FREEIPA_JOBS: '[{"key": "base-unit", "preset_rel": "presets/azure/base.yaml"},
{"key": "base-xmlrpc", "preset_rel": "presets/base-xmlrpc.yaml"}]'FREEIPA_JOBS lists the suites that run. The presets come from this repository.
The guest chooses the source in this order:
FREEIPA_SRPM_URLKOJI_TASK_ID(reuses the scratch build)FREEIPA_DISTGIT- The upstream clone and SRPM build
This section is about the podman provider's channel image. If you do not name a source, the image uses the packages from the distribution. A source is named by --srpm or build.srpm, by --rpms, or by --copr. --copr alone installs from that COPR. See the end of this section.
The image is made from the distribution's own packages. This tests a Fedora release as it was released.
images/build.sh runs dnf -y install for each IPA subpackage. It installs these packages:
- The server, with DNS and trust-ad.
- The client.
- The
python3-*packages. python3-ipatests.
The packages come from the repositories of --dist. These are the repositories that fedora-toolbox enables: fedora and updates. updates-testing is not used.
The tests that run are the tests that python3-ipatests of that release installs. So the tests always match the packages.
The image has the label source: distro. freeipa-env show prints this label with the installed builds.
--ipa-from-distro states this choice. It cannot be used with --copr or --srpm. A COPR gives dnf its own builds. The result would then not be the distribution's.
The image is a snapshot of the repositories at bake time. To get newer updates, bake the image again.
--add-ipa-package SPECadds a package to the default list.--ipa-packages 'SPECS'replaces the list.
-
One machine, a dedicated image. Use this on a host that also has a worktree or a COPR build. A channel image that is already present is never built again. So
upwould test that image. Use this command:images/build.sh --dist 45 --tag freeipa-distroIn the env file, set
image: freeipa-distro/full:45anddist: 45. To run a catalog preset with this image, copy the preset. Change those two lines. -
One machine, the channel image.
freeipa-env up ENV.YAMLorensuremakes the missing channel image for thedist:of the env.ensure --buildmakes a new image. It replaces the build that the channel tag points to. -
Queue.
freeipa-env queue run QUEUE --runner local(orUSER@HOST) makes the missing channel images on each runner. This happens before any job starts. Host runners make images for dist 44. For another release, make the channel first withimages/build.sh --dist N. -
Testing Farm. Use
--runner testing-farm --tf-compose Fedora-N. See Testing Farm transport.
To use builds from a COPR repository, add --copr OWNER/PROJECT to any command above. You can repeat it. --ipa-from-copr states this choice. dnf then finds the IPA packages in the COPR and in the distribution.
This section is about the podman provider's channel image. By default, freeipa-env uses IPA RPMs. A build farm makes them. The RPMs go into a freeipa-ci/full image, and you distribute that image.
A problem can happen with these RPMs. Their distro dependencies can change after the snapshot. For example, a newer samba enters the repository. The old IPA RPM requires the old samba, which no longer resolves. The image build then fails. The cause is not the IPA change under test.
The self-build lane prevents this problem. It compiles the IPA binary RPMs from the same git snapshot. It does this in a build image on the same host that makes the test image. The build and the test use one snapshot of the distro repository (design §3.10).
-
Control node.
scripts/make-srpms.shruns autogen and configure on the checkout. Then it runsmake srpms. The result isdist/srpms/freeipa-<ver>.src.rpm. This is the only step that needs the IPA source tree. The script uses the same configure options asmakerpms.sh. So the SRPM has the same host and layout as a localmake rpms. -
Ship. The SRPM goes to the host that runs the environment. For one machine, it is already there. For a queue, the supervisor copies it to each runner with rsync.
-
Build and bake.
images/build.sh --srpm <path>compiles the SRPM. It does this in thefreeipa-ci/build:<dist>image. That image has the base image and every BuildRequires of the spec. It is made once for each dist. The build does not rundnf builddepfor each build.The build runs
rpmbuild --nocheck --rebuild.--nocheckskips the%checkunit tests of the spec. CI runs those tests in its ownrunstep. The build then collects the new binary RPMs. It makesfreeipa-ci/full:<dist>from them. It tags the image with the channel.
-
One machine. Add a
build:block to the env file.upbuilds any missing channel image before it finds the image.freeipa-env ensuredoes only this step.name: base-xmlrpc provider: podman image: freeipa-current build: srpm: dist/srpms/freeipa-16.0.0-0.20260101gitabc1234.src.rpm # force: true # rebuild even if the channel image is already present hosts: - {role: master, name: master1}
freeipa-env ensure ENV.YAML # build the missing channel images freeipa-env ensure ENV.YAML --build # force a rebuild -
Queue. Pass the SRPM to
queue run. You can pass a file or thedist/srpms/directory. The supervisor copies it to each runner. It builds any channel image that the runner does not have. This happens before any job starts.freeipa-env queue run queues/gating.yaml \ --runner root@192.168.122.215 \ --srpm dist/srpms \ [--srpm-dir /root/freeipa-ci/srpm] [--image-timeout 9000]
A channel image that is already on the host is not changed. Only missing channel images are built. A full IPA build and image bake can take a long time.
To force a rebuild, use one of these:
- Set
build.forcein the env file. - Run
freeipa-env ensure --build. - Remove the channel image.
Without an SRPM (--srpm, or a build: block), a missing channel image is made with dnf. It uses the distribution packages. If you pass --copr, it uses the COPR repositories.
freeipa-env blend BASE PROVIDER --name NAME [--tests T ...] writes a preset or queue that is ready to run. The file is ~/.cache/freeipa-ci/generated/NAME.yaml. You then run it with run or queue run. You do not repeat any flags.
The generated files are outputs, not sources, so they are written to ~/.cache/freeipa-ci/generated/ and not to the repository. A queue names its presets as generated/NAME/<job>.yaml, relative to env/. A local run finds them in the cache. queue run copies the whole directory to env/generated/ inside each runner's copy of the tree (SSH and local runners), so a blended queue runs on any runner. Testing Farm clones the repository and gets no copy: commit a preset under env/presets/ to run it there. An env/generated/ file in the repository still works and takes precedence.
- BASE is a preset or a queue (a file with
jobs:). For a queue, each job is blended. Each one is written to~/.cache/freeipa-ci/generated/NAME/<job>.yaml. - PROVIDER is a setup file. It has keys such as
provider,inner,vm,image,dist, anddomain. Its top-level keys replace the keys of BASE.hostsandresourcesalways come from BASE, so the test's topology and memory stay. Itsnameandrunare ignored. --testsreplacesrun.testsin each generated preset.- The test's
image:names a podman image. A result that runs in VMs (nestedwithinner: external) or onexternalhosts does not carry it, unless PROVIDER setsimageitself. Such a result has no image to build or find.
freeipa-env blend presets/base-xmlrpc.yaml presets/nested-qemu-external.yaml \
--name xmlrpc-vm --tests test_xmlrpc
freeipa-env run ~/.cache/freeipa-ci/generated/xmlrpc-vm.yaml
blend checks each generated file before it writes it. A bad merge fails in blend, not during the run.
freeipa-env check [PRESETS ...] checks preset files. It does not start containers and it does not use the network. Use it as the CI check for the 1300 or more presets.
With no arguments, it checks every *.yaml under env/presets/, and the top-level presets. To check fewer files, pass files or directories.
Each preset must parse as an EnvSpec. It must also pass the checks of its provider:
provider: podman. Each containerized host must find an image. A channel name must be a known channel:freeipa-current,freeipa-next, orfreeipa-previous. A name that is neither a channel nor an image reference is reported as a likely typo.provider: nested. Thevm:block must use a known VM backend.innermust name a known provider.provider: external. Each host must have an address.
check prints one line for each failing preset, then a summary. The exit code is 0 only when every preset is valid and no path is missing.
Add the check to your pull request gate. A preset change then cannot break a queue without notice.
freeipa-env check # run from the repository root. It checks all presets and exits non-zero on any problem.
This runs the PRCI customized_ds_config_install test on two VMs from two Fedora releases. The test asks for master1 and replica1. nested-qemu-mixed.yaml puts master1 on the F44 cloud image (vm.vms). Every other host gets the default F45 image. Both VMs get the ipa-distro packages from their own release, including firewalld.
Download the two images first. Verify them against the CHECKSUM file in the same directory:
mkdir -p ~/.cache/freeipa-ci/images
# Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2 and ...-45-1.4.x86_64.qcow2
# from https://dl.fedoraproject.org/pub/fedora/linux/releases/{44,45}/Cloud/x86_64/images/
freeipa-env blend prci/nightly-latest-testing/customized_ds_config_install.yaml \
presets/nested-qemu-mixed.yaml --name mixed-cds
freeipa-env up ~/.cache/freeipa-ci/generated/mixed-cds.yaml
freeipa-env run ~/.cache/freeipa-ci/generated/mixed-cds.yaml
freeipa-env down ~/.cache/freeipa-ci/generated/mixed-cds.yaml
Check the result: master1 must report freeipa-server-*.fc44, and replica1 must report freeipa-server-*.fc45. The FreeIPA server build must match its release in both VMs. If not, the VM image is wrong.
Then investigate the logs. This is the walkthrough for the run above, with output from a passing run:
freeipa-env logs ~/.cache/freeipa-ci/generated/mixed-cds.yaml # 1. triage: which categories have lines
freeipa-env logs ~/.cache/freeipa-ci/generated/mixed-cds.yaml --category tests # 2. xunit: pass/fail and failing messages
freeipa-env logs ~/.cache/freeipa-ci/generated/mixed-cds.yaml --category ds --lines 3 --tail 0
freeipa-env logs ~/.cache/freeipa-ci/generated/mixed-cds.yaml --category ca --host master1.ipa.test --lines 2
- The summary shows the
dscategory with 93 matched lines andcawith 114. Most lines are expected noise.Unknown attribute nssslapd-maxbersize will be ignoredrepeats on every 389-ds start. Look for lines that do not repeat. testsreadslogs/collected/master1.ipa.test/nosetests.xml. A failure shows its message and traceback tail under the test name.2 tests: 0 failedmeans the xunit file is the only evidence you need.dsmatches on- ERR -and- CRIT -. Eacherrorsfile of the per-test trees is listed separately. Theno new matching linesrows are the same lines seen in an earlier test, so they are not repeated.caformaster1shows the spawn logs anddebug.<date>.logof the CA. ASEVEREline such asUnable to create profile: Profile already existscan appear in a passing run. Check the timestamp against the test you are debugging.
To find one failing test's logs, look up its directory under logs/collected/<host>/ipa-env/logs/. Then narrow with --pattern:
freeipa-env logs ~/.cache/freeipa-ci/generated/mixed-cds.yaml --category ds --pattern 'krb5_get_default_realm|Attribute' --lines 5 --tail 0
For a single failing test on a bad run, report writes one shareable file:
freeipa-env report --workdir ~/.local/state/freeipa-env/qemu-mixed-prci-nightly-latest-testing-customized_ds_config_install -o /tmp/cds.html
The Python tests for the infrastructure helpers live in env/tests/. They need no VMs or containers. Run them from env/:
python3 -m pytest tests -q
They cover the guest package list (-name subtraction, plain-name addition, duplicates) and the log analysis dedupe (scan_file, LogStore.matched_lines).
- The podman provider uses one network for each env:
10.89.0.0/24and2001:db8:1::/64. Only one env can run on each host in version 1. If these defaults conflict with a podman network that another project made, setsubnet_v4:andsubnet_v6:in the preset. - In base mode,
runcopiesscripts/run-base-tests.shinto the master container and runs it there. The JUnit report is copied back afterward. - Uninstall is part of the workflow: install, test, uninstall. The container is removed only by
down, after the uninstall finishes.