Signed built-in sysexts

[danzatt]

Sign OS-dependent sysexts using an ephemeral key which is baked into the image.

The signed sysexts are generated using systemd-repart and they are built as a DDI with a dm-verity partition and signature partition.

TODO: determine if we enable the CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG in this PR or separate (@chewi was planning to use this for the kernel move, but the plans have shifted now).

How to use

Just enable any OS dependent sysexts, they should be verified using dm-verity has signature when loading.

Testing done

When you boot the built image, all built in sysexts (including OEM) should be signed. When booting, the systemd-sysext will merge them and verify the signature automatically. To merge again with explicitly stricter policy (which forces signed sysexts only), you can use the following command:
systemd-sysext refresh --image-policy="root=verity+signed+absent:usr=verity+signed+absent"

See related sysext-bakery PR#175

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[chewi]

Aside from the security model, I have one concern. systemd-repart only gained the ability to apply compression to erofs in v257, but as far as I can tell, this is only configurable through the repart.d files, which aren't used when creating DDIs. The built-in sysexts will therefore no longer be compressed. Maybe this doesn't matter because USR is already compressed via btrfs. It might even be a good thing as we avoid the double compression. Please compare the compressed size on a live Flatcar instance against an older instance.

We'll presumably need to do work on systemd to be able to publish signed and compressed sysexts in the bakery. CC @t-lo.

[chewi]

Suggested change

	sys-fs/erofs-utils
	sys-fs/erofs-utils

[chewi]

This could have consequences. What needs this, and could it be done against that specific package?

[chewi]

I don't think you meant to include this, and this certificate should not be used in production builds.

Suggested change

config_update "CONFIG_SYSTEM_TRUSTED_KEYS=\"/usr/share/sb_keys/shim.pem\""

[chewi]

The certificate you're signing with is written to /usr/lib/verity.d, so you don't need this. The verification is done in userspace.

However, I have doubts about this security model. You're verifying sysexts in USR with a certificate in USR. Does that add anything, especially when USR's own integrity is already enforced by verity? Having the certificate in the kernel keyring may be a stronger model, but I'm not the best person to ask. @jepio, your input here would be appreciated.

[pothos]

If we use this for the extensions that are downloaded on demand, it makes sense. For the ones itself stored in /usr it might be useful to have them also signed for enforcing a policy.

[chewi]

We were going to have bakery sysexts use certificates in /etc/verity.d because we felt more flexibility was needed here. What if we need to rotate the certificate? Newer sysexts are supposed to work on older Flatcar releases.

[chewi]

systemd-repart only gained the ability to apply compression to erofs in v257, but as far as I can tell, this is only configurable through the repart.d files, which aren't used when creating DDIs.

Actually, I got that slightly wrong. The files in /usr/lib/repart.d aren't used when creating a DDI, but the tool internally uses files from /usr/lib/systemd/repart/definitions. I think it might even pick up files from /etc/systemd/repart/definitions. It doesn't seem to be documented, but it looks like you could just add your own configuration here.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/19230665191

[danzatt]

I've just tested using erofs compression and without (using SYSTEMD_REPART_MKFS_OPTIONS_EROFS="-zlz4hc,12 -C65536 -Efragments,ztailpacking"). There seems to be negligible (8MB difference), so I'm not sure if it's worth the double compression.

Uncompressed

flatcar_production_qemu_uefi_image.img 503M (526516224)

# df -h /usr/
Filesystem       Size  Used Avail Use% Mounted on
/dev/mapper/usr 1016M  546M  331M  63% /usr

# btrfs filesystem usage /usr
Overall:
    Device size:		1015.99MiB
    Device allocated:		 684.00MiB
    Device unallocated:		 331.99MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 542.51MiB
    Free (estimated):		 466.62MiB	(min: 466.62MiB)
    Free (statfs, df):		 330.94MiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   2.87MiB	(used: 0.00B)
    Multiple profiles:		        no

Data+Metadata,single: Size:680.00MiB, Used:542.51MiB (79.78%)
   /dev/mapper/usr	 680.00MiB

System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
   /dev/mapper/usr	   4.00MiB

Unallocated:
   /dev/mapper/usr	 331.99MiB

# ls -lah /usr/share/flatcar/sysext/
total 328M
drwxr-xr-x. 1 root root   80 Oct 23 13:44 .
drwxr-xr-x. 1 root root  166 Oct 23 13:47 ..
-rw-r--r--. 1 root root 156M Oct 23 13:44 containerd-flatcar.raw
-rw-r--r--. 1 root root 192M Oct 23 13:44 docker-flatcar.raw

Compressed

flatcar_production_qemu_uefi_image.img 500M (523436032)

# df -h /usr/
Filesystem       Size  Used Avail Use% Mounted on
/dev/mapper/usr 1016M  538M  331M  62% /usr

# btrfs filesystem usage /usr
Overall:
    Device size:		1015.99MiB
    Device allocated:		 684.00MiB
    Device unallocated:		 331.99MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 534.66MiB
    Free (estimated):		 474.64MiB	(min: 474.64MiB)
    Free (statfs, df):		 330.94MiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   2.69MiB	(used: 0.00B)
    Multiple profiles:		        no

Data+Metadata,single: Size:680.00MiB, Used:534.66MiB (78.63%)
   /dev/mapper/usr	 680.00MiB

System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
   /dev/mapper/usr	   4.00MiB

Unallocated:
   /dev/mapper/usr	 331.99MiB

# ls -lah /usr/share/flatcar/sysext/
total 150M
drwxr-xr-x. 1 root root  80 Oct 23 14:12 .
drwxr-xr-x. 1 root root 166 Oct 23 14:15 ..
-rw-r--r--. 1 root root 86M Oct 23 14:12 containerd-flatcar.raw
-rw-r--r--. 1 root root 85M Oct 23 14:12 docker-flatcar.raw

[chewi]

Interesting, thank you! I was just a little surprised by the results, as I thought plain zstd might come out smaller, but I discussed them with Copilot, and it thought they seemed legitimate. I fully agree that the double compression isn't worth it, especially when it only makes 3MB difference to the QCOW2.

[danzatt]

I think, I've fixed most of the stuff now. To sum it up:

• we don't use FS compression for the built-in sysexts (they'll get compressed by btrfs when placed to the /usr filesystem)
• switch format of the built-in sysexts from raw squashfs to DDI with erofs
• sign all sysexts with ephemeral key (available only during Flatcar build) and place the certificate in /usr/lib/verity.d
  • NOTE: this uses userspace verification, this might cause futre problems if we decide to support IPE, so we might have to switch to baking the sysext key into kernel keyring and switch to kernel-space verification
• this PR doesn't add much security per se, as the content of /usr, including the built-in sysext has already been verified using dm-verity
  • however, if even built-in sysexts are signed, it opens way to enforce signed-only policy system-wide (including bakery/custom sysexts)
  • it pulls in three packages erofs-utils, xxhash (dependency of erofs-utils) and virtual/zlib (Gentoo switched to virtual package for zlib)

[pothos]

Can you start GitHub Actions and a Jenkins build (and link it in the PR description)? That would be nice for testing.

[pothos]

Are the Flatcar extensions such as podman, python, zfs also covered?

[pothos]

Because the PR also says All OS-dependent sysexts which would include podman and so on but the PR title is built-in as well, so the wording needs to be fixed to state what is covered and what not.