Skip to content

init robustness#315

Open
aevykk wants to merge 17 commits into
gyroidos:mainfrom
aevykk:cml-boot-robustness
Open

init robustness#315
aevykk wants to merge 17 commits into
gyroidos:mainfrom
aevykk:cml-boot-robustness

Conversation

@aevykk

@aevykk aevykk commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Rewrite the init script for robustness:

  • wait on devices deterministically instead of using sleeps
  • Program defensively: set -eu, thereby fail if any critical error occurs (e.g. mounts failing, ima policy install fail)
    • Any acceptable failure condition thereby needs to be handled explicitly
    • Cases that are unhandled will lead to init aborting, and thereby become known
    • We thereby now actually properly check all exit codes
    • POSIX shell has no -o pipefail, handle constructs affected explicitly
  • quote properly
  • rewrite constructs such as cat ${src_dev} | pv -s ${size} > /dev/mapper/${CRYPTO_HDD}
  • make more use of ash builtins and thereby reduce the PID space pollution
  • make sure that mountpoints are present before trying to mount

aevykk added 4 commits July 7, 2026 12:33
Currently, init script assembly will break on the first file containing
a space. Further, `ls` is not a stable way of listing files. Improve the
unstable behavior.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…ric wait_for helper

Add a generic wait_for helper that allows to block until a condition is met.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…nt-devices},: Data partition mount behavior

The init tries to mount a partition labeled "gyroidos". If this fails,
we fall back to mounting label "trustme". The outcome is never checked,
and in the worst case, init proceeds without the mount. The userdata
partition suffers from the same behavior.

Wait for one of the labels to become available. Continue to mount the
label available, with "gyroidos" taking precedence. Fail loud if the
mount fails. Implement the same sane mounting behavior for the userdata
partition.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…rden guestos image mount

Replace the `find -name … -exec sha256sum -c … -print -quit` one-liners
that locate modules.img/firmware.img with a small pure-shell helper that
globs the guestos store and hash-checks each candidate. This drops the
reliance on busybox find's -quit/-exec (not guaranteed by the poky-tiny
busybox defconfig).

Also make failures diagnosable instead of a silent boot death: fail loud
with an explicit message and exit 1 when no image matches the trusted
hash or the loop-mount fails (previously an empty result collapsed the
unquoted mount and hit a bare exit), create the /lib/modules mountpoint
before mounting, and quote the image paths.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
@aevykk aevykk force-pushed the cml-boot-robustness branch from 71cb756 to a89faf8 Compare July 7, 2026 10:36
`mountpoint` is not part of our busybox build. Replace it with a shell
helper to be able to check for mountpoints.
Verify the data-partition mount by the mount exit code (if ! mount
"$data_dev" …) instead of mountpoint; pick the label path into data_dev
first.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
@aevykk aevykk changed the title Cml boot robustness DRAFT: Cml boot robustness Jul 7, 2026
…lpers, guard udev nudges

Prepare the early-boot helpers for `set -eu`: use POSIX function syntax
instead of the `function NAME {` bashism, and treat the udev nudges in
scan_devices as best-effort (|| true) -- settle timeouts and the like must
not abort the boot. Guard the init_ascii banner likewise. No observable
change while the init still runs with errexit off.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
@aevykk aevykk force-pushed the cml-boot-robustness branch from fb05efc to b23ff07 Compare July 7, 2026 12:10
aevykk added 10 commits July 7, 2026 14:18
…d for set -eu

Prepare the setup stage for `set -eu`. The core pseudo-filesystems
(/proc, /sys, /dev, /run) are required for the init to function, so mount
them in a loop that aborts loudly if any fails instead of limping into a
broken boot. Make the IMA policy load loud-fatal (security-critical),
which also surfaces a failed securityfs mount. Treat the genuinely
best-effort steps -- securityfs, /dev/pts, the printk quieting and
udevd --daemon -- as non-fatal (|| true); a dead udev is caught later by
the wait_for device waits.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…covery, prefer builtins

Prepare the device-mount stage for `set -eu` and drop external commands in
favor of busybox shell builtins:

- find_verified_image: read the trusted digest with `read` instead of `cat`,
  and strip the sha256sum filename field with `${sum%% *}` instead of piping
  through `cut` -- only the unavoidable sha256sum remains. An unreadable
  candidate is skipped rather than fatal.
- Guard the modules/firmware assignments in an `if !` so `set -e` cannot abort
  at the assignment before the FATAL check runs.
- `for i in {1..4}`: busybox ash does not brace-expand this, so it scanned
  once; use an explicit list so it scans four times as intended.
- `mount -a` is best-effort (fstab has optional entries like efivarfs that
  legitimately fail on non-EFI); the critical mounts are handled loudly above.
- Installing the c0 container template is critical: fail loud if the cp fails
  rather than limp on into a cmld that cannot start c0.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…-eu guards

- 50: mount -a is best-effort -- fstab has optional entries (e.g. efivarfs on
  non-EFI) that legitimately fail, and mount -a reports failure if any entry
  fails; tolerate it so it does not abort under set -eu.
- 99: quote the LOGTTY path in the final exec redirect.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…e extdata branch

Replace the `mount ...; if [ $? ]` pattern with `if mount ...; then`. Under
set -e the bare mount would abort the boot when the optional extdata
filesystem is absent, instead of taking the "No extdata fs supplied" branch
that falls back to a plain bind. The /data bind mounts stay unguarded: they
are critical, so a failure aborts loudly rather than being swallowed.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…afe crypto stage

Behavior-preserving rewrite of the TPM/dm-crypt stage in preparation for set -eu:

- Initialize do_otf/src_dev, which are read on the normal encrypted path
  where they would otherwise be unset (set -u abort).
- Resolve the disk labels with readlink + the POSIX ${x##*/} instead of
  basename $(readlink ...), and guard an absent label -- which is expected
  (it is the "derive partition 2" signal) -- so set -e does not abort on it.
- Replace the three `mount ...; if [ $? -ne 0 ]` blocks with `if ! mount`,
  preserving the AES-128 legacy fallback and the post-OTF mount.
- Drop the useless cat: pv -s "$size" "$src_dev" > /dev/mapper/... .
- POSIX cleanups: ${BOOT_HDD%?}2 instead of bash-only ${::-1}, `=`/`-z`
  instead of `==`, and `blkid` instead of echo "$(blkid)".

Operational commands (tpm2_control, the mounts, pv) are left unguarded so a
failure aborts loudly under set -eu.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…ssing 9p backend

The dev-only cmld hot-swap mounts a virtio-9p backend (extcmld) that is not
present on non-VM boots. Make mounting it a handled branch (note and skip)
rather than an operation that, once set -eu is on, would abort the boot of a
DEVELOPMENT_BUILD image that has no 9p backend attached. The binary copies
inside the mounted branch stay unguarded (critical). 85 also collapses the six
repeated per-binary if/cp blocks into a loop.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…mbled init

Emit `set -eu` as the second line of /init so any unhandled command failure or
unset-variable reference aborts the boot loudly instead of silently continuing
into a broken state. This is now safe: the preceding commits made every
fragment set -eu-clean -- critical operations fail loud, genuinely optional
ones are guarded with `|| true`, `cmd; if [ $? ]` patterns became `if ! cmd`,
and conditionally-set variables are initialized.

POSIX shell has no pipefail, so the pipelines that matter are guarded
explicitly rather than relying on $?.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
Replace the scan_devices call (which re-triggered udev and slept) with a single
proper coldplug: trigger subsystems then devices, then settle once. udevd
processes any later uevents on its own, so re-triggering is unnecessary -- the
per-device waits in later stages block on the actual device instead.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…vices

Remove scan_devices (which re-triggered udev and slept 4s per retry) and
re-purpose wait_for to poll the actual condition once per second, aborting
loudly on a hard timeout. After the one-time coldplug (fragment 20) udevd
processes new uevents on its own, so the waits just block on the real device
node / by-label symlink instead of re-triggering udev on a hunch.

- 10: wait_for signature `DESC MAX NUDGE CHECK` -> `DESC TIMEOUT CONDITION...`.
- 30: boot-label loop, tpm2d control-socket wait and the dm-crypt mapper waits
  become wait_for. The socket/mapper loops were previously unbounded (hang
  forever on a missing device); they are now bounded and fail loud.
- 40: drop the scan_devices nudge from the data-partition-label wait.
- 60: remove the redundant pre-`mount -a` scan loop; update the /mnt/userdata
  wait to the new signature.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
…string

Add missing quoting of the tty path.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
@aevykk aevykk changed the title DRAFT: Cml boot robustness Cml init robustness Jul 7, 2026
@aevykk aevykk changed the title Cml init robustness CML init robustness Jul 7, 2026
@aevykk aevykk changed the title CML init robustness init robustness Jul 7, 2026
Handle $modules and $firmware potentially being unset.

Signed-off-by: Leander Seidlitz <leander.seidlitz@aisec.fraunhofer.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant