[BUG] AL2023 with systemd 252 cause docker not able to run systemd on cgroupv2

[peterzhuamazon]

In old AL2 we have systemd 219 on the host with selinux disabled, which allows to use cgroupv1 for systemd to run in docker.

In new AL2023 we have systemd 252 on the host with selinux enabled by default, with cgroupv2 that cause these errors to show:

Failed to connect to bus: No such file or directory

...


$ docker run --rm -it --entrypoint=/usr/sbin/init -u root --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro opensearchstaging/ci-runner:ci-runner-rockylinux8-systemd-base-integtest-v3
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.33:  No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

There are two ways to resolve this:

1. Revert back to the old cgroupv1: Unable to run systemd in docker with ro /sys/fs/cgroup after systemd 248 host upgrade moby/moby#42275 (comment)
2. Set cgroup rw and use cgroupns=host: Container failling to start izer-xyz/3cx-docker#4

Additionally on AL2023, simply alter the /etc/selinux/config or setenforce will not change the behavior despite selinux showing sestatus as disabled on AL2023.

Use this, which needs to be set on the lib/compute/agent-nodes.ts in opensearch-ci repo.

sudo grubby --update-kernel ALL --args selinux=0

Then use this:

$  docker run --rm -it --entrypoint=/usr/lib/systemd/systemd -u root --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cgroupns=host opensearchstaging/ci-runner:ci-runner-rockylinux8-systemd-base-integtest-v3

Thanks.

[peterzhuamazon]

PRs:

• Disable selinux by default on AL2023 Host opensearch-ci#351
• Update ami ids for AL2023 selinux #351 opensearch-ci#352
• Update systemd images with new user setups and related jenkinsfile changes to support AL2023 #4050
• Update rpm validation and validate workflows docker args for AL2023 #4054

[peterzhuamazon]

More related:

• WIP: Make /sys/fs/cgroup rw moby/moby#42043

[peterzhuamazon]

We have resolved this issue with both deb and rpm fixed in IntegTest now.

Thanks.