fatal: /run belongs to uid 0 instead of 999

[rauanmayemir]

Bumping the image from v3.1.6.2 to v3.2.0.2 resulted in containers failing with:

/package/admin/s6-overlay/libexec/preinit: fatal: /run belongs to uid 0 instead of 999 and we're lacking the privileges to fix it.
s6-overlay-suexec: fatal: child failed with exit code 100

I'm running the pod under a non-privileged uid 999 with read-only root fs, relevant dirs have all the proper chowns, everything worked fine before the upgrade.

[skarnet]

Do you have a /run on your image? How is it mounted?

[rauanmayemir]

I do have it, it's an emptyDir mount:

- mountPath: /run
  name: scratch-volume
  subPath: run

Upon building the image, I verified that it's properly owner by my user. But in runtime /run's owner is changed to root, even though I've set pod's security context to the least privileged.

[skarnet]

OK, two more questions (to inform my diagnosis):

• If your rootfs is read-only, are you running the container with S6_READ_ONLY_ROOT set to 1?
• If you run the container with S6_VERBOSITY set to 2 or more, early on you should see preinit lines with container permissions and /run permissions. Can you please reproduce them here?

Thanks.

[rauanmayemir]

Yes, I set S6_READ_ONLY_ROOT to 1. Will upgrade again tomorrow and collect the details.

[dkorunic]

I can confirm that this happens and preinit exits with fatal failure even with S6_BEHAVIOUR_IF_STAGE2_FAILS=1 set. An example on K8s that uses /run as emptyDir same as the above:

/package/admin/s6-overlay/libexec/preinit: info: read-only root
/package/admin/s6-overlay/libexec/preinit: info: writable /run. Checking for executability.
/package/admin/s6-overlay/libexec/preinit: info: container permissions: uid=1000 (ubuntu), euid=1000, gid=1000 (ubuntu), egid=1000
/package/admin/s6-overlay/libexec/preinit: info: /run permissions: uid=0 (root), gid=0 (root), perms=oxoworgxgwgruxuwuros
/package/admin/s6-overlay/libexec/preinit: fatal: /run belongs to uid 0 instead of 1000 and we're lacking the privileges to fix it.
s6-overlay-suexec: fatal: child failed with exit code 100

[skarnet]

I see.

So it appears that your container manager mounts /run belonging to root:root with permissions 1777, just like a /tmp temporary filesystem would be mounted.

Unfortunately, this is incorrect. The difference between /run and /tmp is that only root - or, in this case, only the uid running the container - should be allowed to create files under /run. Doing otherwise opens the system up to denial of service attacks (or, depending on how programs use files under /run, prediction attacks, which can be worse).

The current version of s6-overlay detects that the /run permissions are incorrect and refuses to run. Previous versions ran, but your system was insecure. I realize this can be perceived as inconvenient, but I believe the change to be an improvement.

The fix is to find where your container manager mounts /run and change how it is mounted. It should have the uid and gid of the process that runs the container, and have 0755 permissions.

If that proves too difficult, I can probably add an option in a future version of s6-overlay that allows it to run on insecure /run configurations, but that won't be right away, and that won't be a solution, just a dirty workaround. Please try and fix your container manager instead.

[dkorunic]

@skarnet At least in K8s world, sadly it's not possible to change emptyDir ownership when running in rootless container (well, it's possible in terms of adding additional initContainers privileged container chowning /run, but that's in many ways regarded as horrible practice). Directory will be created in-memory with UID 0 owner and the only thing we can change is group ownership through pod-level fsGroup field (as it can be seen below with gid=1000 for /run), but preinit still throws a fatal error:

/package/admin/s6-overlay/libexec/preinit: info: container permissions: uid=1000 (ubuntu), euid=1000, gid=1000 (ubuntu), egid=1000
/package/admin/s6-overlay/libexec/preinit: info: /run permissions: uid=0 (root), gid=1000 (ubuntu), perms=oxoworgxgwgruxuwurosgs
/package/admin/s6-overlay/libexec/preinit: fatal: /run belongs to uid 0 instead of 1000 and we're lacking the privileges to fix it.

Since we use s6 extensively in K8s, it would be great if we could silence this or just have it as non-fatal warning, since as long as container runs as non-UID 0, everything inside is also non-root (and mapped to a single user) and /run tmpfs mount security is really not relevant, given all processes belong to the same UID anyway.

[skarnet]

It is true that the mapping to a single uid makes the /run permissions less problematic. I'll see what support I can add for this use case - but it is still a defect in k8s that is worth reporting. /run should be mounted as the container user, always.

[skarnet]

If you have the ability to build s6-overlay yourself, could you please test the latest commit? It should print a warning, but still run your container. If it works for you, I'll cut a new release.

[dkorunic]

@skarnet I can confirm that build from the latest tip (42e9021 and 1705b0f) prints a warning but proceed booting as expected, as seen below:

/package/admin/s6-overlay/libexec/preinit: info: read-only root
/package/admin/s6-overlay/libexec/preinit: info: writable /run. Checking for executability.
/package/admin/s6-overlay/libexec/preinit: warning: /run belongs to uid 0 instead of 1000, but we can still work in single-uid mapping.

Thank you! I can confirm that this helps.