podman in production: unpriviliged user vs. --userns=auto

[tobwen]

Based on this discussion, @rhatdan stated that podman run --userns=auto might be a better solution for containers in production launched by podman. For sure, when starting it as a privileged user, you're way more flexible.

But is the isolation in a unique UserNS as secure as running podman from an unprivileged user?

[rhatdan]

It is actually more secure. If you run two containers as a rootless user, they run in the same user namespace so they can attack each other from a User Namespace point of view.

If you run two containers as root with podman run --userns auto, then they run in unigue user namespace and are isolated.

Rootless containers are great for containers run by users on a system, but if you are just running containers on a server, then --userns=auto is a more secure solution. (I plan on writing a blog on this).

[sbrivio-rh]

Don't put all of your trust into SELinux when you're outside of RedHat. Seems like many Linux distributions have issues with it.
Article: https://klecko.github.io/posts/selinux-bypasses/

...which explains how to disable SELinux on some Android devices. 😕

@sbrivio-rh: in a rootfull podman networking context, what is the worst that malicious packets can do if SELinux is enabled and correctly configured?

Exactly the same as malicious packets can do without Podman, or without SELinux. You can spoof ARP, craft semi-random things to crash/DoS routers, etc. Network isolation is still there at Layer-2, but all you need to do is to get control over the network interface in the container to send out whatever you want. I covered that quickly in slide 6 of this deck for this presentation.

[Luap99]

But doesn't spoofing and crafting require CAP_NET_RAW? This is not given to a container by default and I see very little reason to do so and if an application really needs this they most likely cannot use pasta/slirp4netns anyway unless they only use it for local connections inside the netns.

[tobwen]

...which explains how to disable SELinux on some Android devices. 😕

I knew the comment was coming, so I copied the discussion page 😃. Some users say that SELinux unfortunately isn't always implemented reliably.

[sbrivio-rh]

But doesn't spoofing and crafting require CAP_NET_RAW? This is not given to a container by default

Sure, but if you start the container as root, you rely on Podman to drop it (and here I'm assuming a bug in Podman). Or one could also pass --privileged for whatever other reason and miss this aspect.

On top of that, what is sent to the container namespace (not necessarily to processes running into it) is not as restricted.

I knew the comment was coming, so I copied the discussion page 😃. Some users say that SELinux unfortunately isn't always implemented reliably.

Yeah, I read that as well, but... it starts with a mention of SELinux on Debian as an experiment. Now, sure, some people spent effort to get that somehow working, but the "native" LSM on Debian is AppArmor. It's not so much of a matter of SELinux implementation, it's rather about choices, configuration and integration.

[rhatdan]

SELinux by default implements unconfined networking, we rely on network namespace and dropped capabilities to protect the network.

If you start with the assumption that Podman is broken then running containers as root is a risk. Same as with the Kernel is broken or systemd or ...

SELinux confines activity within the container, especially about the file system, and using kernel file systems.

[tobwen]

Can you please remember to write a short blog post stating that a rootful podman is safer than a rootless podman? I'm really struggling to convince other users of this. Maybe a short notice in the README would be enough?

[Luap99]

Well this sentence is not true. We explicitly talk about --userns auto is more secure. This is a very important difference.
The secure part is that with userns auto all containers run in a different user namespace (different uids on the host). You can achieve the same as rootless user so I wouldn't say rootful is more secure than rootless.

[tobwen]

Sure, --userns auto is mandatory. However, I assumed that there are other security functions in the kernel that could currently only be accessed by root. It is wonderful that this has now been clarified.

[rhatdan]

I have a blog in progress, the problem is it is arguable which is more secure.
$ sudo podman run --userns=auto ... Runs each containers in a separate user namespace where each container is running with different UIDs and almost assuradly does not match any other UIDs on the host. BUT the podman command runs as root which means it pulls images as root. Pulling images as root can be risky, because flaws in the pulling could (have in the past) cause root to write anywhere.
$ podman run ... Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

[tobwen]

$ podman run ... Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

Would this also be the case if you run each container under a different user? Or would a different namespace then be assigned?

[rhatdan]

If you run each container under a different user and managed the /etc/subuid and /etc/subgid files to maintain separation, this would be safer, although they could access content in user homedir.

$ podman run --userns=nomap ...

Would solve this problem.

But this is way more complex. You could also setup a Huge /etc/subuid range for a user and then run lots of containers for that user with --userns=auto ... Which is probably the most secure.

[cgwalters]

One thing related to this...what would I think be quite cool is integration with systemd DynamicUser=yes at least for quadlet containers (we'd still need to mount the image as privileged, so doing this would probably require some lower level systemd integration).

[cgwalters]

Yeah, the containers user here is IMO pretty suboptimal. Is something actually intended to create that user today automatically?

[pomology]

@rhatdan, were you ever able to write the blog post above regarding the security implications of rootfull podman as potentially more secure vs rootless? I'd love to read it if you can share a link. Thank you!

[rhatdan]

@pomology Send me and email and I can probably preview what I have now to you.

[pomology]

@rhatdan Thank you! I sent you an email just now.

[tobwen]

I'm also interested. Do a MEAP, I'm a legit owner of https://www.manning.com/books/podman-in-action

[rhatdan]

Send me an email, and I will send the preliminary doc to you.

[kavishgr]

I'm also interested. I sent you an email.

[rdbisme]

Hello, what's the best practice here nowadays?

I'm currently deploying containerized services creating a dedicated non-root user on my system per "pod". Enable lingering and running the container within that dedicated user in rootless mode.

How does it look? Would it be better to run them instead with root + --userns=auto?

[giuseppe]

from a security PoV at runtime they behave in the same way. The main difference of using --userns=auto compared to a different system user per pod is that the code for pulling the image and setting up the container is also running in a user namespace. So the "user per pod" setup is slightly safer, than running as root with --userns=auto`. The cost is also higher though, since you are not able to share storage among different users

[videoMonkey]

I was very excited at the thought of not creating a user per pod but I am running into a problem if I run sudo podman run -d --name vaultwarden --userns=auto -p 8000:80 docker.io/vaultwarden/server:latest I get the following feedback

ERRO[0000] Cannot find mappings for user "containers": no subuid ranges found for user "containers" in /etc/subuid
Error: creating container storage: not enough unused IDs in user namespace

and maybe I need to read more about how this all works, but I tried setting subuid's and subgid's for the user 'container', that did nothing, I tried doing it for root and then realized root already had a lot. I tried running the command without sudo, that worked.

I was hoping to have an uncomplicated and secure way to have multiple admins be able to tinker with containers and this seemed to be that promise. If i could get it going....

[sbrivio-rh]

Could that be https://bugzilla.redhat.com/show_bug.cgi?id=2382662?

[vmsh0]

ERRO[0000] Cannot find mappings for user "containers": no subuid ranges found for user "containers" in /etc/subuid
Error: creating container storage: not enough unused IDs in user namespace

What you're saying is correct ("I was hoping to have an uncomplicated and secure way to have multiple admins be able to tinker with containers and this seemed to be that promise.").

But this functionality requires some initial setup in the operating system for the users running the containers. You should read subuid(5) and set up proper mappings, as reported by the error message. This can also be easily integrated in user creation (and it is, in some distros.)

[vmsh0]

I.e., "auto" does not mean that the system configures itself automatically. "auto" means that a suitable ID range is selected from the available range.