problems with running docker with --memory flag

[marrip]

Bug description

When I set a --memory flag for a container inside the gitpod container I get:

failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in threaded mode: unknown
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in threaded mode: unknown.

Steps to reproduce

1. start gitpod container: docker run -it --rm --privileged gitpod/workspace-base:2024-10-31-10-31-39
2. start docker daemon sudo dockerd &;
3. start container sudo docker run -it --rm --memory=6m ubuntu:20.04 bash

Expected behavior

running the ubuntu container the same way it runs without the --memory flag

Example repository

No response

Anything else?

I got some other warnings/error messages:

$ sudo dockerd &                                                                                                                                                                                                                    [1] 17                                                                                                                                                                                                                                       gitpod ~ $ INFO[2024-11-01T08:56:54.186717318Z] Starting up                                                                                                                                                                                  INFO[2024-11-01T08:56:54.187411097Z] containerd not running, starting managed containerd                                                                                                                                                     INFO[2024-11-01T08:56:54.188830776Z] started new containerd process                address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=47
INFO[2024-11-01T08:56:54.218312995Z] starting containerd                           revision=7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c version=1.7.22
INFO[2024-11-01T08:56:54.249287864Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1                                                                                                                INFO[2024-11-01T08:56:54.249437208Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2024-11-01T08:56:54.249459487Z] loading plugin "io.containerd.event.v1.exchange"...  type=io.containerd.event.v1                                                                                                                        INFO[2024-11-01T08:56:54.249473457Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1                                                                                                                       INFO[2024-11-01T08:56:54.249702376Z] loading plugin "io.containerd.warning.v1.deprecations"...  type=io.containerd.warning.v1
INFO[2024-11-01T08:56:54.249720488Z] loading plugin "io.containerd.snapshotter.v1.blockfile"...  type=io.containerd.snapshotter.v1                                                                                                           INFO[2024-11-01T08:56:54.249784232Z] skip loading plugin "io.containerd.snapshotter.v1.blockfile"...  error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1                                                       INFO[2024-11-01T08:56:54.249796187Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1                                                                                                               INFO[2024-11-01T08:56:54.249927205Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2024-11-01T08:56:54.249942812Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
INFO[2024-11-01T08:56:54.249955075Z] skip loading plugin "io.containerd.snapshotter.v1.devmapper"...  error="devmapper not configured: skip plugin" type=io.containerd.snapshotter.v1                                                        INFO[2024-11-01T08:56:54.249964203Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2024-11-01T08:56:54.250049306Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1                                                                                                           INFO[2024-11-01T08:56:54.250261619Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1                                                                                                                 INFO[2024-11-01T08:56:54.250344085Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2024-11-01T08:56:54.250358081Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1                                                                                                                     INFO[2024-11-01T08:56:54.250449489Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
INFO[2024-11-01T08:56:54.250501574Z] metadata content store policy set             policy=shared                                                                                                                                             INFO[2024-11-01T08:56:54.251973635Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2024-11-01T08:56:54.252016289Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2024-11-01T08:56:54.252042202Z] loading plugin "io.containerd.lease.v1.manager"...  type=io.containerd.lease.v1
INFO[2024-11-01T08:56:54.252066280Z] loading plugin "io.containerd.streaming.v1.manager"...  type=io.containerd.streaming.v1                                                                                                                 INFO[2024-11-01T08:56:54.252081197Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1                                                                                                                       INFO[2024-11-01T08:56:54.252230891Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1                                                                                                                     INFO[2024-11-01T08:56:54.252724843Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2                                                                                                                        INFO[2024-11-01T08:56:54.253212403Z] loading plugin "io.containerd.runtime.v2.shim"...  type=io.containerd.runtime.v2                                                                                                                        INFO[2024-11-01T08:56:54.253267456Z] loading plugin "io.containerd.sandbox.store.v1.local"...  type=io.containerd.sandbox.store.v1                                                                                                           INFO[2024-11-01T08:56:54.253318098Z] loading plugin "io.containerd.sandbox.controller.v1.local"...  type=io.containerd.sandbox.controller.v1                                                                                                 INFO[2024-11-01T08:56:54.253353902Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1                                                                                                          INFO[2024-11-01T08:56:54.253385773Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1                                                                                                             INFO[2024-11-01T08:56:54.253414962Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1                                                                                                                INFO[2024-11-01T08:56:54.253447679Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1                                                                                                              INFO[2024-11-01T08:56:54.253480419Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1                                                                                                       INFO[2024-11-01T08:56:54.253511330Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1                                                                                                          INFO[2024-11-01T08:56:54.253541066Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1                                                                                                           INFO[2024-11-01T08:56:54.253568070Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1                                                                                                               INFO[2024-11-01T08:56:54.253624762Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1                                                                                                                        INFO[2024-11-01T08:56:54.253658136Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1                                                                                                                           INFO[2024-11-01T08:56:54.253688664Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2024-11-01T08:56:54.253720433Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1                                                                                                                            INFO[2024-11-01T08:56:54.253750613Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1                                                                                                                            INFO[2024-11-01T08:56:54.253783022Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1                                                                                                                     INFO[2024-11-01T08:56:54.253812114Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1                                                                                                                            INFO[2024-11-01T08:56:54.253855255Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1                                                                                                                        INFO[2024-11-01T08:56:54.253886701Z] loading plugin "io.containerd.grpc.v1.sandbox-controllers"...  type=io.containerd.grpc.v1                                                                                                               INFO[2024-11-01T08:56:54.253921685Z] loading plugin "io.containerd.grpc.v1.sandboxes"...  type=io.containerd.grpc.v1                                                                                                                         INFO[2024-11-01T08:56:54.253951214Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1                                                                                                                         INFO[2024-11-01T08:56:54.253980308Z] loading plugin "io.containerd.grpc.v1.streaming"...  type=io.containerd.grpc.v1                                                                                                                         INFO[2024-11-01T08:56:54.254017158Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2024-11-01T08:56:54.254074744Z] loading plugin "io.containerd.transfer.v1.local"...  type=io.containerd.transfer.v1                                                                                                                     INFO[2024-11-01T08:56:54.254172776Z] loading plugin "io.containerd.grpc.v1.transfer"...  type=io.containerd.grpc.v1                                                                                                                          INFO[2024-11-01T08:56:54.254234729Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1                                                                                                                           INFO[2024-11-01T08:56:54.254288178Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1                                                                                                                   INFO[2024-11-01T08:56:54.254461998Z] loading plugin "io.containerd.tracing.processor.v1.otlp"...  type=io.containerd.tracing.processor.v1                                                                                                    INFO[2024-11-01T08:56:54.254515518Z] skip loading plugin "io.containerd.tracing.processor.v1.otlp"...  error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1                                          INFO[2024-11-01T08:56:54.254546365Z] loading plugin "io.containerd.internal.v1.tracing"...  type=io.containerd.internal.v1
INFO[2024-11-01T08:56:54.254587777Z] skip loading plugin "io.containerd.internal.v1.tracing"...  error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1
INFO[2024-11-01T08:56:54.254633683Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1                                                                                                                       INFO[2024-11-01T08:56:54.254703173Z] loading plugin "io.containerd.nri.v1.nri"...  type=io.containerd.nri.v1                                                                                                                                 INFO[2024-11-01T08:56:54.254760482Z] NRI interface is disabled by configuration.
INFO[2024-11-01T08:56:54.255385104Z] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock                                                                                                  INFO[2024-11-01T08:56:54.255572686Z] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc                                                                                                  INFO[2024-11-01T08:56:54.255821469Z] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2024-11-01T08:56:54.255875262Z] containerd successfully booted in 0.038585s                                                                                                                                                             ERRO[2024-11-01T08:56:54.276594325Z] failed to mount overlay: invalid argument     storage-driver=overlay2                                                                                                                                   ERRO[2024-11-01T08:56:54.277307332Z] exec: "fuse-overlayfs": executable file not found in $PATH  storage-driver=fuse-overlayfs                                                                                                               INFO[2024-11-01T08:56:54.286250127Z] Loading containers: start.                                                                                                                                                                              INFO[2024-11-01T08:56:54.489427432Z] Loading containers: done.
INFO[2024-11-01T08:56:54.503248462Z] Docker daemon                                 commit=fca702d graphdriver=vfs version=24.0.9
INFO[2024-11-01T08:56:54.503601303Z] Daemon has completed initialization
INFO[2024-11-01T08:56:54.543618420Z] API listen on /var/run/docker.sock

$ sudo docker run -it --rm ubuntu:20.04 bash                                                                                                                                                                                        Unable to find image 'ubuntu:20.04' locally                                                                                                                                                                                                  20.04: Pulling from library/ubuntu                                                                                                                                                                                                           d9802f032d67: Pull complete                                                                                                                                                                                                                  Digest: sha256:8e5c4f0285ecbb4ead070431d29b576a530d3166df73ec44affc1cd27555141b                                                                                                                                                              Status: Downloaded newer image for ubuntu:20.04                                                                                                                                                                                              time="2024-11-01T08:57:17.046618683Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1                                                                                                                                                                                                                                             time="2024-11-01T08:57:17.047162155Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1                                                                                                                                                                                                                                             time="2024-11-01T08:57:17.047189932Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1                                                                                                                                                                                                                                             time="2024-11-01T08:57:17.047282180Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1                                                                                                                                                                                                                                             time="2024-11-01T08:57:17.298755339Z" level=error msg="failed to enable controllers ([cpuset cpu io memory hugetlb pids rdma misc])" error="failed to write subtree controllers [cpuset cpu io memory hugetlb pids rdma misc] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory"                                                                                                                                                                                                                                             time="2024-11-01T08:57:17.298968403Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/85a15a36b08be03e5fb3c1233cf45d9ab6845883de71ad4b220a1ea4a9770236/memory.events\": no such file or directory"
                                                                                                     root@85a15a36b08b:/# exit                                                                                                               exit                                                                                                                                                                                                                                         INFO[2024-11-01T08:57:33.810720502Z] ignoring event                                container=85a15a36b08be03e5fb3c1233cf45d9ab6845883de71ad4b220a1ea4a9770236 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"                                                                                                                                                                                                                                             INFO[2024-11-01T08:57:33.811060876Z] shim disconnected                             id=85a15a36b08be03e5fb3c1233cf45d9ab6845883de71ad4b220a1ea4a9770236 namespace=moby                                                                                                                                                                                                                                             WARN[2024-11-01T08:57:33.811182351Z] cleaning up after shim disconnected           id=85a15a36b08be03e5fb3c1233cf45d9ab6845883de71ad4b220a1ea4a9770236 namespace=moby                                                                                                                                                                                                                                             INFO[2024-11-01T08:57:33.811202018Z] cleaning up dead shim                         namespace=moby                                                                                                                                                                                                                                             WARN[2024-11-01T08:57:33.835531564Z] failed to close stdin: task 85a15a36b08be03e5fb3c1233cf45d9ab6845883de71ad4b220a1ea4a9770236 not found: not found

[marrip]

adding --cgroupns=host to the gitpod container seems to solve the cgroup issue.