How do I do "DooD" (i.e. mount docker.sock) with nerdctl and containerd (in EKS)?

[charlesritchea]

I know it's "wrong" to do this, but it's already been done and used all over the place in production. In particular we have an nvidia cuda based pod that loads models via docker. The nvidia part has made DinD much more complicated. Anyway, what all do I need to mount to get nerdctl to work inside of a container in EKS?

[charlesritchea]

PS I do already have nvidia-dind working and nvidia-smi works, but something else is broken with our container, so I'm looking for alternatives

[fahedouch]

What DooD refer to ?

what all do I need to mount to get nerdctl to work inside of a container in EKS?

reference that my help you: #2008 #1383 #2460 (reply in thread)

[charlesritchea]

Docker-out-of-Docker, It's where you mount the docker.sock so you can install docker in a container and run it on the host (big ol security hole there), it's how some logging daemonsets work. A "slightly" safer alternative is DinD Docker-in-Docker where you run a sidecar container with the whole dockerd, but you have to have a special one for GPU stuff and also install a k8s device plugin. Anyway, being able to run kubectl and it target the host containerd would be the closest to what I'm already doing and require the least amount of changes

…

On Sat, Sep 9, 2023, 5:22 PM fahed dorgaa ***@***.***> wrote: What DooD refer to ? what all do I need to mount to get nerdctl to work inside of a container in EKS? reference that my help you: #2008 <#2008> #1383 <#1383> #2460 (reply in thread) <#2460 (reply in thread)> — Reply to this email directly, view it on GitHub <#2484 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMUMHMYRJRY6TPTCJ6VB3TXZTM2VANCNFSM6AAAAAA4ROCQNE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[fahedouch]

Running such things seems like antipattern in kubernetes. As you said is totaly not safe to run things on the host nodes within kubernetes cluster as you can impact every thing running on this node.
From 1.24 we no longer talk about DooD, this being said you need to have container mount point from you host so you can introduce the docker socket to the host from containerd container and them do what you want eitheir from the host or an other container.

[charlesritchea]

The vast majority of people that used to do this were building docker images inside of a k8s pipeline, but there are better alternatives to that now, whereas my company was just containerized CLI tools to avoid dependency hell with like cuda or libpcl or needing ubuntu14 etc. But these tools need to work on colocated files. Basically before all of this we didn't see any reason to treat docker containers any differently from regular programs, and of course regular programs can call each other and are using colocated files. This was all trivial to do before 1.24

…

On Sat, Sep 9, 2023, 5:28 PM Charles Ritchea ***@***.***> wrote: Docker-out-of-Docker, It's where you mount the docker.sock so you can install docker in a container and run it on the host (big ol security hole there), it's how some logging daemonsets work. A "slightly" safer alternative is DinD Docker-in-Docker where you run a sidecar container with the whole dockerd, but you have to have a special one for GPU stuff and also install a k8s device plugin. Anyway, being able to run kubectl and it target the host containerd would be the closest to what I'm already doing and require the least amount of changes On Sat, Sep 9, 2023, 5:22 PM fahed dorgaa ***@***.***> wrote: > What DooD refer to ? > > what all do I need to mount to get nerdctl to work inside of a container > in EKS? > > reference that my help you: #2008 > <#2008> #1383 > <#1383> #2460 (reply in > thread) > <#2460 (reply in thread)> > > — > Reply to this email directly, view it on GitHub > <#2484 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMUMHMYRJRY6TPTCJ6VB3TXZTM2VANCNFSM6AAAAAA4ROCQNE> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[charlesritchea]

Yes, I know. docker is gone, it only needed docker.sock. nerdctl has specific mount requirements that I haven't figured out yet, containerd.sock is not enough.

…

On Sun, Sep 10, 2023, 7:06 AM fahed dorgaa ***@***.***> wrote: Running such things seems like antipattern in kubernetes. As you said is totaly not safe to run things on the host nodes within kubernetes cluster as you can impact every thing running on this node. From 1.24 we no longer talk about DooD, this being said you need to have container mount point from you host so you can introduce the docker socket to the host from containerd container and them do what you want eitheir from the host or an other container. — Reply to this email directly, view it on GitHub <#2484 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMUMHMACXZTG56DYTSICVLXZWNMVANCNFSM6AAAAAA4ROCQNE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[fahedouch]

please check these threads (#2008 #1383 #2460 (reply in thread)) ; it illustrates examples with what to mount