Enable other containers to join network namespace of the none network #3443

Shubhranshu153 · 2024-09-20T17:50:55Z

Fixes: Unable to attach to container network created with none

Description

There is a use case with pause containers, where other containers attaches to the pause container network. The pause container is launched with network none. This requires the pause container have a copy of hosts, hostname and resolv conf.
It also seems to share a net namespace, the containers must also share a user namespace.

The solution is to have a copy of the hosts/hostname and resolv.conf. In case of container network, add userns and netns both.

Want to confirm is this an acceptable solution and i can send out an PR for it.
Steps to reproduce the issue

Create a pause container with network none.
Create another container with --net container:

Describe the results you received and expected

It would display errors with resolv.conf not found and once those configs are added would see an error with sys fs.
Expected result is to be able to connect to the network of pause container.
What version of nerdctl are you using?

1.7.5
Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None
Host information

lima vm (fedora image), but can be reproduced in any architecture.

AkihiroSuda · 2024-09-20T18:14:43Z

Want to confirm is this an acceptable solution

Yes if it works with Docker.

Needs an integration test.

pkg/containerutil/container_network_manager.go

apostasie · 2024-09-20T23:06:45Z

pkg/containerutil/container_network_manager.go

+	}
+
+	resolvConfPath := filepath.Join(stateDir, "resolv.conf")
+	copyFileContent("/etc/resolv.conf", resolvConfPath)


That would make containers using none to now use the host resolv.conf, right?
This does not seem right.

checked with docker and they have it for none, personally i dont think it should be part of none, but if it is not attaching to a none network namespace (custom use cases) seems to throw error.

It does not look like they have it here...

docker run --net none debian cat /etc/resolv.conf

# Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified. nameserver 192.168.5.2 search . # Based on host file: '/run/systemd/resolve/resolv.conf' (legacy) # Overrides: []

vs.

cat /etc/resolv.conf

nameserver 127.0.0.53 options edns0 trust-ad search .

this is my output (docker 25)

docker run --network none alpine:latest cat /etc/resolv.conf <redacted> nameserver 10.4.4.10 options timeout:1 attempts:2

sudo docker run --network none alpine:latest cat /etc/hostname 9d25f3a65441

to be fair i tested in another system i got same as yours too, so not sure which one is correct, but for some system is taking from /run/systemd/resolv/resolv.conf?

It likely does not matter much, since there will be no network anyhow. Concern here is more about information leakage.
I would suggest we either leave it empty or mimic the behavior of cniNetworkManager

not sure if information leak is a concern in network none, if one can ssh from host they have access to /etc/resolv.conf of the host anyway.

I like the idea of mimicing the behavior of cniNetworkManager, that seems to take into account adding host, dns etc.

while we are on the topic, if we use --network container:containerd id with

nameServers = m.netOpts.DNSServers searchDomains = m.netOpts.DNSSearchDomains dnsOptions = m.netOpts.DNSResolvConfOptions

dont we want to add them to the resolv.conf, similarly if --network host with these options, not sure they are getting added

[shubhum@lima-finch nerdctl]$ sudo nerdctl run --network host --hostname default --dns test --dns-search testname alpine:latest cat /etc/resolv.conf # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8). # Do not edit. # # This file might be symlinked as /etc/resolv.conf. If you're looking at # /etc/resolv.conf and seeing this text, you have followed the symlink. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs should typically not access this file directly, but only # through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a # different way, replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. nameserver 127.0.0.53 options edns0 trust-ad search ant.amazon.com amazon.com

changed this configs to some generic default values

pkg/containerutil/container_network_manager.go

apostasie · 2024-10-27T14:02:22Z

@Shubhranshu153 will be slow to review - travelling right now. Will do asap.

apostasie

Left a note about WriteDefaultResolveConf.
Otherwise, these changes seem good and will bring us more in line with what docker does, so +1.

This needs tests - especially around your use case of another container using the none container net.

apostasie · 2024-10-28T20:06:59Z

pkg/containerutil/container_network_manager.go

@@ -150,6 +150,11 @@ type noneNetworkManager struct {
 	client        *containerd.Client
 }

+func WriteDefaultResolvConf(filePath string) error {


I think we should use the existing resolvconf.Build.
It is not currently safe to use it (not atomic, nor locking), but it is better IMO than duplicating the problem.

apostasie · 2024-10-28T20:22:23Z

pkg/containerutil/container_network_manager.go

+	}
+
+	resolvConfPath := filepath.Join(stateDir, "resolv.conf")
+	if err := WriteDefaultResolvConf(resolvConfPath); err != nil {


See note above.

Shubhranshu153 · 2024-10-29T22:24:56Z

will add tests.

Shubhranshu153 · 2024-10-30T05:45:00Z

Left a note about WriteDefaultResolveConf. Otherwise, these changes seem good and will bring us more in line with what docker does, so +1.

This needs tests - especially around your use case of another container using the none container net.

so overall behaviour wise it is like docker but the configs dont still match, which is probably fine as we dont expose host details if we add specifically the things we want to expose.

Shubhranshu153 · 2024-10-30T06:29:42Z

@apostasie added the tests, for the configs i have made it docker non compat because of the explanation above.
PTAL.
Thanks.

cmd/nerdctl/container/container_run_network_linux_test.go

go.mod

apostasie · 2024-10-30T12:16:06Z

@apostasie added the tests, for the configs i have made it docker non compat because of the explanation above. PTAL. Thanks.

Thanks @Shubhranshu153

I left a few comments on the tests. I will give this a spin locally later today too, but otherwise looks good.

Shubhranshu153 · 2024-10-30T21:15:37Z

@apostasie PTAL
Not sure why FreeBSD is failing

Signed-off-by: Shubharanshu Mahapatra <[email protected]>

Shubhranshu153 · 2024-10-31T00:05:03Z

@AkihiroSuda @apostasie
need to add some test but need some guidance here
In Docker you can add something like:

docker run --rm   --dns 8.8.8.8   --dns 8.8.4.4 --add-host somewhat.example.com:127.0.0.1   --dns-search example.com  --network none  -it ubuntu:latest

and you can get them added

dev-dsk-shubhum-2b-5b633f8c % docker run --rm   --dns 8.8.8.8   --dns 8.8.4.4 --add-host somewhat.example.com:127.0.0.1   --dns-search example.com  --network none  -it ubuntu:latest cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.0.1       somewhat.example.com

(24-10-31 0:01:58) <0> [~]  
dev-dsk-shubhum-2b-5b633f8c % docker run --rm   --dns 8.8.8.8   --dns 8.8.4.4 --add-host somewhat.example.com:127.0.0.1   --dns-search example.com  --network none  -it ubuntu:latest cat /etc/resolv.conf 
search example.com
nameserver 8.8.8.8
nameserver 8.8.4.4
options timeout:1 attempts:2

So added these functionalities to none network, want some guidance if am missing functionality wise, i reused the code from host network as it seems it just used values from netopts. But i plan to refactor it to a single function (cleanup wise)

apostasie · 2024-10-31T18:02:04Z

Will have a look as time allows.
Thanks @Shubhranshu153

Shubhranshu153 · 2024-12-04T00:25:26Z

@AkihiroSuda @apostasie
now that nerdctl 2.0 is out want to check if i can get some feedback on the above proposal.
Idea is to support the flags and behaviour as close as possible to docker.
Thank You

AkihiroSuda · 2024-12-04T09:23:58Z

go.mod

@@ -124,6 +124,7 @@ require (
 	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/spaolacci/murmur3 v1.1.0 // indirect
 	github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6 // indirect
+	github.com/stretchr/objx v0.5.2 // indirect


Need to run go mod tidy use the nerdctl test framework and fix a few things in code, just checking if my logic is correct to add the options DNS and host options for none network and probably need to be done for host network as well

I believe you added it initially with #3443 (review)

But really it should not be necessary.

Can you try removing the line from go.mod then doing go mod tidy again?

Will look at the rest later today.

Thank you, the current version is more for making sure the direction is correct. I will be having another pr using the new test framework and updates for the host network config.

apostasie · 2024-12-05T00:23:30Z

@Shubhranshu153

I am a bit confused about the thing overall.

docker run -ti --net container:paused debian bash
docker: Error response from daemon: cannot join network of a non running container: c66b2c44df029075540e67ceb7ed1de63ee89f98baad49d823eeeab31dc1e08a.
ERRO[0000] error waiting for container: context canceled

So, Docker does not seem to allow starting a container using the network of a paused container - which does make sense.

We should still support using a started container with --net none (or host) as a network for another container though (doesn't seem to work currently).

apostasie · 2024-12-05T00:26:03Z

And sorry for the late review.
Was busy elsewhere lately.

Shubhranshu153 · 2024-12-05T00:31:35Z

@Shubhranshu153

I am a bit confused about the thing overall.
docker run -ti --net container:paused debian bash
docker: Error response from daemon: cannot join network of a non running container: c66b2c44df029075540e67ceb7ed1de63ee89f98baad49d823eeeab31dc1e08a.
ERRO[0000] error waiting for container: context canceled
So, Docker does not seem to allow starting a container using the network of a paused container - which does make sense.

We should still support using a started container with --net none as a network for another container though (doesn't seem to work currently).

The container i am trying to connect is a pause container (the container is in running state) which is basically sleep infinity container with a non network. From what I understand it is used to create a network namespace to which other containers can join and share.

apostasie · 2024-12-05T00:35:32Z

Sorry for the confusion :-)

I thought by "pause container" you meant "created".

Looking again.

apostasie · 2024-12-05T00:55:27Z

pkg/containerutil/container_network_manager.go

@@ -179,9 +248,75 @@ func (m *noneNetworkManager) InternalNetworkingOptionLabels(_ context.Context) (

 // ContainerNetworkingOpts Returns a slice of `oci.SpecOpts` and `containerd.NewContainerOpts` which represent
 // the network specs which need to be applied to the container with the given ID.
-func (m *noneNetworkManager) ContainerNetworkingOpts(_ context.Context, _ string) ([]oci.SpecOpts, []containerd.NewContainerOpts, error) {
+func (m *noneNetworkManager) ContainerNetworkingOpts(_ context.Context, containerID string) ([]oci.SpecOpts, []containerd.NewContainerOpts, error) {


I am increasingly uncomfortable with the amount of duplication between ContainerNetworkingOpts implementations (and even more so about the differences in there). But then, refactoring that should probably be a separate endeavor.

Not really as I was writing it I too saw that so I plan to refactor it but before that just wanted to confirm the direction so that the effort is not put in the wrong place

apostasie · 2024-12-05T01:00:29Z

@Shubhranshu153 I think directionally that is ok.

Though, I would suggest really strong testing on this in making sure we are truly aligned with Docker:

specifically test what the content of /etc/hosts, /etc/resolv.conf and /etc/hostname are for none containers and container:foo containers, to make sure we are fully aligned with docker

Also, I am wondering if hosts files are now going to updated for none network containers - I do not think that should happen, as:

this has a (small?) performance angle
part of the reason for running none containers is complete networking isolation from the other containers and host - so, let's make sure we do not break that promise by leaking networking details from other containers info in there

apostasie · 2024-12-05T01:01:17Z

As I said in my comment, I am also concerned about the amount of duplication / differences between network managers (specifically ContainerNetworkingOpts).

We really need to clean that stuff up eventually and revise our approach there.

Shubhranshu153 · 2024-12-05T01:03:31Z

Yes testing needs to be rigorous not to break the promise of isolation as in that case it will be sec vulnerability.

Will send out a revision once I clean and have some robust testing in place.

Thanks

apostasie · 2024-12-05T01:03:56Z

Yes testing needs to be rigorous not to break the promise of isolation as in that case it will be sec vulnerability.

Will send out a revision once I clean and have some robust testing in place.

Thanks

Awesome.
Tag me when done - will help if I can.

Shubhranshu153 marked this pull request as ready for review September 20, 2024 18:03

apostasie reviewed Sep 20, 2024

View reviewed changes

pkg/containerutil/container_network_manager.go Outdated Show resolved Hide resolved

apostasie reviewed Sep 20, 2024

View reviewed changes

pkg/containerutil/container_network_manager.go Outdated Show resolved Hide resolved

Shubhranshu153 force-pushed the fix-network-none branch 2 times, most recently from e99fd1d to 29760dc Compare October 26, 2024 14:54

Shubhranshu153 requested a review from apostasie October 26, 2024 14:59

apostasie reviewed Oct 28, 2024

View reviewed changes

Shubhranshu153 force-pushed the fix-network-none branch 6 times, most recently from 1a627e3 to e80e165 Compare October 30, 2024 05:40

Shubhranshu153 force-pushed the fix-network-none branch from e80e165 to 516b8c0 Compare October 30, 2024 05:53

Shubhranshu153 force-pushed the fix-network-none branch 2 times, most recently from a2a9820 to ebc6d87 Compare October 30, 2024 09:42

apostasie reviewed Oct 30, 2024

View reviewed changes

cmd/nerdctl/container/container_run_network_linux_test.go Outdated Show resolved Hide resolved

apostasie reviewed Oct 30, 2024

View reviewed changes

cmd/nerdctl/container/container_run_network_linux_test.go Outdated Show resolved Hide resolved

apostasie reviewed Oct 30, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Shubhranshu153 force-pushed the fix-network-none branch 4 times, most recently from c64c4f7 to cdb4929 Compare October 30, 2024 19:21

Shubhranshu153 force-pushed the fix-network-none branch 2 times, most recently from e2dc511 to eba7437 Compare October 30, 2024 20:36

Shubhranshu153 force-pushed the fix-network-none branch from eba7437 to 3bc581d Compare October 30, 2024 21:22

enable host name and default resolv.conf and hosts file in none network

b477824

Signed-off-by: Shubharanshu Mahapatra <[email protected]>

Shubhranshu153 force-pushed the fix-network-none branch from 3bc581d to b477824 Compare October 30, 2024 23:59

AkihiroSuda added this to the v2.0.2 milestone Dec 4, 2024

AkihiroSuda reviewed Dec 4, 2024

View reviewed changes

apostasie reviewed Dec 5, 2024

View reviewed changes

apostasie mentioned this pull request Dec 5, 2024

--net=host prints WARN[0000] failed to clean up container networking: [...] meta.json: no such file or directory on exit #3648

Open

AkihiroSuda modified the milestones: v2.0.2, v2.0.3 Dec 10, 2024

Enable other containers to join network namespace of the none network #3443

Are you sure you want to change the base?

Enable other containers to join network namespace of the none network #3443

Conversation

Shubhranshu153 commented Sep 20, 2024

Fixes: Unable to attach to container network created with none

Description

Describe the results you received and expected

AkihiroSuda commented Sep 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shubhranshu153 Sep 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apostasie commented Oct 27, 2024

apostasie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shubhranshu153 commented Oct 29, 2024

Shubhranshu153 commented Oct 30, 2024

Shubhranshu153 commented Oct 30, 2024 • edited Loading

apostasie commented Oct 30, 2024

Shubhranshu153 commented Oct 30, 2024

Shubhranshu153 commented Oct 31, 2024

apostasie commented Oct 31, 2024

Shubhranshu153 commented Dec 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apostasie commented Dec 5, 2024 • edited Loading

apostasie commented Dec 5, 2024

Shubhranshu153 commented Dec 5, 2024

apostasie commented Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apostasie commented Dec 5, 2024

apostasie commented Dec 5, 2024

Shubhranshu153 commented Dec 5, 2024

apostasie commented Dec 5, 2024

Shubhranshu153 Sep 21, 2024 •

edited

Loading

Shubhranshu153 commented Oct 30, 2024 •

edited

Loading

apostasie commented Dec 5, 2024 •

edited

Loading

apostasie commented Dec 5, 2024 •

edited

Loading