Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nerdctl kill isn't working after update to 2.x (unable to cleanup network) #3765

Open
mathias-ioki opened this issue Dec 16, 2024 · 6 comments · May be fixed by #3771
Open

nerdctl kill isn't working after update to 2.x (unable to cleanup network) #3765

mathias-ioki opened this issue Dec 16, 2024 · 6 comments · May be fixed by #3771
Labels
kind/unconfirmed-bug-claim Unconfirmed bug claim

Comments

@mathias-ioki
Copy link

mathias-ioki commented Dec 16, 2024 •
edited
Loading

Description

Hey,

with the update to version 2.0, nerdctl kill stopped working for us. Instead of sending the signal to the container, we just receive an error:

unable to cleanup network for container: abc0123456789, "unexpected end of JSON input"

It doesn't matter, if the container is spawned in the host network or if a overlay network is used. Not sure, if I missed something during the 2.0 release, but at least in the changelog I can't find any related breaking change.

The tested configuration is working fine with version 1.7.7 of nerdctl.

Steps to reproduce the issue

  1. Start a container via containerd
  2. Send SIGHUP to the container: sudo nerdctl kill -s HUP abc0123456789

Describe the results you received and expected

received result:

Just an error message - no signal was send:

FATA[0000] 1 errors:
unable to cleanup network for container: bfc5d8b4fa8da, "unexpected end of JSON input"

expected result:

Container receives a SIGHUP.

What version of nerdctl are you using?

Client:
 Version:	v2.0.2
 OS/Arch:	linux/amd64
 Git commit:	1220ce7ec2701d485a9b1beeea63dae3da134fb5
 buildctl:
  Version:

Server:
 containerd:
  Version:	v2.0.0
  GitCommit:	207ad711eabd375a01713109a8a197d197ff6542
 runc:
  Version:

Tested also with containerd 2.0.1

config:

debug           = false
debug_full      = false
address         = "unix:///run/containerd/containerd.sock"
namespace       = "k8s.io"
cgroup_manager  = "systemd"
experimental    = true
cni_netconfpath = "/etc/cni/net.d"

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
 Namespace:	k8s.io
 Debug Mode:	false

Server:
 Server Version: v2.0.0
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log:     fluentd journald json-file none syslog
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version:   6.8.0-41-generic
 Architecture:     x86_64
@mathias-ioki mathias-ioki added the kind/unconfirmed-bug-claim Unconfirmed bug claim label Dec 16, 2024
@apostasie
Copy link
Contributor

Hey @mathias-ioki

Can you clarify what you mean with Start a container via containerd or provide the exact command you are using?

Things seem to work fine for me with:

dmp@lima-v2:/Users/dmp/Projects/go/lepton$ nerdctl run -d --name foo debian sleep inf
a5e259cef2f696dfdb3281f1dc82141007036d7266c1a39259886f99441ee960
dmp@lima-v2:/Users/dmp/Projects/go/lepton$ nerdctl kill -s HUP foo
a5e259cef2f696dfdb3281f1dc82141007036d7266c1a39259886f99441ee960

The error message and location suggests that there might be something wrong with your network CNI conf.

@apostasie
Copy link
Contributor

Could also be that the container was not created by nerdctl, or has otherwise a problem with the label nerdctl/networks.

nerdctl inspect offendingcontainer < could provide some insight.

@mathias-ioki
Copy link
Author

Hey @apostasie, thank you for your fast reply!

We are using a setup with containerd + kubelet in standalone mode. So we are shipping container manifest files to kubelet, which is taking care of starting the container via containerd. So yes, you are right: These containers are neither managed via nerdctl nor have nerdctl related labels attached.

As far as I understood, nerdctl can be always used for debugging, even if it doesn't manage the containers itself. Please let me know, if this isn't the case.

@apostasie
Copy link
Contributor

Hey @apostasie, thank you for your fast reply!

We are using a setup with containerd + kubelet in standalone mode. So we are shipping container manifest files to kubelet, which is taking care of starting the container via containerd. So yes, you are right: These containers are neither managed via nerdctl nor have nerdctl related labels attached.

As far as I understood, nerdctl can be always used for debugging, even if it doesn't manage the containers itself. Please let me know, if this isn't the case.

@mathias-ioki agreed with you - nerdctl should at least bail out politely if something is not manageable instead of planting face.

I'll send a patch in a minute.

@apostasie apostasie linked a pull request Dec 16, 2024 that will close this issue
Open
@apostasie
Copy link
Contributor

@mathias-ioki ^

@jandubois jandubois mentioned this issue Dec 16, 2024
Open
@mathias-ioki
Copy link
Author

Oh, that was super fast again! Thank you very much @apostasie ! I will give it a try later and let you know, if this fix my issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/unconfirmed-bug-claim Unconfirmed bug claim
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Relax error on network annotation read failure apostasie/nerdctl
2 participants
@mathias-ioki @apostasie