Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rke up v1.2.3 and v1.2.2 fails when upgrade_strategy is drain: true #2349

Open
ranchersvet opened this issue Nov 23, 2020 · 3 comments
Open

Comments

@ranchersvet
Copy link

ranchersvet commented Nov 23, 2020

RKE version:
v1.2.3 and v1.2.2

Docker version:

root@onlinestr03:~# docker info
Client:
 Debug Mode: false

Server:
 Containers: 17
  Running: 14
  Paused: 0
  Stopped: 3
 Images: 12
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-51-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.775GiB
 Name: onlinestr03
 ID: YZSH:IQKS:7NXM:2BSU:RT7H:CNXM:EOHQ:D7DM:NPJJ:CMTG:JI7I:H62J
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false**

Operating system and kernel: (cat /etc/os-release, `root@onlinestr03:~# uname -r)

5.4.0-51-generic
root@onlinestr03:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
DigitalCloud
cluster.yml file:

nodes:
  - address: 157.230.84.103
    user: root
    ssh_key_path: /root/.ssh/id_rsa
    role: [etcd,controlplane,worker]
  - address: 142.93.114.58
    user: root
    ssh_key_path: /root/.ssh/id_rsa
    role: [etcd,controlplane,worker]
  - address: 142.93.201.74
    user: root
    ssh_key_path: /root/.ssh/id_rsa
    role: [etcd,controlplane,worker]

upgrade_strategy:
  drain: true
    #  - address: 192.168.1.40
    #user: root
    #ssh_key_path: /root/.ssh/id_rsa
    #role: [etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h
    uid: 52034
    gid: 52034

# Required for external TLS termination with
# ingress-nginx v0.22+
ingress:
  provider: nginx
  options:
    use-forwarded-headers: "true"

    #kube-controller:
    #:cluster_cidr: "10.42.0.0/22"
kubernetes_version: v1.16.8-rancher1-2

Steps to Reproduce:
Try to update rke cluster.
Configure upgrade_strategy is drain: true

Results:
Upgrade fails with error:

INFO[0056] [controlplane] Processing controlplane hosts for upgrade 1 at a time
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13bb572]

goroutine 1 [running]:
github.com/rancher/rke/services.getDrainHelper(...)
        /go/src/github.com/rancher/rke/services/node_util.go:58
github.com/rancher/rke/services.UpgradeControlPlaneNodes(0x19da5a0, 0xc00003a088, 0xc0001f0420, 0xc000a544a0, 0x3, 0x4, 0x0, 0xc000646c00, 0xc000515b30, 0xc00085b300, ...)
        /go/src/github.com/rancher/rke/services/controlplane.go:72 +0x3f2
github.com/rancher/rke/cluster.(*Cluster).UpgradeControlPlane(0xc0003c3800, 0x19da5a0, 0xc00003a088, 0xc0001f0420, 0xc000515b30, 0xc000373f50, 0xc000373f80, 0xc000373fb0, 0xc0003ae000)
        /go/src/github.com/rancher/rke/cluster/cluster.go:225 +0xb46
github.com/rancher/rke/cluster.(*Cluster).DeployControlPlane(0xc0003c3800, 0x19da5a0, 0xc00003a088, 0xc000a7e4c0, 0x1, 0x0, 0x1b, 0xb, 0x0)
        /go/src/github.com/rancher/rke/cluster/cluster.go:161 +0x770
github.com/rancher/rke/cmd.ClusterUp(0x19da5a0, 0xc00003a088, 0x0, 0x0, 0x0, 0x0, 0x0, 0x176f959, 0xb, 0x0, ...)
        /go/src/github.com/rancher/rke/cmd/up.go:195 +0xcee
github.com/rancher/rke/cmd.clusterUpFromCli(0xc00035c9a0, 0x197e800, 0x0)
        /go/src/github.com/rancher/rke/cmd/up.go:302 +0x652
github.com/urfave/cli.HandleAction(0x15549e0, 0x1826af8, 0xc00035c9a0, 0xc000375f00, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:490 +0xc8
github.com/urfave/cli.Command.Run(0x176913c, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1779baa, 0x14, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:210 +0x9e8
github.com/urfave/cli.(*App).Run(0xc0004689c0, 0xc00000c080, 0x2, 0x2, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:255 +0x741
main.mainErr(0xc0001e2f50, 0x19a0e20)
        /go/src/github.com/rancher/rke/main.go:80 +0x14b9
main.main()
        /go/src/github.com/rancher/rke/main.go:25 +0x4e

gz#13573

@superseb
Copy link
Contributor

@ranchersvet Does it still panic when you add the following under upgrade_strategy?

  node_drain_input:
    force: false

@ranchersvet
Copy link
Author

ranchersvet commented Nov 24, 2020

Test with:
node_drain_input:
force: false
Finished successfully with no errors

@superseb
Copy link
Contributor

Thanks for testing, we can put a fix in for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants