Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

“Observed a panic in reconciler” on the backup restore request #1719

Open
sekot opened this issue May 31, 2024 · 2 comments
Open

“Observed a panic in reconciler” on the backup restore request #1719

sekot opened this issue May 31, 2024 · 2 comments
Labels

Comments

@sekot
Copy link

sekot commented May 31, 2024

Report

restoring a backup from another cluster no longer works, the operator panic and restart. the job is never launch.

More about the problem

the restore.yaml

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterRestore
metadata:
  name: check-backup-pxc-restore-manual-qyp3i
  namespace: pxc-check-backup
spec:
  pxcCluster: percona-cluster
  backupSource:
    destination: azure://mysql80-pxc-backups/percona-cluster-2024-05-31-00:00:29-full
    azure:
      container: mysql80-pxc-backups
      credentialsSecret: percona-azure-secret-check-backup

the operator logs :

2024-05-31T16:14:02.290Z	INFO	Password expiration policy updated	{"controller": "pxc-controller", "namespace": "pxc-check-backup", "name": "percona-cluster", "reconcileID": "f62cc541-995a-4499-8253-d58aaf4c2b5b", "user": "root"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 2024-05-31T16:16:16.876Z	INFO	backup restore request	{"controller": "pxcrestore-controller", "namespace": "pxc-check-backup", "name": "check-backup-pxc-restore-manual-qyp3i", "reconcileID": "1076b45e-e86d-4031-93f4-10de9627614b"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 2024-05-31T16:16:16.893Z	INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "pxcrestore-controller", "namespace": "pxc-check-backup", "name": "check-backup-pxc-restore-manual-qyp3i", "reconcileID": "1076b45e-e86d-4031-93f4-10de9627614b"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator panic: runtime error: invalid memory address or nil pointer dereference [recovered]
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	panic: runtime error: invalid memory address or nil pointer dereference
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x1634635]
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator goroutine 197 [running]:
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 +0x1e5
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator panic({0x1aa5fe0?, 0x2e707b0?})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/usr/local/go/src/runtime/panic.go:914 +0x21f
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PXCBackupStatus).GetStorageType(...)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_backup_types.go:140
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.RestoreJob(0xc000bf7d40, 0xc000740fc0, 0xc000e98000, {0xc00089bf40, 0x44}, 0x0)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/restore.go:140 +0x75
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*azure).Job(0xc000704600?)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restorer.go:191 +0x32
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).validate(0x1a8a960?, {0x204fa08, 0xc000bb9980}, 0xc000bf7d40, 0x204f998?, 0xc000e98000?)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restore.go:80 +0x4b
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).Reconcile(0xc000704600, {0x204fa08, 0xc000bb9980}, {{{0xc0008f6580?, 0x5?}, {0xc000c3acf0?, 0xc000f5ed08?}}})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/controller.go:190 +0xf14
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2053448?, {0x204fa08?, 0xc000bb9980?}, {{{0xc0008f6580?, 0xb?}, {0xc000c3acf0?, 0x0?}}})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0xb7
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00014f720, {0x204fa40, 0xc0004d0f50}, {0x1b4eb40?, 0xc00005ec40?})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316 +0x3cc
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00014f720, {0x204fa40, 0xc0004d0f50})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x1af
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x79
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 105
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x565

Steps to reproduce

try to restore from another cluster with a different storage account

  1. install a cluster with an azure storage account for backup
  2. launch backup
  3. install another cluster with a different azure storage account
  4. add the secret of the first azure storage account
  5. restore the backup with this secret and backup of step 2

Versions

  1. Kubernetes 1.27
  2. Operator 1.14
  3. Database mysql 5.7 & 8.0

Anything else?

it worked with operator 1.13

same issue here with s3 : https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/1

@sekot sekot added the bug label May 31, 2024
@nicolasgrieco
Copy link

I also have the same issue.

@sekot
Copy link
Author

sekot commented Jun 5, 2024

ok backup section in cr must be present.
so opérator panic is sade :) and I wonder if the backup section should be mandatory to restore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants