[bitnami/rabbitmq] Cluster on Kubernetes Failing with one node go down

[JalisDiehl]

Name and Version

3.17.3

What architecture are you using?

amd64

What steps will reproduce the bug?

We are using Rabbitmq on Kubernetes Cluster, we started with spot instances and 3 pods and pvcs, when one node go down the cluster seems to stop

Are you using any custom parameters or values?

  valuesInline:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions: 
              - key: "karpenter.sh/capacity-type"
                operator: "In"
                values: ["on-demand"]  
    nodeSelector: 
      karpenter.sh/capacity-type: on-demand
    clustering:
      forceBoot: true
    podManagementPolicy: "Parallel"
    resources:
      requests:
        cpu: 100m
        memory: 812Mi
      limits:
        cpu: 2
        memory: 3072Mi    
    replicaCount: 3
    persistence:
      size: 40Gi
    auth:
      username: xpto
      password: bar
      erlangCookie: foo
      tls: 
        enabled: true
        failIfNoPeerCert: false
        existingSecret: "certtls"
        existingSecretFullChain: true
        sslOptionsVerify: "verify_none"
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
        external-dns.alpha.kubernetes.io/hostname: dns-value
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ""
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "5671"
        prometheus.io/scrape: "true"
        prometheus.io/port: "9419"
        prometheus.io/path: "/metrics/per-object"
      epmdPortEnabled: false
      distPortEnabled: false
    ingress:
      enabled: true
      hostname: xpto
      path: /
      tls: true
      ingressClassName: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt

What is the expected behavior?

Not stop work

What do you see instead?

Rabbit down

Additional information

[javsalgar]

Hi,

Could you share the logs of the instances?

[JalisDiehl]

There are no logs, only pods try to start again...
But when happen again, I will try to collect

[JalisDiehl]

Buuut this happen when we have a node switch, because we use karpenter with ec2-spot and ec2-on-demand, my question is, how we can active HA on rabbit, it seems we have 3 pods but each pod has own queue, so it isn't on HA

[dgomezleon]

Hi @JalisDiehl ,

Could you please take a look at this previous case where that was discussed?

[JalisDiehl]

What discussion?

[dgomezleon]

My bad: #6276