[Bug]: zookeeper can't start because they communicate with “internal_error”

[huntkalio]

Bug Description

Does this due to too long name?

avax.net.ssl|ERROR|A2|ListenerHandler-ab-cd-ef-online-xxxdata-client-zookeeper-0.ab-cd-ef-online-xxxdata-client-zookeeper-nodes.ab-cd-data-bigdata-client-kafka.svc/172.21.5.201:3888|2024-11-08 12:40:08.826 UTC|TransportContext.java:370|Fatal (INTERNAL_ERROR): -128 (
"throwable" : {
  java.lang.NegativeArraySizeException: -128
  	at java.base/sun.security.ssl.SSLSessionImpl.<init>(SSLSessionImpl.java:412)
  	at java.base/sun.security.ssl.SessionTicketExtension$T12CHSessionTicketConsumer.consume(SessionTicketExtension.java:468)
  	at java.base/sun.security.ssl.SSLExtension.consumeOnLoad(SSLExtension.java:609)
  	at java.base/sun.security.ssl.SSLExtensions.consumeOnLoad(SSLExtensions.java:201)
  	at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:964)
  	at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:842)
  	at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801)
  	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
  	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
  	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
  	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
  	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
  	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
  	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
  	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
  	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
  	at java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:371)
  	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
  	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
  	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
  	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
  	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
  	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
  	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
  	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
  	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
  	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
  	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
  	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
  	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:840)}
)
javax.net.ssl|DEBUG|A2|ListenerHandler-ab-cd-ef-online-xxxdata-client-zookeeper-0.ab-cd-ef-online-xxxdata-client-zookeeper-nodes.ab-cd-data-bigdata-client-kafka.svc/172.21.5.201:3888|2024-11-08 12:40:08.827 UTC|SSLSocketImpl.java:1759|close the underlying socket
javax.net.ssl|DEBUG|A2|ListenerHandler-ab-cd-ef-online-xxxdata-client-zookeeper-0.ab-cd-ef-online-xxxdata-client-zookeeper-nodes.ab-cd-data-bigdata-client-kafka.svc/172.21.5.201:3888|2024-11-08 12:40:08.827 UTC|SSLSocketImpl.java:1785|close the SSL connection (passive)
2024-11-08 12:40:08,827 INFO Accepted TLS connection from /172.21.5.71:39922 - NONE - SSL_NULL_WITH_NULL_NULL  

Steps to reproduce

create a cluster like this

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka   # must set to "kafka" if migrate from old no KafkaNodePool cluster
  namespace: ab-cd-data-bigdata-client-kafka
  labels:
    strimzi.io/cluster: ab-cd-ef-online-xxxdata-client
spec:
  replicas: 3
  roles:
    - broker   # must only to "broker" use zookeeper; else can use controller
  resources:
    requests:
      memory: 4Gi 
      cpu: "1.3"
    limits:
      memory: 4Gi
      cpu: "1.3"
  jvmOptions:
    -Xms: 1G
    -Xmx: 1G    
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        class: xxxxxxxxxxxxxxxx
        size: 200Gi
        deleteClaim: false
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: ab-cd-ef-online-xxxdata-client
  namespace: ab-cd-data-bigdata-client-kafka
  annotations:
    strimzi.io/node-pools: enabled
spec:
  kafka:
    version: 3.8.0   # 1)
    # rack:
    #   topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        topologySpreadConstraints:
        - maxSkew: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              strimzi.io/name: ab-cd-ef-online-xxxdata-client-kafka          
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: service_platform
                  operator: In
                  values:
                  - bmp-kafka
          podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  strimzi.io/name: ab-cd-ef-online-xxxdata-client-zookeeper
              topologyKey: topology.kubernetes.io/zone
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/name: ab-cd-ef-online-xxxdata-client-kafka
                topologyKey: kubernetes.io/hostname
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    listeners:
      - name: plain
        port: 9092
        type: nodeport
        tls: false
    readinessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 60
    livenessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 60
    config:
      auto.leader.rebalance.enable: "true"
      auto.create.topics.enable: "true"
      default.replication.factor: 2
      delete.topic.enable: true
      inter.broker.protocol.version: "3.8"              # 2)
      log.message.timestamp.type: "LogAppendTime"
      log.retention.minutes: "60"
      log.segment.bytes: "536870912"
      min.insync.replicas: 1
      num.io.threads: "8"
      num.network.threads: "5"
      num.partitions: "4"
      num.recovery.threads.per.data.dir: "1"
      num.replica.fetchers: "2"
      offsets.topic.replication.factor: 3
      replica.fetch.max.bytes: "1048576"
      replica.lag.time.max.ms: "30000"
      replica.socket.receive.buffer.bytes: "65536"
      socket.receive.buffer.bytes: "102400"
      socket.request.max.bytes: "104857600"
      socket.send.buffer.bytes: "102400"
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
      unclean.leader.election.enable: "false"
      zookeeper.session.timeout.ms: "18000"      
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    template:
      zookeeperContainer:
        env:
        - name: STRIMZI_JAVA_SYSTEM_PROPERTIES # (8)
          value: "-Djavax.net.debug=ssl "
      pod:
        topologySpreadConstraints:
          - maxSkew: 2
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                strimzi.io/name: ab-cd-ef-online-xxxdata-client-zookeeper
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/name: ab-cd-ef-online-xxxdata-client-zookeeper
                topologyKey: kubernetes.io/hostname
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: service_platform
                  operator: In
                  values:
                  - bmp-kafka
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "DEBUG"
    resources:
      requests:
        memory: 1Gi
        cpu: "0.1"
      limits:
        memory: 1Gi
        cpu: "0.5"
    # jvmOptions:
    #   -Xms: 128M
    #   -Xmx: 1024M
    readinessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 60      
    livenessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 60
    storage:
      type: persistent-claim
      class: xxxxxxxxxxxxxxxx
      size: 20Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml      
  entityOperator:
    # topicOperator: 
    #   resources:
    #     requests:
    #       memory: 400Mi
    #       cpu: "0.1"
    #     limits:
    #       memory: 400Mi
    #       cpu: "0.5"
    template:
      pod:
        affinity:
          podAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/name: ab-cd-ef-online-xxxdata-client-kafka
                topologyKey: kubernetes.io/hostname
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: service_platform
                  operator: In
                  values:
                  - bmp-kafka
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    # resources:
    #   requests:
    #     memory: 256Mi
    #     cpu: "0.01"
    #   limits:
    #     memory: 256Mi
    #     cpu: "0.01"
    template: 
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: service_platform
                  operator: In
                  values:
                  - bmp-kafka
  clusterCa:
    validityDays: 3650
    renewalDays: 365
  clientsCa:
    validityDays: 3650
    renewalDays: 365
  # maintenanceTimeWindows:
  #   - "* * 0-1 ? * SUN,MON,TUE,WED,THU *"

Expected behavior

No response

Strimzi version

0.43.0

quay.io/strimzi/kafka:0.43.0-kafka-3.8.0

Kubernetes version

1.28.12

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[huntkalio]

zookeeper.log

[scholzj]

I never saw this error, but it could be related to the long names you have. You should try with shorter ones. From what you shared, it is not really clear even where does the error show up, so not sure if we can do anything about it, but likely not.