Fleet agent never associates with kibana if Nodeport is provided in kibana manifest

[sagargulabani]

If we set the service and nodeport configuration for kibana, the fleet server 'Agent' custom resource never associates with kibana

    # service:
    #   spec:
    #     type: NodePort
    #     ports:
    #       - port: 5601
    #         nodePort: 32000

Fleet server manifest

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elastic-system
spec:
  version: 8.13.4
  kibanaRef:
    name: log-kibana
  elasticsearchRefs:
  - name: log-cluster
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            resources:
              requests:
                memory: 1Gi
                cpu: 400m
              limits:
                memory: 1Gi
                cpu: 400m

Kibana manifest

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: log-kibana
  namespace: elastic-system
spec:
  version: 8.13.4
  count: 1
  elasticsearchRef:
    name: log-cluster
  podTemplate:
    metadata:
      namespace: elastic-system
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 3Gi
              cpu: 1.5
            limits:
              memory: 4Gi
              cpu: 2
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            exec:
              command:
                - /bin/bash
                - -c
                - curl  http://127.0.0.1:5601/login -k -s
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        type: NodePort
        ports:
          - port: 5601
            nodePort: 32000
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["http://log-cluster-es-http.elastic-system.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic-system.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: apm
        version: latest
      - name: log
        version: latest
      - name: kubernetes
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        is_default_fleet_server: true
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent policy Three
        id: eck-agent-three
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
          - name: kubernetes-2
            id: kubernetes-2
            package:
              name: kubernetes
            inputs:
            - type: filestream
              enabled: true
              streams:
              - enabled: true
                data_stream:
                  type: logs
                  dataset: kubernetes.container_logs

Events:
  Type     Reason                   Age                From                                 Message
  ----     ------                   ----               ----                                 -------
  Warning  AssociationError         20s (x5 over 20s)  agent-controller                     Association backend for elasticsearch is not configured
  Warning  AssociationError         20s (x8 over 20s)  agent-controller                     Association backend for kibana is not configured
  Normal   AssociationStatusChange  20s                agent-kibana-association-controller  Association status changed from [] to [Pending]
  Normal   AssociationStatusChange  20s                agent-es-association-controller      Association status changed from [] to [elastic-system/log-cluster: Established]

It gets stuck in this state.

[pyaillet]

I had the same problem.

I was able to solve it by looking at the logs. They were showing an error related to the unavailability of a port named https on the service.

So the following modification should do the trick (at least, it did for me):

    # service:
    #   spec:
    #     type: NodePort
    #     ports:
    #       - name: https
    #         port: 5601
    #         nodePort: 32000

[lloydmcl]

Also having same issue running on an rke2 cluster (eck version 8.15.3) but am having issues with elasticsearch too which is running on ClusterIP and unfortunately setting the name for the service ports did not work for me.

Status:
  Elasticsearch Associations Status:
    elastic-system/elastic-cluster-es-http:  Pending
  Kibana Association Status:                 Pending
  Observed Generation:                       1
Events:
  Type     Reason                   Age                From                                 Message
  ----     ------                   ----               ----                                 -------
  Warning  AssociationError         30s (x5 over 30s)  agent-controller                     Association backend for elasticsearch is not configured
  Warning  AssociationError         30s (x5 over 30s)  agent-controller                     Association backend for kibana is not configured
  Normal   AssociationStatusChange  30s                agent-es-association-controller      Association status changed from [] to [elastic-system/elastic-cluster-es-http: Pending]
  Normal   AssociationStatusChange  30s                agent-kibana-association-controller  Association status changed from [] to [Pending]

Update
I previously had my kibanaRef set to kibana-kb-http and my elasticsearchRefs set to elastic-cluster-es-http. When I changed these to point to my clusters instead of the services, this fixed my association issues but I now have a problem where am getting

Events:
  Type     Reason                   Age                 From                                 Message
  ----     ------                   ----                ----                                 -------
  Warning  AssociationError         18m (x5 over 18m)   agent-controller                     Association backend for elasticsearch is not configured
  Warning  AssociationError         18m (x10 over 18m)  agent-controller                     Association backend for kibana is not configured
  Normal   AssociationStatusChange  18m                 agent-es-association-controller      Association status changed from [] to [elastic-system/elastic-cluster: Established]
  Normal   AssociationStatusChange  18m                 agent-kibana-association-controller  Association status changed from [] to [Established]
  Warning  ReconciliationError      18m (x10 over 18m)  agent-controller                     Reconciliation error: failed to request https://kibana-kb-http.elastic-system.svc:5601/api/fleet/setup, status is 401)

When I look through the fleet server logs, am seeing logs:

Error dialing lookup elasticsearch on <core-dns-ip>:53: no such host
Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get \"http://elasticsearch:9200\": lookup elasticsearch on <core-dns-ip>:53: no such host

strange that it's now trying to connect via http and not using the environment variable set FLEET_SERVER_ELASTICSEARCH_HOST = "https://elastic-cluster-es-http.elastic-system.svc:9200"

but at least I now have a fleet server pod executing.

I suspect the dns is irrelevant and the real issue I should be focusing on is:
"Warning ReconciliationError 18m (x10 over 18m) agent-controller Reconciliation error: failed to request https://kibana-kb-http.elastic-system.svc:5601/api/fleet/setup, status is 401)"

as it seems the agent is trying to authenticate and get it's settings from Kibana but can't. I've tried a number of things but cannot seem to get anywhere.

Hoping your issue is similar and we can find a way forward.