hubble-relay pods are not starting if hubble.listenAddress is set to 127.0.0.1:4244

[lirtistan]

Bug Description:

If i deploy cilium (via helm) and set hubble.enable=true and hubble.listenAddress=127.0.0.1:4244, the hubble-relay pods are not able to get ready.

I want to set the IP to limit the potential attck surface on our nodes, most of them have 2 NICs, one for INET traffic and the other NIC for LAN traffic. Actually i see no way to prevent hubble from listening on every addresses.

The only way i have is to set the host firewall, which can be challenging (many different host configurations, so not every node has an equal configuration).

My actual test deployments (K3s) are on a staging cluster (just 2 nodes), with an Debian 12 minimal setup.

This is how my actual values.yaml look like..

k8sServiceHost: cluster  # a VIP behind a record in /etc/hosts
k8sServicePort: 6443
kubeProxyReplacement: true
identityAllocationMode: kvstore
etcd:
  enabled: true
  ssl: false
  endpoints:
  - http://172.16.0.1:2379
  - http://172.16.0.2:2379
updateStrategy:
  rollingUpdate:
    maxUnavailable: 1
operator:
  replicas: 2
  rollOutPods: true
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 50%
  prometheus:
    enabled: false
ipam:
  operator:
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv4PodCIDRList:
    - 10.42.0.0/16
hubble:
  enabled: true
  #listenAddress: "127.0.0.1:4244"
  metrics:
    enabled: []
  relay:
    enabled: true
    rollOutPods: true
    replicas: 2
    prometheus:
      enabled: false
envoy:
  prometheus:
    enabled: false
prometheus:
  enabled: false

[lirtistan]

with hubble.listenAddress=:4244 (working)

root@staging2:~# ss -lntp | grep cilium
LISTEN 0      4096       127.0.0.1:33665      0.0.0.0:*    users:(("cilium-agent",pid=8341,fd=57))                 
LISTEN 0      4096       127.0.0.1:9234       0.0.0.0:*    users:(("cilium-operator",pid=7929,fd=8))               
LISTEN 0      4096       127.0.0.1:9890       0.0.0.0:*    users:(("cilium-agent",pid=8341,fd=6))                  
LISTEN 0      4096       127.0.0.1:9891       0.0.0.0:*    users:(("cilium-operator",pid=7929,fd=6))               
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7997,fd=35))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7997,fd=34))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7997,fd=33))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7997,fd=32))                 
LISTEN 0      4096       127.0.0.1:9879       0.0.0.0:*    users:(("cilium-agent",pid=8341,fd=69))                 
LISTEN 0      4096               *:4244             *:*    users:(("cilium-agent",pid=8341,fd=32))  # <-- here

with hubble.listenAddress=127.0.0.1:4244 (NOT working)

root@staging2:~# ss -lntp | grep cilium
LISTEN 0      4096       127.0.0.1:33665      0.0.0.0:*    users:(("cilium-agent",pid=8941,fd=59))                 
LISTEN 0      4096       127.0.0.1:9234       0.0.0.0:*    users:(("cilium-operator",pid=7629,fd=6))               
LISTEN 0      4096       127.0.0.1:9890       0.0.0.0:*    users:(("cilium-agent",pid=8941,fd=8))                  
LISTEN 0      4096       127.0.0.1:9891       0.0.0.0:*    users:(("cilium-operator",pid=7959,fd=8))               
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7932,fd=38))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7932,fd=34))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7932,fd=33))                 
LISTEN 0      4096       127.0.0.1:9878       0.0.0.0:*    users:(("cilium-envoy",pid=7932,fd=32))                 
LISTEN 0      4096       127.0.0.1:9879       0.0.0.0:*    users:(("cilium-agent",pid=8941,fd=72))                 
LISTEN 0      4096       127.0.0.1:4244       0.0.0.0:*    users:(("cilium-agent",pid=8941,fd=29)) # <-- here

Sorry havn't copied the pod logs.

I'm using cilium 1.17.1 with K3s v1.32.2+k3s1, with external etcd 3.4.35

[lirtistan]

I also see no way to set multiple addresses.