fluent-bit 3.0.7 breaks kubernetes filter when using tls.verify and Use_Kubelet

[fcuello-fudo]

Bug Report

Describe the bug

If kubernetes filter is configured with tls.verify and Use_Kubelet is used, the requests to the k8s API will succeed but the requests to the node's kubelet will fail with tls errors.

To Reproduce

• Steps to reproduce the problem:
  Use the kubernetes filter and enable Use_Kubelet

Expected behavior
kubernetes filter works as before

Your Environment

• Version used: 3.0.7

Additional context

Fluent Bit v3.0.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/06/14 08:37:58] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/06/14 08:37:58] [error] [filter:kubernetes:kubernetes.0] kubelet upstream connection error

What seems to be happening is that the connection to the kubelet is using the address 127.0.0.1 and that address is not included in the kubelet's cert.

Disabling tls.verify solves the problem, but then the requests that go to the k8s api are not verified anymore.

[fcuello-fudo]

CC @cosmo0920 @edsiper

[kjq]

We see the same issue and disabling tls.verify looked to work.

[mattmooree]

This is also an issue with the "forward" output plugin.

We are sending data from application hosts using fluent-bit to a cluster of servers running fluent-bit which we have called "aggregators" - which are used to apply some logic to the logs and ship them on to OpenSearch.

In v3.0.7, the tls.verify breaks.

Configuration

[SERVICE]
Log_Level debug
Parsers_File /etc/fluent-bit/parsers.conf

[INPUT]
Name tail
Path /opt/tomcat/apache-tomcat-9.0.87/logs/catalina.out
Skip_Long_Lines On
Tag catalina

[OUTPUT]
Name forward
Match *
Compress gzip
Host fluent-bit-aggregator.uswe2.devtools.aws.cwan.io
tls On
Port 443

Logs

fluent-bit[375133]: [2024/06/18 06:20:02] [error] [tls] error: unexpected EOF with reason: certificate verify failed
fluent-bit[375133]: [2024/06/18 06:20:02] [debug] [upstream] connection #58 failed to fluent-bit-aggregator.uswe2.devtools.aws.cwan.io:443

Host and package info

uname -a
Linux 6.5.0-1018-aws #18~22.04.1-Ubuntu SMP Fri Apr 5 17:44:33 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

apt show fluent-bit
Package: fluent-bit
Version: 3.0.7
Priority: optional
Section: devel
Maintainer: Eduardo Silva [email protected]
Installed-Size: 96.1 MB
Depends: libc6 (>= 2.34), libgcc-s1 (>= 4.2), libpq5 (>= 9.0~), libsasl2-2 (>= 2.1.27+dfsg2), libssl3 (>= 3.0.0~~alpha1), libsystemd0 (>= 221), libyaml-0-2, zlib1g (>= 1:1.2.0)
Download-Size: 41.6 MB
APT-Manual-Installed: yes
APT-Sources: https://packages.fluentbit.io/ubuntu/jammy jammy/main amd64 Packages
Description: Fast data collector for Linux
Fluent Bit is a high performance and multi platform Log Forwarder.

apt show ca-certificates
Package: ca-certificates
Version: 20230311ubuntu0.22.04.1
Status: install ok installed
Priority: important
Section: misc
Maintainer: Ubuntu Developers [email protected]
Original-Maintainer: Julien Cristau [email protected]
Installed-Size: 399 kB
Depends: openssl (>= 1.1.1), debconf (>= 0.5) | debconf-2.0
Breaks: ca-certificates-java (<< 20121112+nmu1)
Enhances: openssl
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Common CA certificates
Contains the certificate authorities shipped with Mozilla's browser to allow
SSL-based applications to check for the authenticity of SSL connections.
.
Please note that Debian can neither confirm nor deny whether the
certificate authorities whose certificates are included in this package
have in any way been audited for trustworthiness or RFC 3647 compliance.
Full responsibility to assess them belongs to the local system
administrator.

v3.0.6 works as expected.

[cosmo0920]

Hey, we're trying to restore the previous behavior in #8966 and #8967 (for 3.0 line).
Could you try this branch then?