Fluentbit Azure Blob output works for 3 PCs but not for 1 out of 4

[lucasdaquina]

Hi there,

Please, I need your help to understand what could be happening or a way to debug it.

We are using fluentbit as a Windows service to forward log files, from other Windows services, to Azure, so, in other words, the input is the tail and the output is the Azure blob.

The behavior is strange because we installed the services on 4 PCs to test, and on 3 of them the logs are being forwarded perfectly, but there is 1 PC that doesn't forward the logs to Blob. We can see the logs being created on this PC but it ain't sent to blob storage.

I tried to find a way to "debug" fluentbit to try to discover why this is happening, but I couldn't.

Can someone try to help please, with some ideas or maybe have encountered the same situation to enlighten with what could be the solution?

I appreciate it a lot.

For Service:
[SERVICE]
flush 1
daemon Off
log_level info
parsers_file parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on

For input I have
[INPUT]
Name tail
Parser json
Path C:\Windows\System32\Logs****
Path_Key filename
tag tag/.log
tag_regex ^(?:.*\)?(?[^\\]+).log$
Read_from_Head true

And for output, I removed sensitive information, but all 4 PCs have the exactly same .conf file:
[OUTPUT]
name azure_blob
match *
account_name AccountName
shared_key ShareKey
path ${Path}/
container_name logs
auto_create_container off
endpoint https://frontdoor.net
tls on

Thanks!

[patrick-stephens]

I would use the stdout plugin to confirm Fluent Bit is seeing your logs. If so it narrows it down to an issue with output.

[lucasdaquina]

Thanks Patrick, I will try to do that.

[lucasdaquina]

Hi there, sorry for the delay in answering. I had a vacation last week.
So @patrick-stephens, I did what you suggested, thanks for that, it showed some information I want to share with you.

It's confirmed that the Fluent Bit sees the logs, as we can check on the first 3 lines of the text below. I'm using * to cover some sensitive data
"****/********-20230612.log" is the log file name

On the 4th line, we see that a DNS error happened, and since my network knowledge is limited I don't know what could be happening.
What I know is the PC has a fixed IP and fixed DNS server on IPv4 configuration, but the other PCs where the logs are being forwarded have the same thing, with a fixed IP and fixed DNS server on IPv4 configuration.

There is also some information about "chunk '47108-1686558996.704576000.flb' cannot be retried" in the text below. Could this be the error also? Or it's only because of the DNS?

/*****-20230612.log: [[1686559007.692907500, {}], {"filename"=>"C:\Windows\System32\Logs*****-202306"}]log", "log"=>"2023-06-12 08:36:47Z [014][DBG] Size for *******************: 21

/*****-20230612.log: [[1686559007.692908500, {}], {"filename"=>"C:\Windows\System32\Logs*****-202306"}]log", "log"=>"2023-06-12 08:36:47Z [014][DBG] Collecting data for *********** with **********************9

/*****-20230612.log: [[1686559007.692909500, {}], {"filename"=>"C:\Windows\System32\Logs*****-20230"}].log", "log"=>"2023-06-12 08:36:47Z [014][DBG] Size for *******************: 18

/********-20230612.log: [[[2023/06/12 10:36:48] [ warn] [net] getaddrinfo(host='.******.net', err=11): Could not contact DNS servers
1686559007.692910400, [2023/06/12 10:36:48] [error] [output:azure_blob:azure_blob.0] cannot create upstream connection for container check

{[2023/06/12 10:36:48] [error] [engine] chunk '47108-1686558996.704576000.flb' cannot be retried: task_id=3, input=tail.0 > output=azure_blob.0}
[2023/06/12 10:36:48] [ warn] [net] getaddrinfo(host='.**.net', err=11): Could not contact DNS servers
], [2023/06/12 10:36:48] [error] [output:azure_blob:azure_blob.0] cannot create upstream connection for container check

{[2023/06/12 10:36:48] [ warn] [engine] failed to flush chunk '47108-1686559007.692919500.flb', retry in 8 seconds: task_id=8, input=tail.0 > output=azure_blob.0 (out_id=0)
"filename"=>"C:\Windows\System32\Logs******-20230612.log", "log"=>"2023-06-12 08:36:47Z [014][DBG] Collecting da"}]for *********** with ***********************

/*****-20230612.log: [[1686559007.692911400, {}], {"filename"=>"C:\Windows\System32\Logs*****-20230"}].log", "log"=>"2023-06-12 08:36:47Z [014][DBG] Size for *******************: 13

Thanks a lot for the help.

[patrick-stephens]

Yeah so sounds like you have DNS issues for whatever reason and that's the problem.
I think you need to understand what the DNS set up is for your machines, it sounds like one of them is funky - it doesn't seem to be a Fluent Bit problem but a networking issue.

From: https://docs.fluentbit.io/manual/administration/networking#dns-mode
You could try some of the DNS options - maybe you have switches/firewalls dropping UDP requests for example (I don't know, it's down to your infra):

net.dns.mode tcp
net.dns.prefer_ipv4 true
net.dns.resolver LEGACY

[lucasdaquina]

@patrick-stephens thank you very much for all the help, with your idea and help it worked.
I appreciate.

[patrick-stephens]

No worries, did you figure out what the issue was?

[lucasdaquina]

Since the computer where the blocking was happening is from a client and I can't do any changes to test configuration/firewalls/etc, I followed the idea you gave of forcing dns.mode as TCP and the other two settings. And with that the logs were forwarded to the azure blob without any problem

[patrick-stephens]

OK so sounds like something in the networking was dropping UDP packets then - I mean it is allowed to after all (unreliable is in its name!) but it may be a firewall just dropped all of them or the DNS server did not respond to them (maybe they're using different DNS, or their ISP does not allow UDP or something like that).