Unable to find way to have Fluentd or Fluent Bit reload AWS credentials once session expires, resulting in ExpiredToken: The security token included in the request is expired .

[111a5ab1]

TL;DR Unable to find way to have Fluentd or Fluent Bit reload STS based AWS credentials once session expires, resulting in ExpiredToken: The security token included in the request is expired.

I've spent considerable time (week or more) reading through documentation, trying to read through source code, and trying various configurations and failed miserably trying to get Fluentd/Fluentbit to work with short lived AWS credentials:

• AWS SSM installed in Hybrid activation mode, which writes short lived credentials out to /root/.aws/credentials (e.g. aws_session_token )
• Running latest Fluent(d/Bit) in containers, with .aws directory passed in.
• Sending logs to Cloudwatch--cloudwatch_logs (Fluentd) and cloudwatch (Fluent Bit) plugins. (cloudwatch_log is not an option as templating has not been ported)
• Verified the file in container is updating and matches host file.
• Have tried the plugin arn methods, as well as config file (e.g. credential_process )
• Initial authentication is successful and logs continue to write to Cloudwatch until the token expires.

I'm not even sure if this is a Fluent(d/Bit) issue, or Plugin issue--there seems to be code referencing authentication in both areas (I'm not a C programmer so trying to understand the code as best as possible).

Any ideas on how to get Fluent(d/Bit) to use the new credentials from the credential when it changes would be greatly appreciated (if it's even possible?) Thanks in advance for any help!

Cheers!