Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Systemd service race causes incorrect hostname on Amazon Linux 2 #9506

Open
pdehlke opened this issue Oct 18, 2024 · 0 comments
Open

Systemd service race causes incorrect hostname on Amazon Linux 2 #9506

pdehlke opened this issue Oct 18, 2024 · 0 comments

Comments

@pdehlke
Copy link

pdehlke commented Oct 18, 2024

Bug Report

Describe the bug
init/systemd.in contains this:

[Unit]
Description=@FLB_PROG_NAME@
Documentation=https://docs.fluentbit.io/manual/
Requires=network.target
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/@FLB_OUT_NAME@
EnvironmentFile=-/etc/default/@FLB_OUT_NAME@
ExecStart=@CMAKE_INSTALL_FULL_BINDIR@/@FLB_OUT_NAME@ -c /@CMAKE_INSTALL_SYSCONFDIR@/@FLB_OUT_NAME@/@[email protected]
Restart=always

[Install]
WantedBy=multi-user.target

Unfortunately, Amazon Linux 2 still uses an old hostnamectl invocation like so:

[root@ip-10-0-0-235 ~]# systemctl cat sethostname.service

# /usr/lib/systemd/system/sethostname.service
[Unit]
Description=Set Hostname Workaround coreos/bugs#1272
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/bin/hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/hostname)"

[Install]
WantedBy=multi-user.target
[root@ip-10-0-0-235 ~]#

This is a race condition; often times if not virtually always, fluentd starts before hostnamectl can do its thing, and any output specifiers that contain ${HOSTNAME} end up with an incorrect value.

To Reproduce

  • Build a fluentbit AMI based on amazon linux 2 using packer/ansible/chef/puppet/what have you. Include fluentbit from https://packages.fluentbit.io/amazonlinux/2/
  • Have an output (say to S3) that has ${HOSTNAME} in it: s3_key_format /logs/%Y/%m/%d/%H/$TAG/${HOSTNAME}/$UUID.log.gz
  • Launch an instance of this AMI.
  • Watch as the HOSTNAME portion of the output path is incorrect until either fluentbit is restarted or the system is rebooted.

Expected behavior
${HOSTNAME} should match the hostname configured by hostnamectl

Additional context
I worked around this by adding this to /etc/systemd/system/fluent-bit.service.d/override.conf:

[Unit]
Requires=sethostname.service
After=sethostname.service

But I don't think folks should have to do that :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants