Scheduler log directory created as root

[jgoedeke]

Apache Airflow version

2.10.5

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Every night at 00:00 a new directory is created in /opt/airflow/logs/scheduler/ once in a while this directory is owned by root instead of the airflow user inside the docker container. This leads to an empty Dag list on the webserver and airflow not working at all.

airflow-scheduler docker log shows many of:

airflow-scheduler-1  | Process DagFileProcessor10-Process:
airflow-scheduler-1  | Traceback (most recent call last):
airflow-scheduler-1  |   File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
airflow-scheduler-1  |     self.run()
airflow-scheduler-1  |   File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
airflow-scheduler-1  |     self._target(*self._args, **self._kwargs)
airflow-scheduler-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/dag_processing/processor.py", line 174, in _run_file_processor
airflow-scheduler-1  |     set_context(log, file_path)
airflow-scheduler-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/log/logging_mixin.py", line 276, in set_context
airflow-scheduler-1  |     flag = cast(FileTaskHandler, handler).set_context(value)
airflow-scheduler-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-scheduler-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/log/file_processor_handler.py", line 63, in set_context
airflow-scheduler-1  |     local_loc = self._init_file(filename)
airflow-scheduler-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-scheduler-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/log/file_processor_handler.py", line 151, in _init_file
airflow-scheduler-1  |     open(log_file_path, "a").close()
airflow-scheduler-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^
airflow-scheduler-1  | PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2025-03-27/my_dag.py.log'

What you think should happen instead?

The daily directory should be created always with the same ownership and in case of a wrong permissions I would still expect airflow to show dags on the webserver, but stop scheduling new tasks (at the end only the scheduler is affected, right?).

How to reproduce

It is hard to reproduce since this happens only once in while, maybe every two weeks. It feels like a race condition or some other timing issue.

Operating System

Debian GNU/Linux 12 (bookworm) | Docker version 28.0.4, build b8034c0

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

I use docker compose environment. For unattended-upgrades I start the container as root user and use gosu in my entrypoint to set the correct user with: exec gosu airflow /entrypoint "$@". In the airflow Image I set the airflow UID to 1000 with RUN usermod -u 1000 airflow. The logs directory is locally mounted with

  volumes:
    - /media/airflow-share/logs:/opt/airflow/logs

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

That is something with your setup and race condition with your gosu. You should start your containers with the right user not with root - that is something you should investigate on your side to see what happens when you gosu.

There might be many reasons for that - it also depends on what image you have, what entrypoint you use and whether this entrypoint properly propagest signals and serves as "init" process.

[potiuk]

My recommendation: use the reference image as a base understand how entrypoints work https://airflow.apache.org/docs/docker-stack/entrypoint.html and do not use gosu to change the user just USER setting in the image as we do in the reference image.

[jgoedeke]

Thanks for you replies! The reason I use this approach is to have regular security-upgrades installed into all airflow containers. In my understanding the modification of the entrypoint utilizing gosu should not impact the original entrypoint, here are my exact changes:

Dockerfile

FROM apache/airflow:slim-2.10.5-python3.11

# UID 1000 necessary for logs directory permission
RUN usermod -u 1000 airflow

# root permission necessary to start cronjob
USER root

# install and configure cron and unattended-upgrade to install daily security upgrades
# ...
# cat <<EOF > /etc/apt/apt.conf.d/50unattended-upgrades
# Unattended-Upgrade::Origins-Pattern {
#         "origin=Debian,codename=\${distro_codename},label=Debian-Security";
#         "origin=Debian,codename=\${distro_codename}-security,label=Debian-Security";
# };
# EOF
# echo "0 2 * * * unattended-upgrade" > /etc/cron.d/security-upgrade

# using dump-init as described in the docs
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/usr/local/bin/docker-entrypoint.sh"]

/usr/local/bin/docker-entrypoint.sh

#!/bin/sh
service cron start
exec gosu airflow /entrypoint "$@"

The container is not restarted, therefore all processes should be run as airflow user and from my understanding gosu should not have an impact. The change of the UID with usermod -u 1000 airflow is also permanent and should not have an occasional impact on runtime.

I can not set the user property in my docker-compose file because then the required permission to start the cron service are missing. I have set the AIRFLOW_UID to 1000 for the airflow-init container.

• Is there an "official" option to have regular security upgrades installed without re-building and re-deploying airflow?

[parthp1808]

I also have exact same issue. Seems like race condition or something weird but even AI can’t seem to catch it yet.

Have you tried any solution yet? I am thinking to implement cron job to precreate future dated folder.

[jgoedeke]

Hi @parthp1808,
at the moment we are living with this issue. Do you use gosu as well or do you have a "clean" setup? Our plan for the future is to monitor this when upgrading to Airflow 3.0 and if it persists build a scheduled re-build/re-deploy mechanism instead of using unattanded-upgrade.

[potiuk]

Is there an "official" option to have regular security upgrades installed without re-building and re-deploying airflow?

You should rebuild and redeploy your image whenever you have new image - and you can rebuild image as described "officially" here: https://airflow.apache.org/docs/docker-stack/index.html#what-should-i-do-if-my-security-scan-shows-critical-and-high-vulnerabilities-in-the-image

This makes completely no sense to run unnattended upgrades daemon inside your image because all the upgrades will be immediately lost when your docker container restarts - including cases where your deploymend decides to restart it - and after restart - until your "upgrade" service will do it's job, your image will run older version of software - more than that - in case you have unattended upgrades that require restart - it's impossible to get them in, because ... you are loosing all changes after restart. So - what you are doing is just wrong.

You should have a pipeline to rebuild your image when needed and use the rebuilt image and restart your airflow containers with this new image. Full stop. There is no other way.

[jgoedeke]

Agreed. We actually run unattended-upgrade in the entrypoint once as well, therefore we are save to restart. It just takes a bit longer.