Multiple Airflow Schedulers Processing Same Dagruns?

[JakeKandell]

Hi all, we run a large Airflow cluster and recently encountered some strange scheduling issues. We have a distributed Kubernetes setup with multiple scheduler pods and a MySQL DB on Airflow 2.9.2.

In the scheduler logs, we see that there are multiple schedulers processing the same finished dagrun simulatenously. They all try to mark this dagrun as failed.

POD: airflow-holdem-scheduler-6fb47f86fd-687gw
[2025-06-25T12:41:02.170+0000] {dagrun.py:822} ERROR - Marking run <DagRun hive_tracking_bmr_holdem__data_motion_kafka_airflow @ 2025-06-25 11:00:00+00:00: scheduled__2025-06-25T11:00:00+00:00, state:running, queued_at: 2025-06-25 11:51:22+00:00. externally triggered: False> failed

POD: airflow-holdem-scheduler-6fb47f86fd-nlj74
[2025-06-25T12:41:02.612+0000] {dagrun.py:822} ERROR - Marking run <DagRun hive_tracking_bmr_holdem__data_motion_kafka_airflow @ 2025-06-25 11:00:00+00:00: scheduled__2025-06-25T11:00:00+00:00, state:running, queued_at: 2025-06-25 11:51:22+00:00. externally triggered: False> failed

Furthermore, some of the schedulers report active_runs=1

[2025-06-25T12:41:02.986+0000] {scheduler_job_runner.py:1333} INFO - DAG hive_tracking_bmr_holdem__data_motion_kafka_airflow is at (or above) max_active_runs (1 of 1), not creating any more runs

Other schedulers try to set the next_dagrun (meaning active_runs was correctly identified as 0)

[2025-06-25T12:41:03.767+0000] {dag.py:3954} INFO - Setting next_dagrun for hive_tracking_bmr_holdem__data_motion_kafka_airflow to 2025-06-25 12:00:00+00:00, run_after=2025-06-25 12:30:00+00:00

We believe this may have led to a race condition which caused next_dagrun_create_after to be set to NULL preventing this DAG from being properly scheduled moving forward.

Here are the full logs for this DAG.

From my review of the code, I see that the dagrun rows are supposed to be locked when selected, so it's not clear to me how the same dagrun is being processed by different schedulers. I haven't been able find a way to reproduce this in a test cluster.

Has anyone experienced this? Do we know how this can happen?