Running new tasks for existing runs

[agalst]

Hi,

Lets assume I have a DAG, which has a bunch of successful runs. I have added new tasks to this DAG.

In the UI I can select the run and click on "Queue up new tasks" to make those new tasks run. Is there a functionality in the CLI which would allow me to run those new tasks across many runs for this DAG using a regex for the task name?

Ive considered the following options:

• backfilling: creates backfill runs in UTC while regular runs were in local timezone
• tasks clear: doesnt influence the tasks which never run
• tasks run: doesnt support regex
• create another DAG with new tasks is not an option

[potiuk]

only backfill is your option AFAIK.

[qcha41]

I am in the exact same situation.

Backfill does the job indeed, but is absolutely not convenient, as it transforms existing "scheduled" dag runs into "backfill" dag runs, which prevent to interact with them afterwards from the UI (to clear them, etc.)

It would be really convenient to have the exact same logic as the "Queue up new tasks" action implemented in the CLI / rest api, but with the ability to specify the start_date and the end_date as in the backfill CLI command. In that case, reloaded existing dags would be still usable through the UI.

What's your opinion on that ?

For now, what I'm doing is airflow tasks render followed by airflow tasks clear but it is not practical as the render command is only working for one specific execution_date (or run_id).

I just opened an issue for that #38342

[JurajPalka]

How is this done in Airflow 3.0.2? In the UI the Queue up new tasks is unclickable and I can't find any way to use CLI for it as mentioned above which was true in Airflow 2.x. Is there really no way to queue up new tasks in 3.x?

[december-man]

Same on Airflow 3.0.6.
We've used this functionality a lot on Airflow 2.x.

Now, as a workaround, you'd have to start another task in the latest dagrun to trigger execution of the newly added task. Then, by clicking on it in the task list, you can find the associated task instance, and click on it to find all the information about it.

[potiuk]

If #52177 (coming in 3.1.0) won't solve your issue then I suggest to create a feature request.