'hello_world_dag' is failing on the 3.0.6 image

[agpiermarini]

Describe the bug
The hello_world_dag is failing on the Airflow 3.0.6 image when running locally, with the following error:

Reading remote log from Cloudwatch log_group: arn:aws:logs:us-west-2:<ACCOUNT_ID>:log-group:AIRFLOW_DEV-Task log_stream: dag_id=hello_world_dag/run_id=scheduled__2025-10-05T20:08:41.638098+00:00/task_id=print_task/attempt=1.logAn error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist

Health checks pass and api/v2/monitor/health returns healthy statuses across the board. The only indication of an error during container startup is the following message emitted by the scheduler and worker:

/usr/local/airflow/.local/lib/python3.12/site-packages/watchtower/__init__.py:432 WatchtowerWarning: Received empty message. Empty messages cannot be sent to CloudWatch Logs

---

To Reproduce
Steps to reproduce the behavior:

1. Run python3 ./create_venvs.py --recreate --target development
2. Configure images/airflow/3.0.6/run.sh per the README
3. CD into images/airflow/3.0.6 and run ./run.sh
4. Navigate to webserver URL
5. Enable hello_world_dag DAG; trigger run manually
6. Inspect logs

---

System Information

• OS: Mac Sequoia 15.1 (24B83)
• Kernel version: 24.1.0
• System architecture: arm64
• Browser version (if the bug report is related to the web server): [e.g. Firefox 121.0 (64-bit)]
• Airflow version used: 3.0.6
• aws-mwaa-docker-images commit ID: 7a1d103

---

Expected behavior
hello_world_dag runs and writes to log streams successfully.

---

Screenshots
N/A

---

Additional context
I am attempting to use the amazon-mwaa-docker-images repo as a stand-in for aws-mwaa-local-runner, per the suggestion here: aws/aws-mwaa-local-runner#446.

When testing 3.0.6, I confirmed that:

1. Log groups are being created in CloudWatch successfully.
2. The role tied to the credentials has the requisite access.
3. Using the credentials, I can successfully create the log streams via the AWS CLI.
4. The problem can be reproduced when mounting custom Airflow directories.
5. The problem persists after deleting the images and storage volumes and rebuilding everything via the ./run.sh script.

I was able to run the hello_world_dag after building 2.10.3 from the same repository.

[anrajago]

Thanks for bringing this to our attention. I will try to reproduce this and get back!

[agpiermarini]

Thanks for bringing this to our attention. I will try to reproduce this and get back!

Thanks for taking a look, @anrajago. Please let me know if you need any more info.

[shuklasaharsh]

We are facing a similar issue on production. Any luck?

[anrajago]

Hello @agpiermarini

I tried to replicate this in my local environment, and I was not able to. I have setup my run.sh per the readme and I'm able to run the hello_world DAG and also get logs from the CW stream in the airflow UI:

Reading remote log from Cloudwatch log_group: arn:aws:logs:us-west-2::log-group:anrajago_test_306-Task log_stream: dag_id=hello_world_dag/run_id=scheduled__2025-10-16T05:57:26.830915+00:00/task_id=print_task/attempt=1.log
[2025-10-16, 11:27:27] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-10-16, 11:27:27] INFO - Filling up the DagBag from /usr/local/airflow/dags/hello_world.py: source="airflow.models.dagbag.DagBag"
[2025-10-16, 11:27:27] INFO - Hello, World!: chan="stdout": source="task"
[2025-10-16, 11:27:27] INFO -

Can you please check the credentials and the account_id you're using?

[anrajago]

@shuklasaharsh Is the issue in production related to dynamic task mapping? If so, we have an open PR: #349

[AaronTheApe]

@shuklasaharsh @anrajago I was able to reproduce this issue. How I fixed:

1. I'm in us-east-1 but docker compose is hardcoded to use-west-2
2. this variable needs to be this value.. I was using localhost 🤦 MWAA__CORE__API_SERVER_URL="http://webserver:8080"
3. other places AWS_DEFAULT_REGION... I overwrote that usage with: logs_client: CloudWatchLogsClient = boto3.client("logs", region_name=self.region_name)
4. probably could just get AWS_DEFAULT_REGION defined to what you want and not do 3... but haven't tried that yet... but assuming it was either getting us-east-2 or us-west-2

[agpiermarini]

Hello @agpiermarini

I tried to replicate this in my local environment, and I was not able to. I have setup my run.sh per the readme and I'm able to run the hello_world DAG and also get logs from the CW stream in the airflow UI:

Reading remote log from Cloudwatch log_group: arn:aws:logs:us-west-2::log-group:anrajago_test_306-Task log_stream: dag_id=hello_world_dag/run_id=scheduled__2025-10-16T05:57:26.830915+00:00/task_id=print_task/attempt=1.log
[2025-10-16, 11:27:27] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-10-16, 11:27:27] INFO - Filling up the DagBag from /usr/local/airflow/dags/hello_world.py: source="airflow.models.dagbag.DagBag"
[2025-10-16, 11:27:27] INFO - Hello, World!: chan="stdout": source="task"
[2025-10-16, 11:27:27] INFO -

Can you please check the credentials and the account_id you're using?

Thank you for looking into this, @anrajago and @AaronTheApe!

I managed to get 3.0.6 running with @AaronTheApe's good guidance. The issue was that I had been setting my MWAA__CORE__API_SERVER_URL to http://localhost:8080; it was resolved when I changed it to http://webserver:8080.