OpenSearch Task Logging in Airflow 3

[devgonvarun]

Airflow 3.0.2
apache-airflow-providers-opensearch 1.7.1
Kubernetes Executor

Presently Airflow only supports reading task logs from OpenSearch unlike ElasticSearch where both reading and writing is supported. So logshipper tools like Fluentd are required to take local task logs to OpenSearch and when logs are in OpenSearch then Airflow's OpenSearch provider package can query OpenSearch using a log_id to fetch the logs to the Airflow UI.

The log_id that the OpenSearch task handler uses is {dag_id}-{task_id}-{logical_date}-{try_number} whereas the default value of the log_id_template (AIRFLOW__OPENSEARCH__LOG_ID_TEMPLATE) is {dag_id}-{task_id}-{run_id}-{map_index}-{try_number}. Why this difference? Moreover me overriding the AIRFLOW__OPENSEARCH__LOG_ID_TEMPLATE has no effect, the log_id used to query opensearch is always {dag_id}-{task_id}-{logical_date}-{try_number}.

Also, logical_date is not part of the kubernetes pod labels. So i dont know how to construct the log_id with logical_date in Fluentd? Can someone please guide me? Is this a bug?