Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/add metadata el dag #373

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Conversation

edualvess
Copy link
Contributor

Overview

This PR addresses the development of a new DAG for the ingestion of some relevant Airflow metastore into BigQuery to provide data for the Airflow Monitoring modeling and dashboard.

Key Changes

  • Added JSON schema files for the five selected tables (dag, dag_run, task_instance, task_fail, sla_miss)
  • Added variable with information related to the tables extracted
  • Added metadata_ingestion_el_dag.py script containing the DAG code

Other Notes

This early approach to the related tables did not consider date partitioning, but it can be adjusted if required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant