AIP-66: Add support for parsing DAG bundles #45371
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:API
uranusjr
reviewed
Jan 3, 2025
uranusjr
reviewed
Jan 3, 2025
jedcunningham
force-pushed
the
bundle-parsing
branch
from
8929489 to
5fa6022
Compare
Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).
jedcunningham
force-pushed
the
bundle-parsing
branch
from
5fa6022 to
5f5b30c
Compare
(cherry picked from commit cc6c16c)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DRAFT: I'm expecting a (hopefully not too large) number of test failures to work through.
Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.
This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).