[resources] support async yield_for_execution #28142
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alangenfeld
force-pushed
the
al/02-28-_resources_support_async_yield_for_execution
branch
from
60641e3 to
b0a6b8b
Compare
alangenfeld
force-pushed
the
al/02-28-_resources_support_async_yield_for_execution
branch
from
b0a6b8b to
2ac2079
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
alangenfeld
force-pushed
the
al/02-28-_resources_support_async_yield_for_execution
branch
from
2ac2079 to
d85553d
Compare
Comment on lines
121
to
132
| def gen_from_async_gen( | ||
| context: StepExecutionContext, | ||
| async_gen: AsyncIterator[T], | ||
| ) -> Iterator[T]: | ||
| # prime use for asyncio.Runner, but new in 3.11 and did not find appealing backport | ||
| loop = asyncio.new_event_loop() | ||
| try: | ||
| while True: | ||
| try: | ||
| yield loop.run_until_complete(async_gen.__anext__()) | ||
| except StopAsyncIteration: | ||
| return | ||
| finally: | ||
| loop.run_until_complete(loop.shutdown_asyncgens()) | ||
| loop.close() | ||
| # try: | ||
| while True: | ||
| try: | ||
| yield context.event_loop.run_until_complete(async_gen.__anext__()) | ||
| except StopAsyncIteration: | ||
| return | ||
| # finally: |
There was a problem hiding this comment.
The commented try/finally blocks should be removed since they're not being used. More importantly, this code needs proper cleanup of async generators to prevent resource leaks. The event loop's shutdown_asyncgens() call that was previously in the finally block needs to be preserved somewhere in the execution lifecycle, likely at a higher level where the event loop itself is managed.
Spotted by Graphite Reviewer
Is this helpful? React 👍 or 👎 to let us know.
alangenfeld
force-pushed
the
al/02-28-_resources_support_async_yield_for_execution
branch
from
d85553d to
f3f32c5
Compare
alangenfeld
force-pushed
the
al/02-28-_resources_support_async_yield_for_execution
branch
from
f3f32c5 to
dc5d2e3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for async context manager backed resources by passing the same event loop to resource init and core execution.
The biggest bit of awkwardness here is that to ensure the same event loop is used for both resource init and execution in the direct execution case you have to manually pass the loop in to context creation and then use it to execute your asset/op. This is demonstrated in the test.
resolves #27974
How I Tested These Changes
added tests
Changelog
async def yield_for_executionis now supported onConfigurableResource. Anevent_loopargument has been added to context builders to support direct execution.