Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][experimental] Remove separate asyncio path for accelerated DAGs #46307

Open
stephanie-wang opened this issue Jun 27, 2024 · 0 comments
Open
Labels
accelerated-dag enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks

Comments

@stephanie-wang
Copy link
Contributor

Description

Right now asyncio support in accelerated DAGs uses a slightly different codepath from normal DAGs:

  • you need to specify enable_asyncio=True in dag.experimental_compile()
  • you need to use dag.execute_async
  • you need to await twice, once for the execution and again for the output

This is because asyncio execution currently starts a background thread to read/write channels, so we need these extra APIs for coordinating with the background thread. With timeout support for channel reads and writes (#46259), we might be able to remove some of these APIs and support asyncio with single-threaded execution, using mostly the same codepath as non-asyncio execution.

Use case

No response

@stephanie-wang stephanie-wang added enhancement Request for new feature and/or capability triage Needs triage (eg: priority, bug/not-bug, and owning component) P1 Issue that should be fixed within a few weeks accelerated-dag and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accelerated-dag enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks
Projects
None yet
Development

No branches or pull requests

1 participant