Skip to content

An asyncio API that mimics concurrent.futures, with support for task-graph executors

License

Notifications You must be signed in to change notification settings

agoose77/parallels

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

parallels

pypi-badge

Parallels is a literary powered library. It provides an async API that mimics concurrent.futures, with support for task-graph executors.

These notebooks can be viewed using nbviewer until the documentation generator is complete.

What?

In Python there are several standard APIs for interacting with executors. AsyncIO has the run_in_executor API, concurrent.futures has the Executor API, and other libraries like Dask and Ray have equivalent approaches. concurrent.futures is often available within other libraries, but its reduced features-set prevents the underlying library from implementing useful optimisations like Dask's deferred computation or task graph building.

Parallels implements a standard Executor interface which defines a synchronous submit method, and an asynchronous retrieve method. These methods operate upon value-less asyncio.Future handles which yield True upon task success, and raise an Exception otherwise. The Dask and Ray implementations accept these handles as arguments to future submit() calls, which can be used to build task graphs and avoid copying data to the local machine.

About

An asyncio API that mimics concurrent.futures, with support for task-graph executors

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published