Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rmq move out as an extra module #6

Merged
merged 18 commits into from
Jan 17, 2025
Merged

Rmq move out as an extra module #6

merged 18 commits into from
Jan 17, 2025

Conversation

unkcpz
Copy link
Owner

@unkcpz unkcpz commented Jan 17, 2025

The refactoring is targeting to decouple the dependencies of using kiwipy+rmq as the communicator for the process control.
By forming a Coordinator protocol contract, the different type of rmq/kiwipy related codes are removed out from plumpy logic. The new contract also pave the way to make it clearly show how a new type coordinator can be implemented (future examples will be the tatzelwurm a task broker that has scheduler support and file based task broker require no background service).
For the prototype of how a coordinator should look like, the MockCoordinator in tests/utils is the coordinator that store things in memory, and can serve as the lightweight ephemeral daemon without persistent functionality.

Another major change here is hand write the resolver of future by mimic how tho asyncio does for wrapping concurrent.futures.Future into asyncio.Future. I use the same way to convert asyncio.Future into concurent.futures.Future (which is the kiwipy.Future as alias).

  • move the aio_pika import lazily by moving the rmq exceptions to rmq module, this can increase the performance of import aiida; aiida.orm.
  • CancellableAction using composite to behave as a Future like object.
  • use asyncio.Future in favor of alias plumpy.Future and
  • use concurrent.futures.Future instead of alias kiwipy.Future.
  • Hand write _chain and _copy_future since we can not just rely on the API of asyncio that is not exposed.
  • Forming the coordinator/Communicator protocol.
  • Just forming the coordinator/Coordinator protocol and wrap rmq/communicator as a coordinator that not require changs in kiwipy.
  • Mock the coordinator for unit test.
  • test against aiida-core see what need to be changed there and improve here. (Decouple broker and coordinator interface aiidateam/aiida-core#6675)
  • The API for plumpy process can be more compact instead of using kiwipy/rmq "subscriber" concept. (how to replace rpc pattern??)

@unkcpz unkcpz mentioned this pull request Jan 17, 2025
@unkcpz unkcpz force-pushed the rmq-out branch 4 times, most recently from 2304d5f to 6e84968 Compare January 17, 2025 22:37
@unkcpz unkcpz force-pushed the rmq-out branch 2 times, most recently from 4972dd2 to 6d3101d Compare January 17, 2025 22:42
@unkcpz unkcpz merged commit 5399404 into master Jan 17, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant