Problem
Currently, our task manager effectively parallelizes tasks that are independent of each other. However, it falls short when dealing with cases where parallelization has dependencies among tasks.
Desired Enhancement
We aim to enhance our project's capabilities to effectively manage scenarios involving task dependencies. To illustrate this with an example:
Consider a Reinforcement Learning pipeline where Gen1 comprises a set of 20 agents competing against each other in subsets of 4. The winners from these competitions undergo mutations to create the 20 agents of Gen2. While the competition of each subset in Gen1 can be parallelized, the computation of Gen2 can only begin when it has at least 4 agents ready.
With our current Task Manager, orchestrating these dependent computations proves to be a challenging task.
Why This Enhancement is Important
This enhancement will make our task manager more versatile and adaptable to real-world use cases. Many tasks have dependencies that cannot be ignored, and by introducing this feature, we empower users to make better use of parallelization while respecting dependencies.
Problem
Currently, our task manager effectively parallelizes tasks that are independent of each other. However, it falls short when dealing with cases where parallelization has dependencies among tasks.
Desired Enhancement
We aim to enhance our project's capabilities to effectively manage scenarios involving task dependencies. To illustrate this with an example:
Consider a Reinforcement Learning pipeline where Gen1 comprises a set of 20 agents competing against each other in subsets of 4. The winners from these competitions undergo mutations to create the 20 agents of Gen2. While the competition of each subset in Gen1 can be parallelized, the computation of Gen2 can only begin when it has at least 4 agents ready.
With our current Task Manager, orchestrating these dependent computations proves to be a challenging task.
Why This Enhancement is Important
This enhancement will make our task manager more versatile and adaptable to real-world use cases. Many tasks have dependencies that cannot be ignored, and by introducing this feature, we empower users to make better use of parallelization while respecting dependencies.