Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessor support #3

Open
mbelmadani opened this issue Aug 17, 2016 · 0 comments
Open

Multiprocessor support #3

mbelmadani opened this issue Aug 17, 2016 · 0 comments

Comments

@mbelmadani
Copy link
Owner

mbelmadani commented Aug 17, 2016

Modify engine.py to evaluate of solutions in parallel. (add toolbox.register("map", some_mapping_function)).

Possibilities include python multiprocessing, Scoop, or pathos. The multiprocessing module is standard python, while Scoop appears to integrate well with DEAP and seems to extend beyong multiprocessing (e.g support for clusters), which is nice. The issue with these modules is that they don't automatically serialize instance methods and certain modules, required by the function mapped to the toolbox.evaluate method, which makes it difficult to use with motifgp without significant modifications. Pathos appears to handle serialization by default, so it might a good start point.

An issue with this multiprocessing is that it might break memoization since the dict for the cache is not shared, so it probably won't be updated or shared between evaluations. This might be possible to fix with a workaround the shared memory using. If not, it might be better to add cache-less parallelism, or reconsider python multiprocessing or scoop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant