Skip to content
This repository has been archived by the owner on Feb 4, 2022. It is now read-only.

Investigate paralysation of yielding structured references #507

Open
lizgzil opened this issue Apr 30, 2020 · 0 comments
Open

Investigate paralysation of yielding structured references #507

lizgzil opened this issue Apr 30, 2020 · 0 comments
Assignees
@lizgzil
Labels
Data science investigation Evaluation of data science methods or analysis

Comments

@lizgzil
Copy link
Contributor

lizgzil commented Apr 30, 2020

In split_reach/extracter/extract_refs_task.py we set pool_map = map for use in yield_structured_references. However if we utilise Pool from multiprocessing i.e.

pool = Pool(num_workers)
pool_map = pool.map

we could speed up this task.
However in the past we found using num_workers>1 actually slowed things down.
So worth investigating how this behaves now to see whether it's worth implementing.
@lizgzil lizgzil added the Data science investigation Evaluation of data science methods or analysis label Apr 30, 2020
@lizgzil lizgzil self-assigned this Apr 30, 2020
@lizgzil lizgzil mentioned this issue Apr 30, 2020
Merged
11 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@lizgzil lizgzil

Labels
Data science investigation Evaluation of data science methods or analysis
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lizgzil