Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add page on parallelisation #60

Open
4 tasks
JostMigenda opened this issue Mar 7, 2025 · 1 comment
Open
4 tasks

Add page on parallelisation #60

JostMigenda opened this issue Mar 7, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@JostMigenda
Copy link
Collaborator

JostMigenda commented Mar 7, 2025

In #59 I’ve added a code sample using a concurrent.futures.ThreadPoolExecutor to parallelise downloading files.

I think it would be good to expand on that and have a whole page with more information:

  • Explain the GIL (as of Python 3.13, it can be disabled, but this is not yet ready for general usage: requires a separate Python build, lack of compatibility with modules using the C-API, 5–10% slowdown for single-threaded code)
  • Difference between ThreadPoolExecutor and ProcessPoolExecutor
  • async/await
  • Clear flowchart of which method to use when

I’ve got an example for ThreadPoolExecutor already (in #59); for ProcessPoolExecutor I could adapt an example from an astrophysics code I’ve developed.

I’m not sure about how to handle async/await. Since it’s conceptually more challenging, maybe it’s better to keep that section very short and high level? And just say “if you have a use case for this [enumerate what those would be], then see the following links for further information”?

@Robadob Robadob added the enhancement New feature or request label Mar 9, 2025
@Robadob
Copy link
Member

Robadob commented Mar 9, 2025

Yes, whilst the new example from #59 is clearly worth showing it feels like it's missing some context that more novice course attendees would benefit from.

Similar applies to your astrophysics code here, it would be some of the harder to parse(?) code from the course so some handholding would be preferable (and maybe more detail in an instructor note?).

I’m not sure about how to handle async/await. Since it’s conceptually more challenging, maybe it’s better to keep that section very short and high level?

Yeah something to the effect of, "more advanced parallel concepts such as x,y exist in Python however these are outside the scope of the course (would require a course of their own!)...learn more here".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants