Skip to content

Commit

Permalink
Update notebook
Browse files Browse the repository at this point in the history
  • Loading branch information
Madhukar Mishra committed Oct 27, 2018
1 parent c96d4c6 commit 7700e6d
Showing 1 changed file with 65 additions and 77 deletions.
142 changes: 65 additions & 77 deletions python_concurrency.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@
"metadata": {},
"source": [
"### cocurrency is the way forward\n",
"- Ahmdal's Law provides the upperbound from gains acquired from _parallel_ execution.\n",
"- allows for scaling 'horizontally'\n",
"- explains the sudden surge of interest in Erlang, Scala, Clojure and functional programming."
]
Expand Down Expand Up @@ -190,10 +189,10 @@
"metadata": {},
"source": [
"## Concurrency is not parallelism\n",
"Concurrency is not parallelism\n",
"\n",
"concurrency: Dealing with a lot at once, property of the solution (code)\n",
"parallelism: Doing a lot at once, property of the runtime"
"__concurrency__: Dealing with a lot at once, property of the solution (code)\n",
"\n",
"__parallelism__: Doing a lot at once, property of the runtime"
]
},
{
Expand All @@ -205,13 +204,45 @@
"- python threads don't run parallely, you can still do concurrency."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Multithreading in python\n",
"\n",
"__Advantages__\n",
"- excellent for speeding up blocking I/O bound programs\n",
"- Scheduled preemptively\n",
"- Lighter memory footprint than processes\n",
"- Shared memory - faster communication\n",
"\n",
"__Disdavantages__\n",
"- GIL means that we can't take advantages of multiple cores.\n",
"- Context switch may happen whenever, have to synchronize access to critical section.\n",
"- Hard to test and debug\n",
"- Context switches are hardly free. Can see degradation in performance."
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"scrolled": true
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello, World!, I am thread Thread-1\n",
"Hello, World!, I am thread Thread-2\n",
"Hello, World!, I am thread Thread-3\n",
"Hello, World!, I am thread Thread-4\n",
"Hello, World!, I am thread Thread-5\n",
"elapsed time 5 s\n"
]
}
],
"source": [
"# sequential_client.py\n",
"import requests\n",
Expand Down Expand Up @@ -259,26 +290,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Multithreading in python\n",
"\n",
"__Advantages__\n",
"- excellent for speeding up blocking I/O bound programs\n",
"- Scheduled preemptively\n",
"- Lighter memory footprint than processes\n",
"- Shared memory - faster communication\n",
"\n",
"__Disdavantages__\n",
"- GIL means that we can't take advantages of multiple cores.\n",
"- Context switch may happen whenever, have to synchronize access to critical section.\n",
"- Hard to test and debug\n",
"- Context switches are hardly free. Can see degradation in performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The dreaded GIL\n",
"### GIL the good parts\n",
"- Python has one global lock, on the entire interpreter instead of thousands of granular locks everywhere else.\n",
"- GIL makes it easy to integrate non thread safe C libraries\n",
"- Locking and unlocking is not free, previous attempts at removing the GIL has resulted in severe degradation of the performance of a single thread."
Expand Down Expand Up @@ -314,18 +326,9 @@
},
{
"cell_type": "code",
"execution_count": 48,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True\n",
"took 0:00:07.067246\n"
]
}
],
"outputs": [],
"source": [
"# cpu_bound_sequential.py\n",
"from datetime import datetime\n",
Expand All @@ -338,24 +341,14 @@
"print(all([cpu_bound_task() for i in range(1000)]))\n",
"end_time = datetime.now()\n",
"elapsed_time = end_time - start_time\n",
"print(f\"took {elapsed_time}\")\n",
"\n"
"print(f\"took {elapsed_time}\")"
]
},
{
"cell_type": "code",
"execution_count": 46,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True\n",
"took 0:00:06.767392\n"
]
}
],
"outputs": [],
"source": [
"# cpu_bound_multithreaded.py\n",
"from datetime import datetime\n",
Expand All @@ -376,18 +369,9 @@
},
{
"cell_type": "code",
"execution_count": 47,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True\n",
"took 0:00:03.564588\n"
]
}
],
"outputs": [],
"source": [
"# cpu_bound_multiprocess.py\n",
"from multiprocessing import Pool\n",
Expand Down Expand Up @@ -482,22 +466,9 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"b'Hello, World!, I am thread Thread-44'\n",
"b'Hello, World!, I am thread Thread-42'\n",
"b'Hello, World!, I am thread Thread-45'\n",
"b'Hello, World!, I am thread Thread-43'\n",
"b'Hello, World!, I am thread Thread-46'\n",
"took 1 seconds\n"
]
}
],
"outputs": [],
"source": [
"#async_client.py\n",
"import aiohttp\n",
Expand Down Expand Up @@ -529,6 +500,19 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recapping\n",
"- Sync: Blocking operations.\n",
"- Concurrency: Making progress together.\n",
"- Async: Non blocking operations.\n",
"- Parallelism: Making progress in parallel.\n",
"- locking: avoid"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"| feature/ library | threading | multiprocessing | asyncio |\n",
"|------------------------|-----------------------------------------|-------------------|--------------------------------------------------|\n",
"| scheduling | preemptive | preemptive | cooperative |\n",
Expand All @@ -543,7 +527,11 @@
"metadata": {},
"source": [
"## Links\n",
"[python concurrency story](https://powerfulpython.com/blog/python-concurrency-story-pt1/)"
"\n",
"- [python concurrency story](https://powerfulpython.com/blog/python-concurrency-story-pt1/)\n",
"- [Python concurrency keynote - Raymond Hettinger](https://www.youtube.com/watch?v=9zinZmE3Ogk)\n",
"[Python Concurrency From the Ground Up - David Beazley](https://www.youtube.com/watch?v=MCs5OvhV9S4)\n",
"- [github repo for the talk](https://github.com/madhukar93/python_concurrency)"
]
}
],
Expand Down

0 comments on commit 7700e6d

Please sign in to comment.