Fixes and improves examples in README and documentation. (finos#32)

* Fixes and improves examples in README and documentation.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Shutdown any currently running backend when exiting Python.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Adds new examples (random forest and bigram counter).

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Adds the examples to the documentation.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Bump version number.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Run and test the documentation examples before publishing.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Donwload the example file for the bigram example instead of storing it in the repository.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Removes the use of argparse in examples.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Shows the package version in the documentation's sidebar.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Uses Scaler's default address for the local managed backend.

Signed-off-by: rafa-be <raphael@noisycamp.com>

* Rewrite and improve examples from the quickstart tutorial.

Signed-off-by: rafa-be <raphael@noisycamp.com>

---------

Signed-off-by: rafa-be <raphael@noisycamp.com>

Author: rafa-be

.github/workflows/linter.yml

@@ -34,7 +34,7 @@ jobs:
           python -m pip install --upgrade pip
           pip install flake8 pyproject-flake8 mypy
           pip install -r requirements.txt
-          pip install pandas dask[distributed]
+          pip install pandas dask[distributed] scaler
       - name: Lint with flake8
         run: |
           pflake8 .
@@ -44,3 +44,14 @@ jobs:
       - name: Run python unittest
         run: |
           python -m unittest discover -v tests
+      - name: Run the examples
+        run: |
+          find examples -type f -name '*.py' ! -name '__init__.py' | while read -r file; do
+            echo "Running $file"
+            PYTHONPATH=. python "$file"
+          done
+      - name: Test the documentation examples
+        run: |
+          pip install -r docs/requirements.txt
+          cd docs
+          make doctest

README.md

@@ -31,12 +31,16 @@ and distributed systems**.
 The main feature of the library is its `@parfun` decorator that transparently executes standard Python functions
 following the [map-reduce](https://en.wikipedia.org/wiki/MapReduce) pattern:
 
+
 ```Python
+from typing import List
+
 from parfun import parfun
 from parfun.combine.collection import list_concat
 from parfun.partition.api import per_argument
 from parfun.partition.collection import list_by_chunk
 
+
 @parfun(
     split=per_argument(
         values=list_by_chunk
@@ -45,32 +49,42 @@ from parfun.partition.collection import list_by_chunk
 )
 def list_pow(values: List[float], factor: float) -> List[float]:
     return [v**factor for v in values]
+
+
+if __name__ == "__main__":
+    from parfun.entry_point import set_parallel_backend_context
+
+    with set_parallel_backend_context("local_multiprocessing"):  # use a local pool of processes
+        print(list_pow([1, 2, 3], 2))  # runs in parallel, prints [1, 4, 9]
 ```
 
 
 ## Features
 
-* **Provides significant speedups** to existing Python functions
-* **Does not require any deep knowledge of parallel or distributed computing systems**
-* **Automatically estimates the optimal sub-task splitting** (the *partition size*)
-* **Automatically handles data transmission, caching and synchronization**.
-* **Supports various distributed computing backends**, including Python's multiprocessing,
-  [Scaler](https://github.com/citi/scaler) or Dask.
+* **Provides significant speedups** to existing Python functions.
+* **Only requires basic understanding of parallel and distributed computing systems**.
+* **Automatically estimates the optimal sub-task splitting strategy** (the *partition size*).
+* **Automatically handles data transmission, caching, and synchronization**.
+* **Supports various distributed computing backends**:
+    - Python's built-in [multiprocessing module](https://docs.python.org/3/library/multiprocessing.html).
+    - [Scaler](https://github.com/citi/scaler).
+    - [Dask](https://www.dask.org/).
 
 
-## Benchmarks
+## Quick Start
 
-**Parfun efficiently parallelizes short-duration functions**.
 
-When running a short 0.28-second ML function on an AMD Epyc 7313 16-Cores Processor, Parfun provides an impressive
-**7.4x speedup**. Source code for this experiment [here](benchmarks/california_housing.py).
+Install Parfun directly from PyPI:
 
-![Benchmark Results](benchmarks/california_housing_results.svg)
+```bash
+pip install parfun
+pip install "parfun[pandas,scaler,dask]"  # with optional dependencies
+```
 
+The official documentation is available at [citi.github.io/parfun/](https://citi.github.io/parfun/).
 
-## Quick Start
-
-The official documentation is availaible at [citi.github.io/parfun/](https://citi.github.io/parfun/).
+Take a look at our documentation's [quickstart tutorial](https://citi.github.io/parfun/tutorials/quickstart.html) to get
+more examples and a deeper overview of the library.
 
 Alternatively, you can build the HTML documentation from the source code:
 
@@ -80,10 +94,17 @@ pip install -r requirements.txt
 make html
 ```
 
-The documentation's main page can then ben found at `docs/build/html/index.html`.
+The documentation's main page can then be found at `docs/build/html/index.html`.
 
-Take a look at our documentation's [quickstart tutorial](https://citi.github.io/parfun/tutorials/quickstart.html) to get
-more examples and a deeper overview of the library.
+
+## Benchmarks
+
+**Parfun effectively parallelizes even short-duration functions**.
+
+When running a short 0.28-second ML function on an AMD Epyc 7313 16-Cores Processor, Parfun provides an impressive
+**7.4x speedup**. Source code for this experiment [here](examples/california_housing/main.py).
+
+![Benchmark Results](images/benchmark_results.svg)
 
 
 ## Contributing

docs/source/_static/style.css

@@ -1,4 +1,11 @@
 {% extends "!layout.html" %}
-{% block extrahead %}
-  <link href="{{ pathto("_static/style.css", True) }}" rel="stylesheet" type="text/css">
+
+{% block sidebartitle %}
+  <div class="switch-menus">
+    <div class="version">
+      <strong>Version:</strong> {{ version }}
+    </div>
+  </div>
+
+  {{ super() }}
 {% endblock %}

docs/source/_templates/layout.html

@@ -21,7 +21,6 @@
 project = "Parfun"
 author = "Citi"
 
-
 version = __import__("parfun").__version__
 release = f"{version}-py3-none-any"
 

docs/source/conf.py

@@ -1,13 +1,12 @@
 Welcome to Parfun's documentation!
 ================================================
 
-**Parfun** is a lightweight and user-friendly Python library to assist users in running a pure Python function i
-parallel on multiple core and distributed systems.
+**Parfun** is a lightweight and user-friendly Python library to assist users in running a Python function in
+parallel.
 
-Users, who do not have any deep knowledge about parallelism and distributed computing systems, can benefit from
-significant speedups when running Python code.
+With limited knowledge of parallelism and distributed systems, users can significantly speedup their Python code.
 
-Parfun supports multiple execution backend, including Python's multiprocessing, Dask or Scaler.
+Parfun supports multiple execution backend, including Python's built-in multiprocessing, Dask and Scaler.
 
 
 Content
@@ -17,6 +16,7 @@ Content
    :maxdepth: 2
 
    tutorials/quickstart
+   tutorials/examples
    tutorials/implementation_details
    api/index
 

docs/source/index.rst

@@ -0,0 +1,17 @@
+Examples
+========
+
+Parallely count bigrams in a text
+---------------------------------
+
+.. literalinclude:: ../../../examples/count_bigrams/main.py
+   :language: python
+   :linenos:
+
+
+Parallely train a random tree regressor
+---------------------------------------
+
+.. literalinclude:: ../../../examples/california_housing/main.py
+   :language: python
+   :linenos: