Skip to content

Commit 3c1de43

Browse files
authored
Simplify Parfun imports (finos#40)
* Simplify imports and rename @Parfun in @parallel Signed-off-by: rafa-be <raphael@noisycamp.com> * Refactors Parfun API so it can be imported as `import parfun as pf`. Signed-off-by: rafa-be <raphael@noisycamp.com> * Fixes typos and wording in documentation's front page. Signed-off-by: rafa-be <raphael@noisycamp.com> * Rename the `parfun.collection` module into `parfun.py_list`. Signed-off-by: rafa-be <raphael@noisycamp.com> --------- Signed-off-by: rafa-be <raphael@noisycamp.com>
1 parent b32dcd3 commit 3c1de43

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+656
-558
lines changed

README.md

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -28,33 +28,28 @@
2828
Parfun is a lightweight library providing helpers to **make it easy to write and run a Python function in parallel
2929
and distributed systems**.
3030

31-
The main feature of the library is its `@parfun` decorator that transparently executes standard Python functions
31+
The main feature of the library is its `@parallel` decorator that transparently executes standard Python functions
3232
following the [map-reduce](https://en.wikipedia.org/wiki/MapReduce) pattern:
3333

3434

3535
```Python
3636
from typing import List
3737

38-
from parfun import parfun
39-
from parfun.combine.collection import list_concat
40-
from parfun.partition.api import per_argument
41-
from parfun.partition.collection import list_by_chunk
38+
import parfun as pf
4239

4340

44-
@parfun(
45-
split=per_argument(
46-
values=list_by_chunk
41+
@pf.parallel(
42+
split=pf.per_argument(
43+
values=pf.py_list.by_chunk
4744
),
48-
combine_with=list_concat,
45+
combine_with=pf.py_list.concat,
4946
)
5047
def list_pow(values: List[float], factor: float) -> List[float]:
5148
return [v**factor for v in values]
5249

5350

5451
if __name__ == "__main__":
55-
from parfun.entry_point import set_parallel_backend_context
56-
57-
with set_parallel_backend_context("local_multiprocessing"): # use a local pool of processes
52+
with pf.set_parallel_backend_context("local_multiprocessing"): # use a local pool of processes
5853
print(list_pow([1, 2, 3], 2)) # runs in parallel, prints [1, 4, 9]
5954
```
6055

docs/source/api.rst

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
API
2+
===
3+
4+
.. autofunction:: parfun.parallel
5+
6+
7+
Backend setup
8+
-------------
9+
10+
.. autofunction:: parfun.set_parallel_backend
11+
12+
.. autofunction:: parfun.set_parallel_backend_context
13+
14+
.. autofunction:: parfun.get_parallel_backend
15+
16+
17+
Backend instances
18+
~~~~~~~~~~~~~~~~~
19+
20+
.. automodule:: parfun.backend.mixins
21+
:members:
22+
23+
.. automodule:: parfun.backend.local_single_process
24+
:members:
25+
26+
.. automodule:: parfun.backend.local_multiprocessing
27+
:members:
28+
29+
.. automodule:: parfun.backend.dask
30+
:members:
31+
32+
.. automodule:: parfun.backend.scaler
33+
:members:
34+
35+
36+
Partitioning
37+
------------
38+
39+
.. automodule:: parfun.partition.object
40+
:members:
41+
42+
.. autofunction:: parfun.all_arguments
43+
44+
.. autofunction:: parfun.multiple_arguments
45+
46+
.. autofunction:: parfun.per_argument
47+
48+
.. autofunction:: parfun.partition.utility.with_partition_size
49+
50+
51+
.. _section-lists:
52+
53+
Python lists
54+
------------
55+
56+
.. automodule:: parfun.py_list
57+
:members:
58+
59+
60+
.. _section-dataframes:
61+
62+
Pandas dataframes
63+
-----------------
64+
65+
.. automodule:: parfun.dataframe
66+
:members:

docs/source/api/backend.rst

Lines changed: 0 additions & 17 deletions
This file was deleted.

docs/source/api/combine.rst

Lines changed: 0 additions & 18 deletions
This file was deleted.

docs/source/api/entry_point.rst

Lines changed: 0 additions & 5 deletions
This file was deleted.

docs/source/api/functions.rst

Lines changed: 0 additions & 5 deletions
This file was deleted.

docs/source/api/index.rst

Lines changed: 0 additions & 14 deletions
This file was deleted.

docs/source/api/partition.rst

Lines changed: 0 additions & 36 deletions
This file was deleted.

docs/source/conf.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,6 @@
5757
# Add any paths that contain custom static files (such as style sheets) here,
5858
# relative to this directory. They are copied after the builtin static files,
5959
# so a file named "default.css" will overwrite the builtin "default.css".
60-
html_static_path = ["_static"]
61-
html_css_files = ["style.css"]
62-
6360
# html_static_path = []
6461
# html_css_files = []
6562

docs/source/index.rst

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
Welcome to Parfun's documentation!
2-
================================================
2+
==================================
33

4-
**Parfun** is a lightweight and user-friendly Python library to assist users in running a Python function in
5-
parallel.
4+
**Parfun** is a lightweight and user-friendly Python library designed to help users run Python functions in parallel
5+
with ease.
66

7-
With limited knowledge of parallelism and distributed systems, users can significantly speedup their Python code.
7+
Even with minimal knowledge of parallelism or distributed systems, users can significantly speed up their Python code
8+
using Parfun.
89

9-
Parfun supports multiple execution backend, including Python's built-in multiprocessing, Dask and Scaler.
10+
Parfun supports multiple execution backends, including Python's built-in
11+
`multiprocessing <https://docs.python.org/3/library/multiprocessing.html>`_, as well as
12+
`Scaler <https://github.com/Citi/scaler>`_ and `Dask <https://www.dask.org/>`_.
1013

1114

1215
Content
@@ -18,7 +21,7 @@ Content
1821
tutorials/quickstart
1922
tutorials/examples
2023
tutorials/implementation_details
21-
api/index
24+
api
2225

2326

2427
Indices and tables

0 commit comments

Comments
 (0)