Merge pull request #12 from pyper-dev/dev

Add multiprocessing support

Author: pyper-dev

README.md

@@ -22,15 +22,15 @@
 
 ---
 
-Pyper is a generalized framework for concurrent data-processing, based on functional programming patterns. Used for 🌐 **Data Collection**, 🔀 **ETL systems**, and general-purpose 🛠️ **Python Scripting**
+Pyper is a comprehensive framework for concurrent and parallel data-processing, based on functional programming patterns. Used for 🌐 **Data Collection**, 🔀 **ETL Systems**, and general-purpose 🛠️ **Python Scripting**
 
 See the [Documentation](https://pyper-dev.github.io/pyper/)
 
 Key features:
 
 * 💡**Intuitive API**: Easy to learn, easy to think about. Implements clean abstractions to seamlessly unify threaded and asynchronous work.
 * 🚀 **Functional Paradigm**: Python functions are the building blocks of data pipelines. Let's you write clean, reusable code naturally.
-* 🛡️ **Safety**: Hides the heavy lifting of underlying task creation and execution. No more worrying about race conditions, memory leaks, and thread-level error handling.
+* 🛡️ **Safety**: Hides the heavy lifting of underlying task execution and resource clean-up. No more worrying about race conditions, memory leaks, or thread-level error handling.
 * ⚡ **Efficiency**: Designed from the ground up for lazy execution, using queues, workers, and generators.
 * ✨ **Pure Python**: Lightweight, with zero sub-dependencies.
 
@@ -284,9 +284,11 @@ To explore more of Pyper's features, see some further [examples](https://pyper-d
 
 ## Dependencies
 
-Pyper is implemented in pure Python, with no sub-dependencies. It relies heavily on the well-established built-in modules:
-* [asyncio](https://docs.python.org/3/library/asyncio.html) for handling async-based concurrency
-* [threading](https://docs.python.org/3/library/threading.html) for handling thread-based concurrency
+Pyper is implemented in pure Python, with no sub-dependencies. It is built on top of the well-established built-in Python modules:
+* [threading](https://docs.python.org/3/library/threading.html) for thread-based concurrency
+* [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) for parallelism
+* [asyncio](https://docs.python.org/3/library/asyncio.html) for async-based concurrency
+* [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) for unifying threads, processes, and async code
 
 ## License
 

docs/src/assets/img/diagram1.png

@@ -0,0 +1,14 @@
+---
+title: AsyncPipeline
+parent: API Reference
+layout: default
+nav_order: 3
+permalink: /docs/ApiReference/AsyncPipeline
+---
+
+# pyper.AsyncPipeline
+{: .no_toc }
+
+`AsyncPipeline` is a sublass of [Pipeline](Pipeline) and exposes the same API.
+
+[Example](../UserGuide/CreatingPipelines#asynchronous-code)

docs/src/assets/img/diagram2.png

@@ -0,0 +1,71 @@
+---
+title: Pipeline
+parent: API Reference
+layout: default
+nav_order: 2
+permalink: /docs/ApiReference/Pipeline
+---
+
+# pyper.Pipeline
+{: .no_toc }
+
+* TOC
+{:toc}
+
+## Pipeline
+
+```python
+def __new__(cls, tasks: List[Task]) -> Pipeline:
+```
+
+An object that represents a data flow consisting of a series of (at least one) tasks.
+
+{: .warning}
+It is not recommended to instantiate a `Pipeline` directly. Use the [task](task) class
+
+## Pipeline.\__call__
+
+```python
+def __call__(self, *args, **kwargs) -> Generator[Any, None, None]:
+```
+
+A `Pipeline` is a callable object with the parameter specification of its first task which generates each output from its last task.
+
+[Example](../UserGuide/CreatingPipelines#pipeline-usage)
+
+## Pipeline.pipe
+
+```python
+def pipe(self, other: Pipeline) -> Pipeline:
+```
+
+Allows two `Pipeline` objects to be composed together, returning a new pipeline with a combined list of tasks.
+
+[Example](../UserGuide/ComposingPipelines#piping-and-the--operator)
+
+## Pipeline.\__or__
+
+```python
+def __or__(self, other: Pipeline) -> Pipeline:
+```
+
+Allows the use of the operator `|` as syntactic sugar for `Pipeline.pipe`.
+
+## Pipeline.consume
+
+```python
+def consume(self, other: Callable) -> Callable:
+```
+
+Allows a consumer function to be attached to a `Pipeline`.
+
+[Example](../UserGuide/ComposingPipelines#consumer-functions-and-the--operator)
+
+
+## Pipeline.\__gt__
+
+```python
+def __gt__(self, other: Callable) -> Callable:
+```
+
+Allows the use of the operator `>` as syntactic sugar for `Pipeline.consume`.

docs/src/docs/ApiReference/AsyncPipeline.md

@@ -4,8 +4,4 @@ nav_order: 4
 layout: page
 ---
 
-# API Reference
-
-All pipelines in Pyper are created with the `pyper.task` decorator.
-
-See [Task Parameters](../UserGuide/TaskParameters)
+# API Reference