Skip to content

Commit

Permalink
refactor!: major rewrite
Browse files Browse the repository at this point in the history 
This commit introduces a major (short of complete) rewrite of the Python
bindings for Zenoh-Flow.

The main motivation is to simplify the code base. This translates into
less Python code and more Rust code. Which means that the Python
bindings are more tightly coupled with Zenoh-Flow's code base which
leaves less room for errors and misalignment.

This rewrite is however not complete as mentioned in #30.

This rewrite also introduces few improvements:
- the bindings can now work in a Python virtual environment on macOS,
- the log emitted in a Python node are now propagated to the Zenoh-Flow
  runtime and, thus, displayed.

* 01-python.zfext: deleted and replaced with an example Zenoh-Flow
  configuration in the `examples` folder and an FAQ entry in the README.
* Cargo.toml:
  - forced resolver to version 2,
  - removed no longer used `zenoh-flow-python-commons` crate,
  - removed `[profile.dev]` and `[profile.release]` sections,
  - added a `[workspace.package]` section,
  - removed the dependencies:
    - libloading
    - log
    - zenoh-flow
  - added or updated the dependencies:
    - anyhow
    - pyo3
    - pyo3-asyncio
    - pyo3-pylogger
    - tracing
    - tracing-subscriber
    - zenoh-flow-nodes
* NOTICE.md: removed as no longer appropriate.
* README.md:
  - updated the content to reflect the new changes,
  - added a section "FAG and Troubleshooting".
* examples/flow.yaml: example data flow that leverages a Zenoh built-in
  Source and a Python sink.
* examples/sink.py: Sink node that prints the payload and the timestamp
  of the messages received by the Source.
* examples/zenoh-flow-configuration.yaml: an example configuration for
  a Zenoh-Flow standalone daemon.
* zenoh-flow-python-commons/Cargo.toml: deleted as no longer needed.
* zenoh-flow-python-commons/src/lib.rs: deleted as no longer needed.
* zenoh-flow-python-extension: deleted as no longer needed.
* zenoh-flow-python-extension-plugin: deleted as no longer needed.
* zenoh-flow-python-operator-wrapper/Cargo.toml:
  - added `[package]` section,
  - updated the `[dependencies]` section to point to the workspace,
  - removed the dependencies:
    - libloading
    - log
    - zenoh-flow-python-commons
  - added the dependencies:
    - tracing
    - tracing-subscriber
  - removed no longer needed build configuration,
  - removed `[package.metadata.deb]` section.
* zenoh-flow-python-operator-wrapper/build.rs: deleted as no longer
  needed.
* zenoh-flow-python-operator-wrapper/src/lib.rs: updated the code to the
  latest changes. In particular it is no longer needed to load the
  `libpython` shared library.
* zenoh-flow-python-sink-wrapper/Cargo.toml:
  - added `[package]` section,
  - updated the `[dependencies]` section to point to the workspace,
  - removed the dependencies:
    - libloading
    - log
    - zenoh-flow-python-commons
  - added the dependencies:
    - tracing
    - tracing-subscriber
  - removed no longer needed build configuration,
  - removed `[package.metadata.deb]` section.
* zenoh-flow-python-sink-wrapper/build.rs: deleted as no longer needed.
* zenoh-flow-python-sink-wrapper/src/lib.rs: updated the code to the
  latest changes. In particular it is no longer needed to manually load
  the `libpython` shared library.
* zenoh-flow-python-source-wrapper/Cargo.toml:
  - added `[package]` section,
  - updated the `[dependencies]` section to point to the workspace,
  - removed the dependencies:
    - libloading
    - log
    - zenoh-flow-python-commons
  - added the dependencies:
    - tracing
    - tracing-subscriber
  - removed no longer needed build configuration,
  - removed `[package.metadata.deb]` section.
* zenoh-flow-python-source-wrapper/build.rs: deleted as no longer
  needed.
* zenoh-flow-python-source-wrapper/src/lib.rs: updated the code to the
  latest changes. In particular it is no longer needed to manually load
  the `libpython` shared library.
* zenoh-flow-python/Cargo.toml:
  - added `[package]` section,
  - updated the `[dependencies]` section to point to the workspace,
  - added the dependencies:
    - anyhow
    - pyo3-asyncio
    - serde_json
    - tracing
    - zenoh-flow-nodes
  - removed the dependency to zenoh-flow-python-commons as it no longer
    (need to) exists.
  - removed the features:
    - abi-py37: it is enabled at the workspace level,
    - default: we do not want to activate the extension-module feature
      by default.
* zenoh-flow-python/README.md: deleted as it was redundant with the
  README at the root of the project.
* zenoh-flow-python/examples/loader-config.yml: deleted as no longer
  relevant.
* zenoh-flow-python/examples/operator.py: deleted (will be replaced).
* zenoh-flow-python/examples/py-operator.yml: deleted, no longer
  necessary.
* zenoh-flow-python/examples/py-pipeline.yml: deleted, replaced.
* zenoh-flow-python/examples/py-sink.yml: deleted, no longer
  necessary.
* zenoh-flow-python/examples/py-source.yml: deleted, no longer
  necessary.
* zenoh-flow-python/examples/sink.py: deleted, replaced.
* zenoh-flow-python/examples/source.py: deleted (will be replaced).
* zenoh-flow-python/pyproject.toml:
  - updated the version of maturin used,
  - updated the `[project]` section,
  - added a `[tool.maturin]` section to only enable the
    "pyo3/extension-module" feature when building the Python package of
    Zenoh-Flow,
  - updated the urls.
* zenoh-flow-python/requirements-dev.txt:
  - updated the version of maturin to reflect what is written in
    pyproject.toml,
  - removed wheel.
* zenoh-flow-python/src/lib.rs: almost complete rewrite, leveraging
  the "newtype pattern" to write the bindings.
* zenoh-flow-python/zenoh_flow/__init__.py: updated to reflect the new
  changes.
* zenoh-flow-python/zenoh_flow/interfaces/operator.py: deleted, replaced
  by `nodes.py`.
* zenoh-flow-python/zenoh_flow/interfaces/sink.py: deleted, replaced by
  `nodes.py`
* zenoh-flow-python/zenoh_flow/interfaces/source.py: deleted, replaced
  by `nodes.py`.
* zenoh-flow-python/zenoh_flow/types/__init__.py: deleted as no longer
  needed.
* zenoh-flow-python/zenoh_flow/interfaces/__init__.py -> zenoh-flow-python/zenoh_flow_python/__init__.py
* zenoh-flow-python/zenoh_flow_python/nodes.py: definition of Python
  "Protocol" that users must implement to create Zenoh-Flow nodes.
* zenoh-flow-python/zenoh_flow_python/py.typed: empty file to indicate
  that we are using a stub file to define our types.
* zenoh-flow-python/zenoh_flow_python/zenoh_flow_python.pyi: stub file
  exposing the types of our class, their methods and functions.

Signed-off-by: Julien Loudet <[email protected]>
  • Loading branch information
@J-Loudet
J-Loudet committed Jun 4, 2024
1 parent d1ec832 commit cfb4ad7
Show file tree
Hide file tree
Showing 42 changed files with 1,332 additions and 2,213 deletions.
6 changes: 0 additions & 6 deletions 01-python.zfext
View file

This file was deleted.

42 changes: 22 additions & 20 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2022 ZettaScale Technology
# Copyright © 2022 ZettaScale Technology
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
Expand All @@ -13,32 +13,34 @@
#

[workspace]

resolver = "2"
members = [
"zenoh-flow-python-operator-wrapper",
"zenoh-flow-python-sink-wrapper",
"zenoh-flow-python-source-wrapper",
"zenoh-flow-python",
"zenoh-flow-python-commons",
]

[profile.dev]
debug = true
opt-level = 0

[profile.release]
debug = false
lto = "fat"
codegen-units = 1
opt-level = 3
panic = "abort"
[workspace.package]
authors = ["ZettaScale Zenoh Team <[email protected]>"]
categories = ["network-programming"]
description = "Zenoh-Flow: a Zenoh-based data flow programming framework for computations that span from the cloud to the device."
edition = "2021"
homepage = "https://github.com/eclipse-zenoh-flow/zenoh-flow"
license = " EPL-2.0 OR Apache-2.0"
readme = "README.md"
repository = "https://github.com/eclipse-zenoh-flow/zenoh-flow-python"
version = "0.6.0-dev"

[workspace.dependencies]
async-std = { version = "=1.12.0", features = ["attributes"] }
anyhow = { version = "1" }
async-std = { version = "1.12.0", features = ["attributes"] }
async-trait = "0.1"
libloading = "0.7"
log = "0.4"
pyo3 = "0.16"
pyo3-asyncio = { version = "0.16", features = ["attributes", "async-std-runtime"] }
pyo3-build-config = { version = "0.16", features = ["resolve-config"] }
zenoh-flow = { version = "0.5.0-alpha.1" }
pyo3 = { version = "0.20", features = ["auto-initialize", "abi3-py38"] }
pyo3-asyncio = { version = "0.20", features = ["attributes", "async-std-runtime"] }
pyo3-pylogger = "0.2"
serde_json = { version = "1.0" }
tracing = "0.1"
tracing-subscriber = "0.3"
zenoh-flow-nodes = { git = "https://github.com/eclipse-zenoh-flow/zenoh-flow.git", branch = "main" }
zenoh-flow-python = { path = "./zenoh-flow-python" }
34 changes: 0 additions & 34 deletions NOTICE.md
View file

This file was deleted.

126 changes: 88 additions & 38 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,83 +6,133 @@
<img src="https://raw.githubusercontent.com/eclipse-zenoh/zenoh/master/zenoh-dragon.png" height="150">


# Python Zenoh Flow API
[Zenoh Flow](https://github.com/eclipse-zenoh/zenoh-flow) provides a Zenoh-based dataflow programming framework for computations that span from the cloud to the device.
# Zenoh-Flow Python bindings
[Zenoh-Flow](https://github.com/eclipse-zenoh-flow/zenoh-flow) provides a Zenoh-based data flow programming framework for computations that span from the cloud to the device.

:warning: **This software is still in alpha status and should _not_ be used in production. Breaking changes are likely to happen and the API is not stable.**
:warning: **This software is still in alpha status and we do not recommend using it in production.**

-----------

### Requirements
## Requirements

- Rust: see the [installation page](https://www.rust-lang.org/tools/install)
- a matching version of libpython. On linux systems, it's typically packaged separately as ``libpython3.x-dev` or `python3.x-dev`.
- Python >= 3.7
- Python >= 3.8
- pip >= 22
- virtualenv


## Installation

### Install the Python package: `zenoh_flow_python`

### How to build

Create and activate a python virtual environment:
If it's not already the case, start by activating a virtual environment:

```bash
$ python3 -m virtualenv venv
$ source venv/bin/activate
```

⚠️ On **macOS** the Zenoh-Flow wrappers have to patch the `sys.path` variable of the Python interpreter with the location of the site-packages folder of the currently active virtual environment. To do so, we rely on the `$VIRTUAL_ENV` environment variable. If your favorite environment manager does not set this variable then the `zenoh_flow_python` module will not be found when launching your flow.

Build the Python Wheel **within** a Python virtual environment.

```bash
$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ git clone https://github.com/eclipse-zenoh/zenoh-flow-python
(venv) $ git clone https://github.com/eclipse-zenoh-flow/zenoh-flow-python
(venv) $ cd zenoh-flow-python/zenoh-flow-python
(venv) $ pip3 install -r requirements-dev.txt
(venv) $ maturin build --release
(venv) $ pip install ./target/wheels/<there should only be one .whl file here>
```

**Deactivate** the venv and install the python bindings.

```bash
(venv) deactivate
$ pip3 install ./target/wheels/<there should only be one .whl file here>
```

#### Build the wrappers
#### Build the wrapper shared libraries & generate a configuration for the Zenoh-Flow runtime

Build the Python wrappers.

:warning: **Python Wrappers SHOULD NOT be built within a Python virtual environment**

```bash
$ cargo build --release -p zenoh-flow-python-operator-wrapper -p zenoh-flow-python-sink-wrapper -p zenoh-flow-python-source-wrapper
```

#### Build the docs

Once you have installed the Python binding you can also generate the documentation.
```
$ cd docs
$ pip3 install sphinx_rtd_theme sphinx -y
$ make html
```

The docs will be available under `_build/html/index.html`.
## FAQ and Troubleshooting

### Cannot load library, no extension found for files of type < py >

#### Install the Python extension for Zenoh-Flow
This error indicates that the Zenoh-Flow runtime was not properly configured to support nodes written in Python.

In order to install the Python extension, please execute the following steps:
- Copy the result of the build into `/var/zenoh-flow/python/`: `sudo cp ./target/release/libzenoh_flow_python* /var/zenoh-flow/python/`
- Update the paths in `01-python.zfext` according to your operating system (i.e., replace `.so` with `.dylib` for MacOS).
- Copy `01-python.zfext` into `/etc/zenoh-flow/extensions.d`: `sudo cp 01-python.zfext /etc/zenoh-flow/extensions.d/`
You need to change the configuration of Zenoh-Flow to let it know how to load Python scripts.

*If* you launched the Zenoh-Flow runtime in a **standalone** fashion, you need to provide a configuration that contains the following:

### Run an example
```yaml
name: my-zenoh-flow

Please refer to our getting started example: [Zenoh-Flow Getting started](https://github.com/ZettaScaleLabs/zenoh-flow-examples/tree/master/getting-started).
extensions:
- file_extension: py
libraries:
# Linux
sink: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_sink_wrapper.so
operator: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_operator_wrapper.so
source: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_source_wrapper.so
# macOS
# sink: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_sink_wrapper.dylib
# operator: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_operator_wrapper.dylib
# source: /path/to/zenoh-flow-python/target/release/libzenoh_flow_python_source_wrapper.dylib
```

*If* you launched Zenoh-Flow as a **Zenoh plugin**, you need to update the Zenoh configuration with the following:

```json
{
"plugins": {
"zenoh_flow": {
"name": "my-zenoh-flow",
"extensions": [
{
"file_extension": "py",
"libraries": {
// Linux
"operator": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_operator_wrapper.so",
"sink": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_sink_wrapper.so",
"source": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_source_wrapper.so",
// macOS
// "operator": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_operator_wrapper.dylib",
// "sink": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_sink_wrapper.dylib",
// "source": "/path/to/zenoh-flow-python/target/release/libzenoh_flow_python_source_wrapper.dylib",
}
}
]
}
}
}
```

### Failed to load `zenoh_flow_python` module

First, check that you have activated your virtual environment before launching the Zenoh-Flow runtime (be it through the standalone daemon, runtime or dedicated Zenoh plugin). **Make sure that the environment variable `$VIRTUAL_ENV` is set**.

⚠️ *If your environment manager does not set this variable, please open an issue specifying your setup*.

If your virtual environment is activated, check that the Zenoh-Flow Python package is indeed installed:
1. Open a terminal.
2. Enter the Python interpreter:
```bash
python
```
3. Try to import Zenoh-Flow Python:
```bash
import zenoh_flow_python
```

If you still have the above error when launching your data flow, try reinstalling the package:
1. Open a terminal.
2. `cd` into `zenoh-flow-python/zenoh-flow-python`.
3. Build:
```bash
maturin build --release
```
4. Install:
```bash
pip install ../target/wheels/*.whl --force-reinstall
```

Try again relaunching your data flow.
29 changes: 29 additions & 0 deletions examples/flow.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: zenoh-flow-python

vars:
BASE_DIR: "/path/to/zenoh-flow-python/examples"


sources:
- id: zenoh-source
description: "A Zenoh builtin Source"
zenoh-subscribers:
"out": "zenoh-flow-python/source"


sinks:
- id: python-sink
library: "file://{{ BASE_DIR }}/sink.py"
configuration:
test: configuration-test
inputs:
- in


links:
- from:
node: zenoh-source
output: out
to:
node: python-sink
input: in
29 changes: 29 additions & 0 deletions examples/sink.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from asyncio import *
from typing import Any, Dict
import logging
logging.getLogger().setLevel(0)

import sys
logging.debug(sys.path)

from zenoh_flow_python import *

class MySink(Sink):
input: InputRaw

def __init__(self, context: Context, configuration: Dict[str, Any], inputs: Inputs):
logging.info(configuration["test"])
self.input = inputs.take_raw("in")


async def iteration(self) -> None:
message = await self.input.recv_async()
logging.info("timestamp: {}, payload: {}".format(message.timestamp(), message.payload()))


def finalize(self) -> None:
pass


def register():
return MySink
8 changes: 8 additions & 0 deletions examples/zenoh-flow-configuration.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
name: zf-python

extensions:
- file_extension: py
libraries:
sink: /Users/julien/dev/zenoh-flow-python-2/target/debug/libzenoh_flow_python_sink_wrapper_2.dylib
source: /Users/julien/dev/zenoh-flow-python-2/target/debug/libzenoh_flow_python_sink_wrapper_2.dylib
operator: /Users/julien/dev/zenoh-flow-python-2/target/debug/libzenoh_flow_python_sink_wrapper_2.dylib
39 changes: 0 additions & 39 deletions zenoh-flow-python-commons/Cargo.toml
View file

This file was deleted.

Loading

0 comments on commit cfb4ad7

Please sign in to comment.