This module was created as part of a University project at University of Applied Sciences and Arts Northwestern Switzerland FHNW, Institute of Geomatics Engineering.
The Project was a Specialisation Project as part of a Master of Science - Geomatics degree, titled "INTERNET-OF-THINGS AD-HOC NETWORKS FOR SMART GEO-IOT APPLICATIONS".
The Python3 module is for use with Internet of Things (IoT) ad-hoc networks. The Internet of Things (IoT) comprises of networks of communicating sensors which provide data for analysis and can be used to automate processes. Ad-hoc networks are self-organizing networks of nodes which communicate with each other directly across a dynamically formed mesh.
A task often completed is the acquisition of data from multiple sensors on a wired or wireless Local Area Network (LAN). The software supports a generalised data pipeline. This pipeline is a simple data flow from multiple data sources, to a single data sink. This data flow is considered as a series of data units, or jobs, which may be processed at any point from source to sink.
An exmaple application could be to obtain temperature readings at multiple source devices withn a bulding, send the data to a processor which calculates an average temperature, which is then sent to one of the devices where it is updated in a database.
The library handles network discovery and data transfer, and allows the user to use his/her own functions for the creation and handling of data.
The library uses the Pyre and the ZeroMQ messaging library to provides the user with classes which do the following:
- create a dynamic ad-hoc network automatically, without pre-configuration
- elect a coordinator to manage creation and processing of data in a distributed fashion
- executes user-defined source, processing and sink functions
The module requires the following non-core modules:
- zeromq
- pyre
Some plugins require the following non-core modules:
- psycopg2
- paho.mqtt.client
The software should be running on each LAN node. A node object should be created first;
# Create Node for this device
node = ad_hoc_network.Node()
A named group can be created which carries out discovery of other nodes, leader election and carries out the data creation and processing:
# Create new group called "Thermometers" for this device which pretends to monitor room temperature
# and prints average to sink node terminal
node.new_group("Thermometers", FakeTemperature, AverageTemperature, plugins.PrintAsText, None, 3, is_sink)
The new_group() function takes the following parameters:
grp_name- text - The name of the groupsource_function- The user defined function (see below) used to create data at each group nodeprocess_function- The user defined function (see below) used by the processor on all received source datasink_function- The user defined function (see below) used by the sink node on the processed datasink_extra_params- keyword argument dictionary - Additional parameters which will be passed to the sink functioninterval- Integer/Float, default=1 - The interval between creation of jobsis_sink- boolean, default=False - Whether the node is to provide sink capability to the group (perhaps the sink has a connection to the internet)
Each node has been provided with a Python function explaining how to create source data, how to process data and what to do with the data which has been processed, called the Sink function.
As an example, the Source function could be a temperature measurement, the Processing function could calculate an average temperature from all the Source temperatures, and the Sink function could save the average temperature to a database.
The module is provided with a suite of pre-programmed source, processing and sink functions for typical scenarios. These 'Plugin functions' may be extended by the user, depending on the desired application.
Should accept the Job ID upon initialisation.
Should have a method execute() which returns binary data.
ShellSource- Execution of shell command
Should accept the received source data (list containing binary data) and Job ID upon initialisation.
Should have a method execute() which returns binary data.
ShellProcess- Execution of shell command
Should accept the processed data and Job ID upon initialisation.
Should have a method execute() which does something with the data.
ShellSink- Execution of shell commandPrintAsText- Print data as text to console/outputSaveLocal- Save data locallySendMQTT- Publish MQTT messagePushZMQ/PubZMQ- Send as ZMQ message (Push/Publish)SendPostgres- Update Postgres databaseSendSQLite- Update SQLite databaseSendSMTP- Send SMTP E-Mail
The provision of a suite of extendable plugins provides both convenience and flexibility for the user. The user can realise a wide range of network applications which conforms to the Source-Process-Sink model.
Multiple groups can be created, each with their own functions.
The library includes an example application example.py which generates a random temperature which is sent to a processing node. The processing node calculates an average temperature which is sent to the sink node where the average value is displayed.
The module has been tested withe the following variable combinations:
- Interval : 0.01 - 20s
- Payload : 100 Bytes – 5’000kB
- Number of Nodes
- Single host : 1-20
- WLAN : 1 – 7
The performance on a single machine was excellent. Minimum recommended intervals for different data sizes are shown on the image. On a wireless network test, data above 1kB failed to be completed. This is likely due to bottlenecking across the wireless network and the user should test the module for his/her own application.
It is intended to introduce a maximum waiting time at the processor, instead of waiting for all node data to arrive. This should help to improve the stability of the library where bandwidth is slow or nodes are lost.