-
-MySense sensor kits examples -
+
+ MySense sensor kits examples
# MySense
-L=ast update of the README on 24th March 2018
+
+Last update of the README on 24th March 2018
## Description
-Software Infrastructure or framework for managing environmental sensors and data aquisition
+
+Software Infrastructure or framework for managing environmental sensors and data acquisition.
### MySense Raspberry Pi controller
-MySense is able to act as *air quality measurement kit* or *node broker*. As measurement kit MySense will collect measurements from dust, gas and/or gas sensors and location sensor and forward the data to external data concentrators (databases as well data broker as eg mosquitto and influx), files eg spreadsheets, and display (Adafruit tiny display or console). As dataconcentrator MySense will connect to other data concentrator in stead of collecting the data from sensors.
+
+MySense is able to act as an _air quality measurement kit_ or a _node broker_. As a measurement kit MySense will collect measurements from dust, gas and/or gas sensors and location sensor and
+forward the data to external data concentrators (databases as well as data broker as e.g. Mosquitto and Influx), files e.g. spreadsheets, and display (Adafruit tiny display or console). As a data concentrator MySense will connect to other data concentrators instead of collecting data from sensors.
The controller is based on Raspberry Pi for functionality and easy block building reasons.
The bus used for sensors are: USB (serial and I2C), GPIO (SPI) and I2C.
The scripts are written in Python 2.
-Visual feedback is provided with led/button (power On/Off) and optionally a tiny Oled display from Adafruit.
+Visual feedback is provided with LED/button(power On/Off) and optionally an OLED display from Adafruit.
### MySense Marvin or PyCom controller
-MySense sensor kits can also be build as *air quality satellite* sensorkits. E.g. using Marvin LoRa, LoPy or WiPy PyCom controllers with GPS, dust and meteo sensors. In this case the data will be forwarded to LoRa dataconcentrators as eg The Things Network or Mosquitto server.
-MySense in data concentrator mode has the possiblity to collect these measurements data from e.g. the TTN MQTT dataconcentrator.
-The bus used for sensors are: UART (serial), I2C and GPIO.
-The scripts are written in (embedded) micro Python. Micro python has more functionality as the language C used with Arduino boards.
+MySense sensor kits can also be built as _air quality satellite_ sensor kits e.g. using Marvin LoRa, LoPy or WiPy PyCom
+controllers with GPS, dust and meteo sensors. In this case the data will be forwarded to LoRa data concentrators like
+The Things Network or Mosquitto server.
-Visual feedback is provided with RGB led and optional a tiny Oled display from Adafruit.
+MySense in data concentrator mode has the possibility to collect these measurements data from e.g. the TTN MQTT
+data concentrator.
+The bus used for sensors are: UART(serial), I2C and GPIO.
+The scripts are written in (embedded) micro Python. Micro python has more functionality than the language C used
+on Arduino boards.
+
+Visual feedback is provided with RGB LED and an optional tiny OLED display from Adafruit.
## Goal
+
Provide a generalised dynamic Open Source based infrastructure to allow:
-* environmental measurements with sensors
-* data acquisition
-* dynamic transport of data to other data systems: e.g. databases, mosquitto, Influx,...
-* data storage and archiving
-* access for free visualisation
-* free availability of the data
-* free availability of all software (GPLV4 license)
+
+- environmental measurements with sensors
+- data acquisition
+- dynamic transport of data to other data systems: e.g. databases, Mosquitto, Influx,...
+- data storage and archiving
+- access for free visualisation
+- free availability of the data
+- free availability of all software (GPLV4 license)
## Discussion
-MySense supports calibration of every single sensor. Sensor values will differ between the sensors within a branche and between branches. Correlation software is included. Advised is to calibrate the sensors regularly for a test period of several days (conditions should vary in the test period).
-Dust measurements are done by counting the particles. The most common dust sensor is the Nova SDS011. The Plantower PMS7003 is however 1/3 in size and counts more classes of particles as well provides also the raw values. Both have a fan and laser which are powered off in idle state.
-Dust measurments are influenced by humidity. A correction algorithm to enable to compare the dust measurements with reference sensor equipment (e.g. BAM1020) is in beta test (April 2018).
+MySense supports calibration of every single sensor. Sensor values will differ between the sensors within a branch and between branches. Correlation software is included. Advised is to calibrate the sensors regularly for a test period of several days (conditions should vary in the test period).
+
+Dust measurements are done by counting the particles. The most common dust sensor is the Nova SDS011. The Plantower PMS7003 is however 1/3 in size and counts more classes of particles as well provides raw values. Both have a fan and laser which are powered off in idle state. Dust measurements are influenced by humidity. A correction algorithm to enable to compare the dust measurements with reference sensor equipment (e.g. BAM1020) is in beta test (April 2018).
-
-
+
+ MySense sensor kits examples
## How to start MySense
-* Create MySense user e.g. `ios` and login as this user.
-* Install the software on e.g. the Raspberry Pi 3 on a new user e.g. `ios` in the directory e.g. `MySense`. Use `INSTALL.sh` to install all dependencies and startup scripts.
-* Configure MySense.conf using MySense.conf.example as a lead.
-* Test one by one the input and output scripts in the python debugger as standalone e.g. `pdb MySDS011.py`. Once this is tested, go to the next step.
-* command line start and control (
+The main sensor kit case carrying the Raspberry Pi and all sensor/GPS modules is built from PVC roof gutter pieces: gutter end pieces for keeping the air in and the rain out, and overflow gutter box as casing.
+The case has a poweroff button and small window to show a display with current measurements.
The sensors are fixated on a Lego plate to allow flexibility of sensor changes.
+
+For a How-To: README.case.md
+
# Software
+
## Scripts
+
All scripts are written in Python 2. Python 3 is supported but not tested well.
Scripts have been tested on Raspberry Pi (2 and 3) running Wheezy, Jessie and Stretch Debian based OS.
-Scripts have a -h (help) option. With no arguments the script will be started in interactive mode. Arguments: *start*, *status*, *stop*.
+Scripts have a -h (help) option. With no arguments the script will be started in interactive mode. Arguments: _start_, _status_, _stop_.
### Support scripts
-* MyLed.py: control the Pi with button to power off and put it in wifi WPA mode. Pi will set up a wifi access point `MySense` if no internet connectivity could be established via wifi or LAN.
-* MyDisplayServer.py, a display service: messages received will be shown on a tiny 128X64 oled (I2C) display.
+
+- MyLed.py: Control the Pi with button to power off and put it in wifi WPA mode. Pi will set up a wifi access point `MySense` if internet connectivity could not be established via wifi or LAN.
+- MyDisplayServer.py, a display service: messages received will be shown on a tiny 128X64 oled (I2C) display.
### Main script
-The main python script is MySense.py. It acts as intermediate beween input plugins and output channels. It uses `MySense.conf` (see MySense.conf.example) to configure itself.
+
+The main python script is MySense.py. It acts as an intermediate beween input plugins and output channels. It uses `MySense.conf` (see MySense.conf.example) to configure itself.
The MySense configuration file defines all plugins available for the MySense.py command.
-* input (modules) plugins: temperature, dust, etc. sensor device modules and brokers
-* output (modules) channels: console output, (MySQL) database, (CSV/gspread) spreadsheets, and brokers (mosquitto, InFlux, ...).
+- input (modules) plugins: temperature, dust, etc. sensor device modules and brokers
+- output (modules) channels: console output, (MySQL) database, (CSV/gspread) spreadsheets, and brokers (Mosquitto, InFlux, ...).
Try `./MySense.py --help` to get an overview.
-On the command line the option --input and --output plugins can be switched on (all other configured plugins are disabled).
+On the command line the option `--input` and `--output` plugins can be switched on (all other configured plugins are disabled).
+
+#### Operation phases
+
+MySense starts with a configuring phase (options, arguments, reading configuration, loading modules), whereafter, in the `readsensors()` routine it will first access the input modules to obtain measurement values, combine them into an internal buffer cache per output channel, and finally, per output channel, try to empty the queued records.
+
+The output of sensor values to an output channel will always, on startup, send an identification json info record.
+
+Each configurable interval period, MySense will send (input) measurements values to all configured output channels. For each output channel connected via internet, MySense will keep a queue in case the connection will be broken. If the queue is exceeding memory limits the oldest records in the queue will be deleted first.
+If the configured _interval_ time is reached it will redo the previous loop.
-#### operation phases
-MySense starts with a configuring phase (options, arguments, reading configuration, loading modules), whereafter in the `readsensors()` routine it will first access the input modules to obtain measurement values, combine them into an internal buffer cache per output channel, and finaly tries per output channel to empty the queued records.
+If switched on and configured, an email with identification information will be sent to the configured user.
-The output of sensor values to an output channel will always on startup to send an identification json info record.
-Each configurable interval period of time MySense will send (input) measurements values to all configured output channels. For each output channel connected via internet MySense will keep a queue in the case the connection will be broken. If the queue is exceeding memory limits the oldest records in the queue will be deleted first.
-If the configured *interval* time is reached it will redo the previous loop.
+Make sure one obeys the Personally Identifiable Information [PII](http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-122.pdf) privacy rulings.
-If switched on and configured an email with identification information will be sent to the configured user.
-Make sure one obeys the Personally Identifiable Information ([PII]http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-122.pdf) privacy rulings.
+### Plugin configuration
-### Plugin configuration
MySense.conf is the configuration/init file from which plugin or modules are imported into the MySense process. See the `MySense.conf.example` for all plugins (sections) and the plugin options.
For every plugin module there is an README.plugin with explanations of the input/output plugin.
-The input from sensors is read asynchronous (in parallel) via the module MyTHREAD.py.
-If needed it can be switched to read only in sync with the other input sensors.
+The input from sensors is read asynchronously (in parallel) via the module MyTHREAD.py. If needed, it can be switched to read only in sync with the other input sensors.
A working example of MySense script in todays operation:
+
```
remote access | INTERNET (wired/wifi, wifi-G3/4 mobile)
syst.mgt. webmin -----||_ wifi AP -- webmin/ssh system mgt
ssh tunnel -----||_ BlueTooth -- terminal access
TeamView/Remot3 (Weaved)IoT -----|
|
- |
+ |
INPUT PLUGINs | OUTPUT CHANNELS GATEWAY/BROKER
__|__
DHT11/22-meteo ---GPIO --->| ///|\\\ |>- CSV _____
@@ -150,7 +173,7 @@ A working example of MySense script in todays operation:
O3,NO2,CO SPEC UART USB -->| | | (beta test April 2018)
NH3 - AlphaSense - I2C --->| | | (planned Jun 2018)
Adafruit rain -----GPIO -->| | | (planned Aug 2018)
- | | |
+ | | |
LoRaWan (TTN MQTT) ------->| | |>- broker? (planned)
Mosquitto sub ----server ->| | |>- LoRaWan (planned TTN)
InFlux subscribe -server ->| | |>- Bluetooth (planned)
@@ -159,17 +182,19 @@ A working example of MySense script in todays operation:
|>-raw measurement values -> InFlux server or file
calibration
```
+
-
-See for a How To: README.case.md
-
-The main sensor kit case carrying the Raspberry Pi and all sensor/GPS modules is build from PVC roof gutter pieces: gutter end pieces for keeping the air in and the rain out, and overflow gutter box as casing.
-The case has a poweroff button and small window to show a tiny display with current measurements.
+
+
MySense LoRa air quality measurement kit:
+
```
Arduino/Atom/Makr WiFi/USB --|-- WiFi / BlueTooth
___-__________
DHT11/22-meteo ---GPIO---->| / Marvin \
BME680 -meteo+gas--I2C --->|= < PyCom LoPy >|-LoRa TTN MQTT >-< MySense >
BME280 - meteo ----I2C --->| | PyCom WiPy |
- | |
+ | |
Nova SDS011 -dust -Uart -->| \ ESP /
Plantower PMS7003 -Uart -->| --------------
Grove GPS ---------Uart -->| |
@@ -178,39 +203,48 @@ MySense LoRa air quality measurement kit:
|
|> SSD1306 128X64 oled display
```
+
LoRa TTN is also used e.g. to change sample timings or to force information (e.g. location) to be send from the sensor kit.
## Configuration
+
### Configuration of MySense Pi
-See `MySense.conf.example for an example of `MySense.conf`.
-Use for configuration of plugins/outputchannels the `section` (plugin name in lowercase) and section options.
-The option `input = True or False` and `output = T/F` will define resp input plugin and output channel
-to be imported and to be switched on or off.
+See `MySense.conf.example` for an example of `MySense.conf`.
+
+Use for configuration of plugins/output channels the `section` (plugin name in lowercase) and section options.
+The option `input = True or False` and `output = T/F` will define respectively input plugin and output channel to be imported and to be switched on or off.
Input plugins as for gas and dust (particle counts) will have a configurable sample time (time to get vales) and interval time (time (interval minus sample) to wait before the next sample).
The MySense main loop has an own `interval` time within input plugin sensor values will be collected (sliding average from sample values) and push values to output channels.
### Configuration of MySense MyCom (LoPy or WiPy)
-See for an example the file `Config.py`. Make sure useXXX and the *pins* are defined and wired correctly.
+
+See for an example the file `Config.py`. Make sure use XXX and the _pins_ are defined and wired correctly.
## Interaction data format
-Interaction with plugins and output channels is done in json datastructure:
+
+Interaction with plugins and output channels is done in JSON data structure:
Example of json to display a measurement on the console (and others):
-```javascript
- { "time": UNIXtimeStamp,
+
+```json
+ {
+ "time": UNIXtimeStamp,
"temp": 23.2,
"rh": 30.2,
"pm": 234.2,
- "o3": None }
+ "o3": None
+ }
```
At the startup MySense.py will start with an identification record providing details of the version, the location if available, a unique identifier, sensor types and measurement unit, etc.
This information will define eg the first row of a spreadsheet or the database table with all sensor info (called Sensors).
Towards a broker the output will consist of an (updated e.g. GPS location) combination of the data json record and the infomration json record:
-```javascript
+
+```json
{ "ident": id-record, "data": data-record }
```
+
See for an example the file: `testdata/Output_test_data.py`
The input sensor plugins provide (sliding window of a per plug definable buffer size)) averages in a per input plugin defined interval time in seconds. The output is done on a general interval period timing using the average time of input timings.
@@ -218,46 +252,54 @@ The input sensor plugins provide (sliding window of a per plug definable buffer
Typical input rate from a sensor is 60 seconds (can be tuned) and for brokers it is 60 minute interval (can be tuned).
## Brokers
-MySense can act either *sensor manager* or as *input from broker manager* to a set (dynamic) of output channels.
+
+MySense can act either _sensor manager_ or as _input from broker manager_ to a set (dynamic) of output channels.
Available input plugins:
-* Dust: Dylos DC1100 or 1700 via serial interface, Shinyei GPIO (e.g. Grove dust sensor)
-* Temperature/humidity: Adafruit DHT11/22 or AM3202 and Grove variants
-* RSSI (strength of wifi signal): via the platform
-* Location: GPS (GPS Ultimate from Adafruit/Grove) via TTL serial interface
+
+- Dust: Dylos DC1100 or 1700 via serial interface, Shinyei GPIO (e.g. Grove dust sensor)
+- Temperature/humidity: Adafruit DHT11/22 or AM3202 and Grove variants
+- RSSI (strength of wifi signal): via the platform
+- Location: GPS (GPS Ultimate from Adafruit/Grove) via TTL serial interface
## Remote management
-The Pi allows to install a wifi connectivity with internet as well a virtual wifi Access Point. A backdoor configuration is provided via direct access to `webmin` and `ssh` (Putty), as well via a proxy as *ssh tunneling* and/or using the proxy service of Weaved (`https://www.remot3.it/web/index.html`).
+
+The Pi allows to install a wifi connectivity with internet as well a virtual wifi Access Point. A backdoor configuration is provided via direct access to `webmin` and `ssh` (Putty), as well via a proxy as _ssh tunneling_ and/or using the proxy service of Weaved (`https://www.remot3.it/web/index.html`).
If no access to Internet either via LAN or WiFi is obtained a WiFi AccessPoint is started (SSID MySense-XYZ, and default password) which enable you to use `ssh ios@192.168.2.1` command to obtain shell access. Correct in `/etc/wpa_supplicant/wpa_supplicant.conf` the SSID and psk password phrase to your local access point.
## Hardware Platform
-Sensors have a hardware interface to I2C, GPIO: those sensors are tested on RaspBerry Pi (and Arduino Uno)
+
+Sensors have a hardware interface to I2C, GPIO: those sensors are tested on Raspberry Pi (and Arduino Uno)
Sensors with USB serial are tested on Linux Debian platforms which run Python.
## Installation
+
See README.pi for installation of the Raspberry Pi platform.
MySense plugins: Use the shell file `INSTALL.sh [DHT GPS DB plugin ...]` to download all dependent modules.
-The sensor plugins, and output modules can be tested in *standalone mode*, e.g. for BME280 Bosch chip, use `python MyBME280.py`. Or use the Python debugger `pdb` in stead. See the script for the use of sync and debug options at the end of the script to test.
+The sensor plugins, and output modules can be tested in _standalone mode_, e.g. for BME280 Bosch chip, use `python MyBME280.py`. Or use the Python debugger `pdb` in stead. See the script for the use of sync and debug options at the end of the script to test.
If the Pi supports BlueTooth one could also use BlueTooth terminal access by installing BlueTooth terminal service: `./INSTALL.sh BLUETOOTH`. This type of accesss is not recommanded.
## Documentation
+
See the README's and docs directory for descriptions how to prepair the HW, python software and Pi OS for the different modules.
`CONTENT.md` will give an overview of the files and short description.
## Operation status
+
See the various README/docs directory for the plugin's and modules for the status of operation, development status, or investigation.
Failures on internet connectivity and so retries of access is provided.
## Extensive test support
+
Use the following first if one uses MySense for the first time: test each sensor input or output channel one at a time first.
Use the Conf dictionary to set configuration for the test of the module.
-The sensor plugin as well the output pugin channels *all* have a `__main__` test loop in the script.
+The sensor plugin as well the output pugin channels _all_ have a `__main__` test loop in the script.
This enables one to test each plugin (one at a time) in standalone modus: `pdb MyPLUGIN.py`.
Use for the sensor input plugins `Conf['sync']=False` (to disable multithreading) and switch debug on: `Conf['debug']=True`.
Set the python debugger `pdb` to break on `break getdata` (input plugin) or `break publish` for stepping through the script. Failures in configuration are shown in this way easily.
@@ -277,6 +319,7 @@ The focus is to allow Grove based sensors (easier to plugin to the MySense syste
The gas sensor development (NO2, O3, NH3, CO) is just (Febr 2017) started, Aug 2017 alpha tests.
## Calibration
+
Calibration of dust counters like Shinyei, Nova SDS011 and Dylos is started in May/June 2017.
Outdoor correlation tests started Sept 2017.
Indoor calibration tests with Plantower PMS7003, Nova SDS011 and BME280/BME680 were done in April 2018.
@@ -287,26 +330,34 @@ The SDS011 (and probably PMS7003) are heavily influenced by rel. humidity of 80%
Calibration of Alpha Sense gas sensors is a problematic area. Probably Sept 2017. First tests show Alpha Sense O3, CO2 are OK, NO2 not successfull, NH3 prosponed.
-To facilitate measurements for calibration purposes all sensor plugins are optionaly (set `raw` option to `True` for the particular sensor in `MySense.conf`) able to output on file or to an InFlux DB server the *raw* measurements values, as follows:
+To facilitate measurements for calibration purposes all sensor plugins are optionaly (set `raw` option to `True` for the particular sensor in `MySense.conf`) able to output on file or to an InFlux DB server the _raw_ measurements values, as follows:
+
```
raw,sensor=