Add minimal ONNX support

[e-fominov]

Overview

This is an attempt to extend the ability of ML code to use some different NN library. As I was mentioning here, we can significantly improve AI inference speed by using modern libraries.
To make it work, we need AI weights in ONNX format and this can be done by using this for example, but its original code has no implementation of region layer. So I made a PR there to add that support. I am not expecting that PR to be merged as I see that repository is not supported now.
And also implemented ML code to read and use that ONNX file.

Creating ONNX file

To make an ONNX file from Darknet weights, one can use these steps:

git clone https://github.com/e-fominov/pytorch-YOLOv4 --branch yolo2fix
cd pytorch-YOLOv4
# optional - install requirements
pip3 install -r requirements.txt
python3 model.cfg model.names model.weights sample_image.jpg 1

Here model.cfg, model.names and model.weights comes from obico-server/ml_api/model and weight file should be downloaded first. It also uses an example image as a parameter. You can use anything, even not from printer as the goal is to convert model and not to draw detections.
As a result of this run, you should get yolov4_1_3_416_416_static.onnx file inside pytorch-YOLOv4 directory. I copied that into ml_api/model, but that is not a requirement.

Running AI with ONNX file

For the most simple run, pass path of the yolo onnx file as a parameter of detection_model.py run

• use onnx: python3 lib/detection_model.py sample_image.jpg --weights model/yolov4_1_3_416_416_static.onnx
• use darknet: python3 lib/detection_model.py sample_image.jpg

Additional requirements

This uses onnxruntime which should be installed by pip3 install onxxruntime, that requirement is optional and code should continue working as-is without onnx installed by using darknet only

Example output

So I took one of printer images and run that with Darknet with with ONNX code:

Darknet:

(venv) fm@fmdesk:~/work/obico-server/ml_api$ python3 lib/detection_model.py /data/spaghetty/1.jpg 
 Try to load cfg: model/model.cfg, weights: model/model.weights, clear = 0 
layer     filters    size              input                output
   0 conv     32  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  32 0.299 BF
   1 max          2 x 2 / 2   416 x 416 x  32   ->   208 x 208 x  32 0.006 BF
   2 conv     64  3 x 3 / 1   208 x 208 x  32   ->   208 x 208 x  64 1.595 BF
   3 max          2 x 2 / 2   208 x 208 x  64   ->   104 x 104 x  64 0.003 BF
   4 conv    128  3 x 3 / 1   104 x 104 x  64   ->   104 x 104 x 128 1.595 BF
   5 conv     64  1 x 1 / 1   104 x 104 x 128   ->   104 x 104 x  64 0.177 BF
   6 conv    128  3 x 3 / 1   104 x 104 x  64   ->   104 x 104 x 128 1.595 BF
   7 max          2 x 2 / 2   104 x 104 x 128   ->    52 x  52 x 128 0.001 BF
   8 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
   9 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128 0.177 BF
  10 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
  11 max          2 x 2 / 2    52 x  52 x 256   ->    26 x  26 x 256 0.001 BF
  12 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512 1.595 BF
  13 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256 0.177 BF
  14 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512 1.595 BF
  15 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256 0.177 BF
  16 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512 1.595 BF
  17 max          2 x 2 / 2    26 x  26 x 512   ->    13 x  13 x 512 0.000 BF
  18 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  19 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512 0.177 BF
  20 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  21 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512 0.177 BF
  22 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  23 conv   1024  3 x 3 / 1    13 x  13 x1024   ->    13 x  13 x1024 3.190 BF
  24 conv   1024  3 x 3 / 1    13 x  13 x1024   ->    13 x  13 x1024 3.190 BF
  25 route  16
  26 conv     64  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x  64 0.044 BF
  27 reorg              / 2    26 x  26 x  64   ->    13 x  13 x 256
  28 route  27 24
  29 conv   1024  3 x 3 / 1    13 x  13 x1280   ->    13 x  13 x1024 3.987 BF
  30 conv     30  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x  30 0.010 BF
  31 detection
mask_scale: Using default '1.000000'
Total BFLOPS 29.338 
Loading weights from model/model.weights...
 seen 64 
Done!
Loaded - names_list: /app/model/names, classes = 1 
Execution time: 1.26 sec
[('failure', 0.6757033467292786, (535.153564453125, 474.5736389160156, 250.89207458496094, 136.91781616210938)), ('failure', 0.6715015172958374, (263.4027099609375, 544.7039794921875, 275.8241271972656, 184.3883819580078)), ('failure', 0.5903435349464417, (590.0845336914062, 481.5208740234375, 122.52083587646484, 93.05032348632812)), ('failure', 0.4780697226524353, (405.1553955078125, 396.9340515136719, 173.8264617919922, 80.89949035644531)), ('failure', 0.33798423409461975, (349.5168762207031, 382.8202209472656, 163.79119873046875, 82.55574035644531))]

Onnx:

(venv) fm@fmdesk:~/work/obico-server/ml_api$ python3 lib/detection_model.py /data/spaghetty/1.jpg --weights model/yolov4_1_3_416_416_static.onnx --preheat --cpu
Loaded - names_list: /app/model/names, classes = 1 
Execution time: 0.122 sec
[[('failure', 0.68914497, (535.3487373590469, 474.9534339904785, 252.1463994383812, 138.746013879776)), ('failure', 0.67435086, (263.38869220018387, 544.9194712638855, 276.50628340244293, 185.15077024698257)), ('failure', 0.5817742, (589.4390723705292, 481.2896337509155, 121.88858360052109, 94.02524149417877)), ('failure', 0.47977263, (405.2845135331154, 396.8387522697449, 173.75898277759552, 80.74436911940575)), ('failure', 0.33845562, (349.38735204935074, 382.78335213661194, 164.63769766688347, 82.6472404897213))]]

Here --preheat means to make an additional dry run of network befor measuring execution time and --cpu forces ONNX to use CPU execution.

This code is not tested with full server and docker containers. Not ready to be merged.

[kennethjiang]

This is exciting! Trying it now!

[kennethjiang]

I ran into this error:

ERROR: No matching distribution found for torch==1.4.0

Tried quite a few posts found online without luck.

[e-fominov]

Updates:

1. split detection_model.py into several files to make it modular. Now it tries to load darknet and onnx. If darknet is not ready (no opencv libs, etc), it will be ignored. Same for ONNX. Then it reads model file extension and uses the right backend.
2. added detect.py CLI file to make detection running. It now can dump detections into a file and also compare them with stored previous detections. Example:python3 detect.py test_data/1.jpg --weights model/model.onnx --preheat --cpu --compare-detections-with test_data/1.json
3. added tests. run: python3 -m unittest -v from ml_api. It will run detections with darknet and onnx and compare results with stored vzalues.
4. added environment variable export MODEL_NAME=model.onnx to make server.py load model from that file. This should enable ONNX model load on the server.

[kennethjiang]

@e-fominov Since release branch is for code tested on staging and pre-release, I have merged your PR to master instead.