can I use cv::Mat on the AI deck?

[noahzn]

I need to implement something based on the output of the CNN. I might have two options:

1. implement it in the network as post-processing.
2. implement it on the AI deck.
   But regarding (1) I'm not sure if I can convert my PyTorch model to the ONNX model successfully with the post-processing. And regarding (2) I'm wondering if the OpenCV is supported already on the AI deck. Thanks.

[hmllr]

what exactly is the difference between 1 and 2? Is your network running offboard? Otherwise I don't see how 1 would not run on the AI-deck?

I am not 100% up to date with the possibilities of gapflow/nntool, but I don't think it supports porting post-processing. I think the easiest would be to write the post-processing directly on the gap8 yourself. For this, no, OpenCV is not supported on gap8. Some of the DSP functions in the autotiler might come in handy (https://github.com/GreenWaves-Technologies/gap_sdk/tree/8db7b5b8c83fb604048b95768f68b69e6510506d/tools/autotiler_v3/DSP_Libraries) though.

Hope this helps;
Hanna

[noahzn]

Hi Hanna, in 1) I can implement with Numy and PyTorch functions, and the network directly outputs an angular velocity. The network should be running on the AI deck in real-time, and it means it can directly send a yaw rate to the drone. But in 2, I take the output of the CNN, and process it in C++ code on the AI deck. I already have the post-processing code in C++ but it currently uses cv::Mat

[noahzn]

"but I don't think it supports porting post-processing". I'm also wondering about this. Thank you for your reply!

[hmllr]

Let me rephrase this, I am actually 99% sure it does not support porting post-processing - as post-processing is very individual it is hard to fit it into some generic way which then generates gap8 compatible code :)
If you need some inspiration on how to optimize DSP functions for the RISC-V cores in gap8, https://github.com/pulp-platform/pulp-dsp here you find a library for a very similar chip. Might be an overkill though, if your code doesn't have to be highly optimized.

[noahzn]

Thank you! I will update you once I find a solution.