Poor Classification Performance on Crazyflie AI-deck

[luigifeola]

Following the steps in the Classification Demo, I was able to fine-tune the model with my custom dataset (with an higher number of images respect to the provided training data) as explained in the guide. I tested the model on my laptop using new images, and while it’s not perfectly accurate, it works most of the time and it is able to distinguish between two classes: cap and background. However, when I deploy the model on the AI-deck following the provided instructions, the network mostly detects only the background class.

Does anyone have suggestions on why this might be happening and how I can debug or resolve the issue?

[gemenerik]

A great starting point is to use nntool, as it allows you to feed inputs directly into your model and inspect what the network outputs. You can start debugging there.

Additionally I’ll give some pointers to help you troubleshoot, but the actual implementation and training fall under the user’s responsibility of course.

A good first step is to verify that the network performs well on your computer. This helps ensure that the model is properly trained and hasn’t fallen into issues like overfitting or poor generalization. For example, if your dataset is heavily skewed towards one class, such as mostly background images, the model might have learned to always predict background. This could result in the network getting stuck in a local minimum.

Another important factor to consider is quantization. By default, the GAP8 deploys a quantized version of the network (quantized in TensorFlow), which can introduce performance drops due to reduced precision. Please check whether the quantized version of the network performs well on your computer.

You should also ensure that the dataset used for training closely matches the conditions of the deployed environment. Deep learning models can be highly sensitive to environmental variations. Similarly, verify that the images fed to the network are in the expected format, including dimensions, color channels, and any preprocessing steps like normalization or scaling.

[luigifeola]

Thank you for your response.

• I will look into using nntool to inspect the network outputs and further debug the issue.
• The network performs well on my computer, it works with both Lite and quantized models. The dataset is not heavily skewed toward the cap class versus the background class, and while the accuracy isn’t very high, the model is still able to distinguish between the two classes as I mentioned in the initial comment.
• I have collected a dataset in two different environments and ensured that the deployment environment matches the conditions of the training images, including lighting and other relevant factors.

[gemenerik]

Are you using the color or grayscale camera?
Curious to hear what comes out of your nntool inspection

[luigifeola]

I am using the grayscale camera.
In the meantime this is what I get when I run tools/build/make-example examples/ai/classification clean model build image. Since there are too many rows I will attach a file with the output.
nntool.txt

[gemenerik]

I meant to suggest manually using nntool to debug the model by loading tensors into the inputs and reviewing the outputs. Check out the readme.

classification.c:68:4: warning: pointer targets in passing argument 1 of 'classificationCNN' differ in signedness [-Wpointer-sign]
   (cameraBuffer, Output_1);
    ^~~~~~~~~~~~
In file included from classification.c:32:0:
/tmp/bitcraze/aideck-gap8-examples/examples/ai/classification/BUILD_MODEL_SQ8BIT/classificationKernels.h:535:12: note: expected 'signed char * restrict' but argument is of type 'unsigned char *'

may be problematic