Poor Classification Performance on Crazyflie AI-deck

[luigifeola]

Following the steps in the Classification Demo, I was able to fine-tune the model with my custom dataset (with an higher number of images respect to the provided training data) as explained in the guide. I tested the model on my laptop using new images, and while it’s not perfectly accurate, it works most of the time and it is able to distinguish between two classes: cap and background. However, when I deploy the model on the AI-deck following the provided instructions, the network mostly detects only the background class.

Does anyone have suggestions on why this might be happening and how I can debug or resolve the issue?

[gemenerik]

A great starting point is to use nntool, as it allows you to feed inputs directly into your model and inspect what the network outputs. You can start debugging there.

Additionally I’ll give some pointers to help you troubleshoot, but the actual implementation and training fall under the user’s responsibility of course.

A good first step is to verify that the network performs well on your computer. This helps ensure that the model is properly trained and hasn’t fallen into issues like overfitting or poor generalization. For example, if your dataset is heavily skewed towards one class, such as mostly background images, the model might have learned to always predict background. This could result in the network getting stuck in a local minimum.

Another important factor to consider is quantization. By default, the GAP8 deploys a quantized version of the network (quantized in TensorFlow), which can introduce performance drops due to reduced precision. Please check whether the quantized version of the network performs well on your computer.

You should also ensure that the dataset used for training closely matches the conditions of the deployed environment. Deep learning models can be highly sensitive to environmental variations. Similarly, verify that the images fed to the network are in the expected format, including dimensions, color channels, and any preprocessing steps like normalization or scaling.

[luigifeola]

Thank you for your response.

• I will look into using nntool to inspect the network outputs and further debug the issue.
• The network performs well on my computer, it works with both Lite and quantized models. The dataset is not heavily skewed toward the cap class versus the background class, and while the accuracy isn’t very high, the model is still able to distinguish between the two classes as I mentioned in the initial comment.
• I have collected a dataset in two different environments and ensured that the deployment environment matches the conditions of the training images, including lighting and other relevant factors.

[gemenerik]

Are you using the color or grayscale camera?
Curious to hear what comes out of your nntool inspection

[luigifeola]

I am using the grayscale camera.
In the meantime this is what I get when I run tools/build/make-example examples/ai/classification clean model build image. Since there are too many rows I will attach a file with the output.
nntool.txt

[gemenerik]

I meant to suggest manually using nntool to debug the model by loading tensors into the inputs and reviewing the outputs. Check out the readme.

classification.c:68:4: warning: pointer targets in passing argument 1 of 'classificationCNN' differ in signedness [-Wpointer-sign]
   (cameraBuffer, Output_1);
    ^~~~~~~~~~~~
In file included from classification.c:32:0:
/tmp/bitcraze/aideck-gap8-examples/examples/ai/classification/BUILD_MODEL_SQ8BIT/classificationKernels.h:535:12: note: expected 'signed char * restrict' but argument is of type 'unsigned char *'

may be problematic

[luigifeola]

• Regarding the following warning:

classification.c:68:4: warning: pointer targets in passing argument 1 of 'classificationCNN' differ in signedness [-Wpointer-sign]
   (cameraBuffer, Output_1);
    ^~~~~~~~~~~~
In file included from classification.c:32:0:
/tmp/bitcraze/aideck-gap8-examples/examples/ai/classification/BUILD_MODEL_SQ8BIT/classificationKernels.h:535:12: note: expected 'signed char * restrict' but argument is of type 'unsigned char *'

What could I do to address this? As you probably know, classificationKernels.h is automatically generated in BUILD_MODEL_SQ8BIT folder.

• I attempted to debug the model using nntool, as you suggested. I took inspiration from nntool_script_q.
  There is attached output.md with the terminal output, and maybe could be also helpful the save_state output reported in classification_q.json.

• The link you provide me is really scarne in debug details, is anything more I can do to debug?

Thanks

output.md
classification_q.json

[gemenerik]

What could I do to address this? As you probably know, classificationKernels.h is automatically generated in BUILD_MODEL_SQ8BIT folder.

You can convert your cameraBuffer to the expected type.

... is anything more I can do to debug?

I would try to get the model working as expected in nntool first. So I would indeed suggesting feeding it images and seeing if you can get your expected output. You can see how it was done in the script you linked.

[luigifeola]

Which version of docker-aideck are you using?

[gemenerik]

4.22.0.1

[luigifeola]

I meant to suggest manually using nntool to debug the model by loading tensors into the inputs and reviewing the outputs. Check out the readme.

classification.c:68:4: warning: pointer targets in passing argument 1 of 'classificationCNN' differ in signedness [-Wpointer-sign]
   (cameraBuffer, Output_1);
    ^~~~~~~~~~~~
In file included from classification.c:32:0:
/tmp/bitcraze/aideck-gap8-examples/examples/ai/classification/BUILD_MODEL_SQ8BIT/classificationKernels.h:535:12: note: expected 'signed char * restrict' but argument is of type 'unsigned char *'

may be problematic

I am using the same version. Do you also get the same warning when using the nntool?

[gemenerik]

I am getting the same, try adding imageformat input_1 bw8 offset_int8 after the adjust command in your nntool_script file.

[luigifeola]

Hi @gemenerik, thank you for your help. The warning has been resolved, but now, regardless of the input, the network only outputs the custom class (in my case, the cap class). I'll continue investigating on my own, but any further assistance would be greatly appreciated.