Error while processing the prediction output

[bartbutenaers]

Dear,

I am trying to integrate the code from the Coral codelab into the Node-RED application, to allow our community to detect objects fast in images (e.g. persons in IP cam images for video surveillance). Our current solution can already draw rectangles around detected objects (via tfjs), but I would like to add Coral support.

First I load the a pre-trained TfLite model for object detection (which I download from this site):

var modelUrl = "https://raw.githubusercontent.com/google-coral/test_data/master/tf2_ssd_mobilenet_v2_coco17_ptq_edgetpu.tflite";
const options = {delegates: [new CoralDelegate()]};
var tfLiteModel = await tflite.loadTFLiteModel(modelUrl, options);

Once the model is loaded, the input of my code is an image (as a NodeJs buffer):

var tf = require('@tensorflow/tfjs-node');
var img = tf.node.decodeImage(new Uint8Array(imgBuffer));
const expanded = tf.expandDims(img, 0);
                
const prediction = tfLiteModel.predict(expanded);
const percentage = tf.div(tf.mul(prediction, tf.scalar(100)), tf.scalar(255));
const data = percentage.dataSync();

However the tf.mul throws following exception:

Error: Argument 'a' passed to 'mul' must be a Tensor or TensorLike, but got 'Object'
    at convertToTensor (/home/pi/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:5433:15)
    at mul_ (/home/pi/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:11026:14)
    at Object.mul__op [as mul] (/home/pi/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:5485:29)
    ...

The prediction object looks like this:

Noob question: does this mean perhaps that it has recognized 4 different classes of objects in my image, and that I need to loop these 4 tensors (simply via Object.keys(...))? Which I then must map to a label, and somehow extract the coordinates of the bounding box?

It would also be nice if somebody could explain (in simple words ;-) ) why the expandDims(img, 0) is required, and what the tf.div(tf.mul(prediction, tf.scalar(100)), tf.scalar(255)) is doing.

Thanks!!!
Bart Butenaers

[pyu10055]

@bartbutenaers I will try to answer your questions:

1. expandDims is to transform 3D image into 4D tensor that the model requires, check out its api here.
2. the tf.div(tf.mul(prediction, tf.scalar(100)), tf.scalar(255)) is typically used to normalize the input images from [0, 255] to [0,1]. You should probably use it for input image const normalizedImg = tf.div(tf.mul(img, tf.scalar(100)), tf.scalar(255))
3. It is very likely your model output is a map of tensors, you should print them out to see the structure.

[bartbutenaers]

Hi @pyu10055,
Thanks for joining!!!

2. .... is typically used to normalize the input images from

Ah ok, I wasn't aware of that since the prediction doesn't throw an exception when I pass not-normalized images to it. But I assume in that case it will be slower and less accurate? I will implement it.

In this codelab code snippet, it is used to normalize (for some reason) the prediction result. Is that also required?

3. It is very likely your model output is a map of tensors, you should print them out to see the structure.

Yes you are correct. In my prediction output I get 4 tensors:

I'm afraid I am going to ask a noob question now :-(
I thought that the output would contain scores, classes and bboxes for detected objects (like with tfjs models).
But seems I need to extract that kind of information somehow from these tensors? If you have any link to some page with more information about how to accomplish that, it would be very welcome!

[bartbutenaers]

I need to extract that kind of information somehow from these tensors?

I assume these 4 tensors represent this:

• 00 = Locations
• 01 = Classes (index in the labels file)
• 02 = Scores
• 03 = Number of detections

What I for example don't understand how to get the bounding box dimensions array [top, left, bottom, right] from this output...

[bartbutenaers]

don't understand how to get the bounding box dimensions array

Using the following code snippet, I can extract some useful data from the output:

var bboxes = prediction["StatefulPartitionedCall:3;StatefulPartitionedCall:2;StatefulPartitionedCall:1;StatefulPartitionedCall:0"].arraySync();
var classes = prediction["StatefulPartitionedCall:3;StatefulPartitionedCall:2;StatefulPartitionedCall:1;StatefulPartitionedCall:01"].arraySync()[0];
var scores = prediction["StatefulPartitionedCall:3;StatefulPartitionedCall:2;StatefulPartitionedCall:1;StatefulPartitionedCall:02"].arraySync()[0];
var objectCount = prediction["StatefulPartitionedCall:3;StatefulPartitionedCall:2;StatefulPartitionedCall:1;StatefulPartitionedCall:03"].arraySync()[0];

At first sight this seems to be working fine, but it must be my most ugliest code snippet ever.
If anybody knows a nicer way to do this, please share it!
I tried this tip, but in my case the model.outputNodes is undefined. I don't have a clue why...

[bartbutenaers]

@pyu10055,

Now that I can do some basic detection, I wanted to prepare my input tensor based on your feedback above:

// Convert the float image to uint8, because tensor parameters for Coral TPU need to be quantized (8-bit fixed-point numbers like int8 or uint8)
var uint8ImageBuffer = new Uint8Array(imageBuffer);
            
// Decode the image and convert it to a 3D or 4D tensor (based on the encoded bytes). Supports BMP, GIF, JPEG and PNG.
var imageTensor = tf.node.decodeImage(uint8ImageBuffer);
            
// Normalize the input images from values in the range [0, 255] to [0,1]. 
// One of the reasons is to reduce the computation time to convergence.
var normalizedImageTensor = tf.div(tf.mul(imageTensor, tf.scalar(100)), tf.scalar(255));

// Transform the 3D image into a 4D tensor that the TfLite model requires
var expandedImageTensor = tf.expandDims(normalizedImageTensor, 0);
                
 // Run the object detection on the Coral TPU
var prediction = node.model.predict(expandedImageTensor);

However then the predict function throws another error:

Data type mismatch: input tensor expects 'uint8', got 'float32'

My variables have these data types:

• uint8ImageBuffer -> Uint8Array
• imageTensor -> int32
• normalizedImageTensor -> float32
• expandedImageTensor -> float32

Since the model expects uint8, I assume I am not allowed to normalize my image because otherwise I can only pass 0 or 1?
Or perhaps this indicates that I somehow have loaded the model incorrectly, and it should work with float32?

Thanks!! My brain started melting at the moment...

[bartbutenaers]

BTW I realize that this Github issue might not be the best place to ask for help. But I had understood from @mattsoulanille
that the tfjs-tflite-node is quite new, and I indeed don't find many examples yet that use it. And since I started experimenting with this code based on the codelab examples, I decided to register my issue(s) here. But if I should move my question(s) to somewhere else, don't hesitate to let me know!