Unable to interpret prediction values by Ultralytics Yolov8 tflite edgetpu model inference on Coral USB Accelerator #11
Comments
|
How many classes does your model have? |
Arslan-Mehmood1
changed the title
|
it has 247 classes. label.txt |
|
Have you tried the example code in this repository? Please try the
detect.py script here and pass in your model + label file as arguments -
you can use -h to see help.
There are several steps you need to perform to convert the output but
generally the format is:
<bounding box x4> <per class probability>
You get one prediction vector per anchor point. You need to first rescale
your result from the quantized values (int > float) and then perform
softmax and non-maximum suppression.
…On Fri, 25 Aug 2023 at 14:15, Arslan Mehmood ***@***.***> wrote:
I'm using PyCoral api default procedure.
—
Reply to this email directly, view it on GitHub
<#11 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJ4U5P4IVDMLNX6SRWLXXCQP7ANCNFSM6AAAAAA36PZ52E>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Also I havent followed anything in your repo. I have followed the Pycoral basic detection workflow and used the 'detect.py'. One thing I dont get is that the output tensor length is 33600. How will it get mapped to [coordinates x1,x2,x3,x4,probability] (5-values). |
|
Each row in the tensor is an object prediction. The first four valued are xy coordinates representing the box. The remaining values are class probabilities. Almost all of them should be zero and you need to run NMS to filter the ones that work. 4 + 247 = 251 columns Please try using detect.py - it does all the post processing. Though yolov8 is anchor free I believe. If you're not sure what the output tensor should be then you should post an issue on ultralytics' repo |
|
Also see ultralytics/ultralytics#751 It's possible the order of the values is changed in v8 but I doubt it. |
|
Also just a note that the detect.py from the coral team assumes you're
using an SSD model (eg the examples from Google), so I'm not sure it will
work unless the output shapes/interpretations are comparable.
…On Fri, 25 Aug 2023 at 14:28, Arslan Mehmood ***@***.***> wrote:
Also I havent followed anything in your repo. I have followed the Pycoral
basic detection workflow and used the detect.py. One thing I dont get is
that the output tensor length is 33600. How will it get mapped to coordinates
x1,x2,x3,x4,probability <http://5-values>.
—
Reply to this email directly, view it on GitHub
<#11 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJ2LI4ACYI6NHTYLTX3XXCSAXANCNFSM6AAAAAA36PZ52E>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
understood. |
|
Getting this error when trying this repo detect.py for my model and image. python3 detect.py --model yolov8_edgetpu.tflite --names label.yaml --image image.jpg ERROR: |
|
Hmm, I'm not sure why your model is expecting a float input, it should be
looking for int8.
…On Fri, 25 Aug 2023 at 16:42, Arslan Mehmood ***@***.***> wrote:
Getting this error when trying this repo detect.py for my model and image.
*ERROR:*
INFO:EdgeTPUModel:Confidence threshold: 0.25
INFO:EdgeTPUModel:IOU threshold: 0.45
INFO:EdgeTPUModel:Loaded 247 classes
INFO:EdgeTPUModel:Successfully loaded
/home/arslan//learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/yolov8_edgetpu.tflite
Traceback (most recent call last):
File "detect.py", line 51, in
model.forward(x)
File
"/home/arslan/nixense_vixion/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/edgetpumodel.py",
line 160, in forward
self.interpreter.set_tensor(self.input_details[0]['index'], x)
File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line
572, in set_tensor
self._interpreter.SetTensor(tensor_index, value)
ValueError: Cannot set tensor: Got value of type UINT8 but expected type
FLOAT32 for input 0, name: inputs_0
—
Reply to this email directly, view it on GitHub
<#11 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJ7Z5AEMLO5ATMNZAVLXXC2UFANCNFSM6AAAAAA36PZ52E>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Is your model actually quantised?
On Fri, 25 Aug 2023 at 17:02, Josh Veitch-Michaelis <
***@***.***> wrote:
… Hmm, I'm not sure why your model is expecting a float input, it should be
looking for int8.
On Fri, 25 Aug 2023 at 16:42, Arslan Mehmood ***@***.***>
wrote:
> Getting this error when trying this repo detect.py for my model and image.
>
> *ERROR:*
> INFO:EdgeTPUModel:Confidence threshold: 0.25
> INFO:EdgeTPUModel:IOU threshold: 0.45
> INFO:EdgeTPUModel:Loaded 247 classes
> INFO:EdgeTPUModel:Successfully loaded
> /home/arslan//learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/yolov8_edgetpu.tflite
> Traceback (most recent call last):
> File "detect.py", line 51, in
> model.forward(x)
> File
> "/home/arslan/nixense_vixion/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/edgetpumodel.py",
> line 160, in forward
> self.interpreter.set_tensor(self.input_details[0]['index'], x)
> File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line
> 572, in set_tensor
> self._interpreter.SetTensor(tensor_index, value)
> ValueError: Cannot set tensor: Got value of type UINT8 but expected type
> FLOAT32 for input 0, name: inputs_0
>
> —
> Reply to this email directly, view it on GitHub
> <#11 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAYDMJ7Z5AEMLO5ATMNZAVLXXC2UFANCNFSM6AAAAAA36PZ52E>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
|
|
Yes , after converting the yolov8 model to tf saved model using ultralytics. I have quantized my tf saved model to INT8 using tf lite converter and a small calibration dataset from my training images. |
|
You also need to use the edgetpu compiler, otherwise I don't see how the model is running on the device?
https://coral.ai/docs/edgetpu/compiler/ But either way, it seems like your model is not quantised because the input layer is asking for a float tensor instead of the int8 one that we're passing in:
|
|
sorry for my confusing explainations and all. First of all I have exported my yolov8 model to edgetpu model using ultralytics as ultralytics provide this export feature. After successful export, I get all these models as shown in screenshot.
and I'm using my_yolov8_full_integer_quant_edgetpu.tflite for inference by your detect.py script in repo. I'm getting this error. python3 detect.py --model my_yolov8_full_integer_quant_edgetpu.tflite --names label.yaml --image image.jpg Note: |
|
You can try just changing the cast to np.int8 here: Line 158 in 784d9be |
|
from here I have two approaches, either I can use this edgetpu model provided by ultralytics or I can quantize the savedtf model to INT8 only using tflite converter with a calibration dataset. The above error is of 1st approach. |
|
one thing I'm not finding good is that inference is taking a lot of time and my local cpu usage gets high when I start inference which means a part of model in being executed on my local cpu, and this thing cancels the idea of edgetpu for fast inference. :( |
|
man I would be very thankful to you for helping me in this debugging and successful execution of yolov8 on edgeTPU. |
|
Specs of model: input: output: |
|
The edgetpu compiler will tell you which bits of the model are executed on CPU, but this: |
|
yes I had the idea that input size is very large and edgetpu ram is only 7-8mb. inference results: But I cant find any image with detections in my local repo dir. one thing I can do is print the detections received by post processing function. |
|
You should get a JPEG out, there's a process function which is called during |
|
Yes I've checked and understood the source code for detection. in the working dir, there is no .jpeg file. Meanwhile, I've restarted the inference and this time detections will also be printed out in terminal. |
|
If you put a print statement here: Line 130 in 784d9be Is it generating an output path? Though you're right, currently if there aren't any detections then nothing will be saved. |
|
as suspected python3 detect.py --model my_yolov8_full_integer_quant_edgetpu.tflite --names label.yaml --image image.jpg Detections :
|
|
I suggest stepping through the various models and seeing where you start to lose detections. Start with the full precision one and then check the quantised ones. |
|
alright that's one approach, but fp32 and fp16 models are useless, they won't even execute on edgeTPU, yes to check for detections getting lost, we can follow it |
|
I would suggest ignoring the TPU for now, just check that you actually get sensible predictions on CPU for all models. Presumably the fp32 models actually work? You will get a significant drop in performance moving to quantised (8-bit), especially for small objects,but the EdgeTPU compilation process shouldn't affect the results any further. You can also significantly lower the confidence threshold and try some of your training images to see if it helps. |
|
alright, I will follow this approach. got your point |
Hi man,
I have exported my yolov8 model to edge tpu model using ultralytics. I'm able to perform the inference using USB coral accelerator. But I'm not sure how to handle the predictions. I'm sharing the input and output details of my interpreter.
Model : yolov8_int8_edgetpu.tflite
Input_details:
[{'name': 'serving_default_images:0', 'index': 0, 'shape': array([ 1, 1280, 1280, 3], dtype=int32), 'shape_signature': array([ 1, 1280, 1280, 3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
Output_details:
[{'name': 'PartitionedCall:0', 'index': 517, 'shape': array([ 1, 251, 33600], dtype=int32), 'shape_signature': array([ 1, 251, 33600], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
INFERENCE_LOG:
python3 detect_image.py -m yolov8_int8_edgetpu.tflite -i image.jpg -o inference/detection.jpg
Note: The first inference is slow because it includes loading the model into Edge TPU memory.
----INFERENCE TIME----
21368.82 ms
Output Details :
[{'name': 'PartitionedCall:0', 'index': 517, 'shape': array([ 1, 251, 33600], dtype=int32), 'shape_signature': array([ 1, 251, 33600], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
Output Data Shape:
(1, 251, 33600)
Output Data :
[[[6.8984828e-03 8.4312847e-03 1.4987010e-02 ... 9.1422123e-01
9.3176806e-01 9.5353013e-01]
[8.2258508e-03 8.4541356e-03 8.1687719e-03 ... 9.7039759e-01
9.6757478e-01 9.6137410e-01]
[1.3865395e-02 1.7534753e-02 1.8626634e-02 ... 2.2020416e-01
1.8798628e-01 1.2658052e-01]
...
[8.6820194e-07 1.4569782e-06 1.3537380e-06 ... 4.9203813e-05
3.5585672e-05 3.4364115e-05]
[1.6130493e-06 1.2267126e-06 2.5768195e-06 ... 4.6514473e-05
4.1418200e-05 3.8001206e-05]
[4.9328037e-07 1.4711668e-06 1.1877161e-06 ... 3.2545999e-05
3.1352149e-05 2.2766470e-05]]]
CONFUSION:
I'm not sure how to handle the output tensor which has shape [ 1, 251, 33600]. Please guide I'm stuck here.
The text was updated successfully, but these errors were encountered: