You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After upgrading from version 2023.* to 2024.*, our inference results on CPU and GPU are not the same, in fact they are completely different. So if I use the 2024.*.* version for prediction, the result will not be the same on CPU and GPU. Not the version of OpenVINO which the model was built on counts, rather the version used in the inference. If I optimize the network with version 2023.*.*, but use version 2024.*.* in inference, the error persists. Of course in inference with version 2023.*.* the prediction works fine. I wrote a few lines of code in order to reproduce the result with a dummy network.
Step-by-step reproduction
Firstly create your network with this code snippet. I used Tensorflow 2.10.1.
The results are completely different and it is not an epsilon difference. We were curious which layer was causing the difference, so we started to pick layers from the network (working backwards from the very end) continuously. This continued until we got to the very first convolution layer, for which we ran the model optimizer and then the prediction, and there was still a huge difference between the CPU and GPU results. (We added a Flatten layer for easier understanding of the output)
importosimporttensorflowastffromkeras.layersimportFlattenfromkeras.modelsimportModelprint(tf.version.VERSION)
model=tf.keras.models.load_model('vino_bug_2024_inference/dummy_model', compile=False)
model.summary()
input_model=Model(inputs=model.input, outputs=model.get_layer("conv2d_8").output) #the name of my first conv layerinput_model_output=input_model.output# Add a Flatten layerflatten_output=Flatten()(input_model_output)
# Create a new model with the Flatten layer as its outputinput_conv_flatten_model=Model(inputs=input_model.input, outputs=flatten_output)
input_conv_flatten_model.summary()
defmyprint(s):
withopen('vino_bug_2024_inference/input_conv_flatten_model_summary.txt','a') asf:
print(s, file=f)
input_conv_flatten_model.summary(print_fn=myprint)
input_conv_flatten_model.save("vino_bug_2024_inference/input_conv_flatten_model")
So in summary, it came out that a single convolution layer on CPU and GPU gives completely different results. In the example I gave a classification problem, but the difference between CPU and GPU results is even more noticeable when talking about e.g. detection.
We tested what happens if we use 2023.0.0 or 2023.3.0 for inference prediction. In this case we looked at the difference between CPU and GPU and found only epsilon difference.
Relevant log output
No response
Issue submission checklist
I'm reporting an issue. It's not a question.
I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
There is reproducer code and related data files such as images, videos, models, etc.
The text was updated successfully, but these errors were encountered:
OpenVINO Version
2024.1.0
Operating System
Ubuntu 20.04 (LTS)
Device used for inference
GPU
Framework
Keras (TensorFlow 2)
Model used
Custom classifier model
Issue description
After upgrading from version 2023.* to 2024.*, our inference results on CPU and GPU are not the same, in fact they are completely different. So if I use the 2024.*.* version for prediction, the result will not be the same on CPU and GPU. Not the version of OpenVINO which the model was built on counts, rather the version used in the inference. If I optimize the network with version 2023.*.*, but use version 2024.*.* in inference, the error persists. Of course in inference with version 2023.*.* the prediction works fine. I wrote a few lines of code in order to reproduce the result with a dummy network.
Step-by-step reproduction
Firstly create your network with this code snippet. I used Tensorflow 2.10.1.
Then I got this model: dummy_model.zip
I used your model optimizer. In this case, I used 2024.1.0 for optimization, but a network optimized with 2023.0.0 or 2023.3.0 also caused errors.
mo --saved_model_dir vino_bug_2024_inference/dummy_model/ --input_shape [?,192,192,1] --layout NHWC --output_dir vino_bug_2024_inference/vino_model --use_legacy_frontend
The vino model: vino_model.zip
I used raw_image.txt to predict on both CPU and GPU with this script. (You need to modify device at compile_model)
And got these results
The results are completely different and it is not an epsilon difference. We were curious which layer was causing the difference, so we started to pick layers from the network (working backwards from the very end) continuously. This continued until we got to the very first convolution layer, for which we ran the model optimizer and then the prediction, and there was still a huge difference between the CPU and GPU results. (We added a Flatten layer for easier understanding of the output)
Keras savedmodel: input_conv_flatten_model.zip
Vino model from this savedmodel (using the script above): vino_model_input_conv_flatten.zip
Predicting again... (You need to modify device at compile_model)
And I got there results on CPU and GPU.
So in summary, it came out that a single convolution layer on CPU and GPU gives completely different results. In the example I gave a classification problem, but the difference between CPU and GPU results is even more noticeable when talking about e.g. detection.
We tested what happens if we use 2023.0.0 or 2023.3.0 for inference prediction. In this case we looked at the difference between CPU and GPU and found only epsilon difference.
Relevant log output
No response
Issue submission checklist
The text was updated successfully, but these errors were encountered: