Correctness checking for security_barrier_camera_demo w/ 1 network multi channels with inputting 1 image #3392
Conversation
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…_demo. Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…_damo. Signed-off-by: Wang, Yang <[email protected]>
…ty_barrier_camera_dem. Signed-off-by: Wang, Yang <[email protected]>
…video. 2 Update case parameters in order to capturing ROI data from this demo. 3 Enable parse function for AUTO device. Signed-off-by: Wang, Yang <[email protected]>
…ted. Signed-off-by: Wang, Yang <[email protected]>
…nputing single iamge for each channel. Signed-off-by: Wang, Yang <[email protected]>
…UTO_plugin Signed-off-by: Wang, Yang <[email protected]>
…era_demo. Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
|
there are some basic pre-commit checks failed: 06:15:26 + ./venv/bin/python ./src/ci/check-basics.py |
|
I'm generally fine with the changes, yet the correctness checking seems to happen on the OMZ side? how do we get the results of such correctness, is there a weekly/nightly etc report or what? also the question to @vladimir-dudnik |
|
@myshevts the idea is to extend OMZ tests with inference results correctness check (currently we do not have this). OMZ tests currently used in automatic public runs, but there is no any reports on this. |
There was a problem hiding this comment.
I would suggest to refer my branch with scratch validation scenario and simplify the implementation of parser() for multi channel correctness checking.
Since in parser() the results are parsed from log file, these results obtained from runtime should be saved in the parser while run_tests.py running.
The main loop in run_tests.py consists of 3 loops: iteration over demos -> iteration over test cases -> iteration over devices. While the third loop, the output log might be saved in the parser directly (just need to fill self.results[device][test_case]).
Finally, after all combinations of devices + test cases are ended, checker() might be called for the demo having parsed results.
And if parser() with checker() are transformed into some class MultiChannelCorrectnessChecker, the list of demos will be (it's aligned to the implementation of performance_scope, validation_scope):
DEMOS = [
deepcopy(BASE['security_barrier_camera_demo/cpp'])
.update_option({'-r': None})
.add_parser(MultiChannelCorrectnessChecker)
]
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
| @@ -42,7 +42,7 @@ | |||
|
|
|||
| log.basicConfig(format='[ %(levelname)s ] %(message)s', level=log.DEBUG, stream=sys.stdout) | |||
|
|
|||
| DEVICE_KINDS = ['CPU', 'GPU', 'MYRIAD', 'HETERO', 'HDDL'] | |||
| DEVICE_KINDS = ['CPU', 'GPU', 'AUTO', 'AUTO:CPU,GPU', 'MULTI:CPU,GPU', 'MYRIAD', 'HETERO', 'HDDL'] | |||
There was a problem hiding this comment.
This should not be required for security_barrier_camera_demo ?
Signed-off-by: Wang, Yang <[email protected]>
|
there are still some precommit checks failed 09:06:39 running documentation checks... |
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…inputting video. 2. Update the correctness checker that allows insignificant difference for the ROI. Signed-off-by: Wang, Yang <[email protected]>
…urity_barrier_camera_demo. Signed-off-by: Wang, Yang <[email protected]>
…2. Update the raw data format for the demo. Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…ty_barrier_camera demo. Signed-off-by: Wang, Yang <[email protected]>
… to avoid changing. Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…rred frames. 2. update correctness checking logic that could validate the correctness the multi results within 1 frame individually. Signed-off-by: Wang, Yang <[email protected]>
…alue assign logic for the output port name of vehicle attribution. Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…l_zoo into ywang2/analysis_result_automatically_for_AUTO_plugin
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
Signed-off-by: Wang, Yang <[email protected]>
…eout_occurs' into ywang2/analysis_result_automatically_for_AUTO_plugin
Signed-off-by: Wang, Yang <[email protected]>
…eout_occurs' into ywang2/analysis_result_automatically_for_AUTO_plugin
Correctness checking for demo security_barrier_camera_demo with single image inputting for each channel.
JIRA:
https://jira.devtools.intel.com/browse/CVS-74969