-
Notifications
You must be signed in to change notification settings - Fork 55
Make cache_dir and num_stream options session-local #809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make cache_dir and num_stream options session-local #809
Conversation
|
@MayureshV1 could you please review it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors cache directory and number of streams configuration from global (per-process) settings to session-local (per-model) settings in the OpenVINO provider. This change enables parallel creation of multiple models with different configuration options.
- Removes global
SetCacheandSetStreamsmethods from theOVCoreclass - Updates
EnableCachingandEnableStreamsmethods to accept and populate a device configuration map parameter - Shifts from setting properties globally on the core to setting them locally via device configuration
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| ov_interface.h | Removes global SetCache and SetStreams method declarations from OVCore |
| ov_interface.cc | Removes global SetCache and SetStreams method implementations from OVCore |
| basic_backend.h | Updates EnableCaching and EnableStreams method signatures to accept device config parameter |
| basic_backend.cc | Refactors EnableCaching and EnableStreams to populate device config instead of setting global properties |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implementation looks good and would make the existing provider options thread safe.
The same two provider options: cache_dir and num_streams are OV properties and can be set using load_config provider option.
|
@mklimenk As this is refactoring the properties to set cache and num streams, can it all be broght into the same function , "PopulateConfigValue" instead of seperate calls which does populate the same map.
It can also be loaded from OV config. Can that be validated as well? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consolidate seperate function calls into single PopulateConfigValue
Can be taken as a follow up as its only cosmetic |
This is a follow-up PR to #782, that moved most of the options from being set globally (per-process) to locally (per-model).
It is required to properly enable parallel session creation of multiple models with different set of options for each model.
https://jira.devtools.intel.com/browse/CVS-169874