intel gpu support via target_device parameter

[dtrawins]

No description provided.

[tanmayv25]

We should add some testing in our CI for the support target_device.
Something like this: https://github.com/triton-inference-server/server/blob/main/qa/L0_libtorch_inference_mode/test.sh

We can call it L0_openvino_target_device?

[tanmayv25]

What is the reason for removing the error checking on reading and parsing the parameters, here and everywhere?

[dtrawins]

That is not related to the GPU support but those error messages were misleading. Error suggests that something is wrong with the setup but it is perfectly fine to skip those parameters

[nnshah1]

Suggestion: Update ReadParameter to take a value indicating if a missing value is ok / provide a default value

RETURN_IF_ERROR(ReadParameter(params,"OPTIONAL_KEY",&(cpu_ext_path),default_value='foo');

That way it is a little clearer the intent.

Note: suggestion only.

[tanmayv25]

Does OpenVINO support models whose computation is spread across CPU and GPU cores?

[dtrawins]

Yes, that is possible via using a virtual target device called MULTI which is loadbalancing the request or HETERO target device which spreads a single inference.

[tanmayv25]

I believe this check needs to be updated. From what I see we support following kinds:

1. CPU
2. GPU
3. AUTO: If GPU cores are available and model supports GPU deployment, then use GPU otherwise use CPU.

[tanmayv25]

I don't think it is a good idea introducing a model level TARGET_DEVICE parameter in the model config. Triton allows you to have multiple model instances. And each instance can specify which device to use for the inference.
So they can have model A, which can specify something like below:

  instance_group [
    {
      count: 1
      kind: KIND_GPU
    },
    {
      count: 1
      kind: KIND_CPU
    }
  ]

You can rely on information from TRITONBACKEND_ModelInstanceKind API call to determine the kind of the instance.
If the kind is CPU and the model is not loaded (within model_state_), then load the model on CPU and use it.
If the kind is GPU and the model is not loaded (within model_state_), then load the model on GPU and use it.
This allows sharing of the model across the model instances.

[tanmayv25]

Even if the use-case of loading two separate kinds of model instances is not required, I would still prefer using TRITONBACKEND_ModelInstanceKind API instance of another config parameter.

[dtrawins]

In openvino, target device contains more options then just the selection between CPU and GPU. It can also set virtual device like MULTI, HETERO or AUTO or BATCH. It can also include extra options like a priority list AUTO:GPU,CPU. My understanding is that settings kind as GPU validated if CUDA GPU is present so that could not be used with other types of devices like iGPU. I couldn't find a way for using KIND as the target device without changes outside of the openvino backend code

[ryanloney]

In openvino, target device contains more options then just the selection between CPU and GPU. It can also set virtual device like MULTI, HETERO or AUTO or BATCH. It can also include extra options like a priority list AUTO:GPU,CPU. My understanding is that settings kind as GPU validated if CUDA GPU is present so that could not be used with other types of devices like iGPU. I couldn't find a way for using KIND as the target device without changes outside of the openvino backend code

@tanmayv25 do you think we can move forward? Do you need more input from @dtrawins?

[nnshah1]

@ryanloney - proposal in works - will update

[tanmayv25]

We are in middle of discussing how to best support new hardware platforms. Directing this to @nnshah1.

[tanmayv25]

I think we can update the model configuration to accommodate these fields. It would help keeping the device specification in a single place. The benefit is that in future we might get other backends targetting these devices besides current openVINO.

[dtrawins]

We should add some testing in our CI for the support target_device. Something like this: https://github.com/triton-inference-server/server/blob/main/qa/L0_libtorch_inference_mode/test.sh

We can call it L0_openvino_target_device?

Functional tests are under preparation in another PR. They should be easy to integrate in CI.

[nnshah1]

pending discussion