Use MKL for verification

[AD2605]

Adds support to use oneMKL for verification and verifies example 14.

[aacostadiaz]

LGTM

[mehdi-goli]

you are building oneMKL interface for Nvidia GPU, you are linking it to the mkl close source for Intel even if it is on Nvida? what is your mechanism to have a correct linking and make sure that the oneMKL does not run on Intel CPU instead of calling cublas behind the scene to go to GPU. If you are always testing it on the IntelCPU there is no need to have this :

  -DENABLE_MKLGPU_BACKEND=${ENABLE_MKLGPU_BACKEND}
    -DENABLE_CUBLAS_BACKEND=${ENABLE_CUBLAS_BACKEND}
    ```

[AD2605]

what is your mechanism to have a correct linking and make sure that the oneMKL does not run on Intel CPU

From what I understood about the oneMKL interface, the dispatch is in accordance to the queue. Whether it is a GPU or a CPU queue, and if a GPU queue, then what backend it will select, depending on the Device.
Since our queue will never be a CPU queue, It will never go to the CPU BLAS side of things.

[mehdi-goli]

The oneMKL does not have all the backend unless you provide, for example if you dont give a correct pass to the backed there is no GPU or nvidia backend. OneMKL interface is a wrapper over 3 close source actual implementation, meaning oneMKL product, cuBLAs and rocBLAS. if the backend is not available it does not link to the correct one. the oneMKL close source product is only for intel CPU and GPU, so when you are running on Nvidia GPU you dont need the oneMKL product backend at all, similarly, when running on Intel GPU you dont need the cublas Backend at all

[AD2605]

if the backend is not available it does not link to the correct one. the oneMKL close source product is only for intel CPU and GPU, so when you are running on Nvidia GPU you dont need the oneMKL product backend at all, similarly, when running on Intel GPU you dont need the cublas Backend at all

Only one backend is being built. CPU backend is explicitly turned off, and either of the CUBLAS or the MKL_GPU (closed source oneMKL) backend is being built, here, in accordance to the SYCL target being passed.

[mehdi-goli]

what is the default queue in here, how can you make sure the queue is nvidia gpu queue. This test that you are running is specifically set for Amper device, and needed to be compiled only for that device due to definition of the architecture at compile time. how do you make sure that you get a correct queue in all systems when you use default here

[AD2605]

This test that you are running is specifically set for Amper device, and needed to be compiled only for that device due to definition of the architecture at compile time.

This example only builds if SYCL_NVIDIA_TARGET is defined (here).

how can you make sure the queue is nvidia gpu queue

Since we do not explicitly create a queue, or pass any sort of device selection of to syclcompat, I would expect that the device selector is being set when calling the binary assuming multiple backends on the platform.

The reason I use the default_queue from sycl compat is because that's being used throughout, for memory allocations etc...

[mehdi-goli]

The selector should be passed somehow probably through SYCLcompat, sycl default queue does not guaranty selection of any devices and you wont have any control when multiple GPU is available

[aacostadiaz]

You can create a queue using the GPU device selector and then use syclcompat::set_default_queue. After that, you can add a check like the one in the cuda example.

  cudaDeviceProp props;

  cudaError_t error = cudaGetDeviceProperties(&props, 0);
  if (error != cudaSuccess) {
    std::cerr << "cudaGetDeviceProperties() returned an error: " << cudaGetErrorString(error) << std::endl;
    return -1;
  }

  if (!((props.major * 10 + props.minor) >= 80)) {
    std::cerr << "Ampere Tensor Core operations must be run on a machine with compute capability at least 80."
              << std::endl;
    notSupported = true;
  }

  if (notSupported) {
    // Returning zero so this test passes on older Toolkits. Its actions are no-op.
    return 0;
  }