Efficient RGB Image to Tensor in dotnet

[skywalkerisnull]

Hello everyone,

We are trying to find a more efficient method of converting the RGB values from an image, into a tensor. The current process that we are using is modeled on the sample code: https://github.com/microsoft/onnxruntime-csharp-cv-template/blob/af9061afe3077b2b7b137bf2d280541ecc316c4d/utils/ImageHelper.cs#L29

or more specifically, our variation:

for (var j = 0; j < numberBatches; j++) {
  inputDimension[0] = batchSizes[j];
  Tensor<float> input = new DenseTensor<float>(inputDimension);

  for (var i = 0; i < batchSizes[j]; i++) {
    var image = tileList[tileIndex].Image;
    image.ProcessPixelRows(pixelAccessor => {

      for (var y = 0; y < image.Height; y++) {
        var rowSpan = pixelAccessor.GetRowSpan(y);

        for (int x = 0; x < image.Width; x++) {
          input[i, 0, y, x] = rowSpan[x].R;
          input[i, 1, y, x] = rowSpan[x].G;
          input[i, 2, y, x] = rowSpan[x].B;
        }
      }
    });
    tileList[tileIndex].Image = null;
    tileIndex++;
  }
  var tempList = new List<NamedOnnxValue>(){
      NamedOnnxValue.CreateFromTensor($"image", input)};
  batchList.Add(tempList);
}

We are building batches that may be up to 500 images and can run the inference in less than 200ms for this batch on a Nvidia V100 (non optimized model). However, creating the batch is more than 11 seconds per batch. i.e. we would need approx. 55 threads creating the batches to keep up with the GPU throughput.

I found a discussion in the ImageSharp discussions that I jumped onto and were discussing this: SixLabors/ImageSharp#1955

What methods have other people used, or even other libraries that may be able to move the RGB to the tensor batch in an efficient way?

[vpenades]

In my case, I am doing something like that, but my code needs to run in realtime because the image comes from a camera. Furthermore, I need to crop a rotated region of the source image, which makes things even more complicated.

@skywalkerisnull I am using OpenCVSharp just for the image processing, which is a bit overkill to use such a huge library just for cropping and rotating.

Given that a large amount of users are using OnnxRuntime for artificial vision, it would be nice to have some helper classes to deal with images.

[skywalkerisnull]

@vpenades Have a look at the model I created for this to make it faster: https://github.com/skywalkerisnull/onnxruntime-csharp-cv-template/blob/82211fbd6cabb5b82ea60647a88b432214623468/utils/ImageHelper.cs#L62

This model took our batch creation from 11.2 second down to about 0.15 seconds

[andife]

@skywalkerisnull I am using OpenCVSharp just for the image processing, which is a bit overkill to use such a huge library just for cropping and rotating.

I would be quite interested in using OpenCVSharp in combination with onnxruntime. Were you successful in preparing the input tensor in e.g. NCHW format? If so would you be willing to share this?

[vpenades]

@skywalkerisnull you can make your code even more faster by casting to System.Numerics.Vector3 instead of float, because the pixel processing is done in one shot.

Anyway, I cannot use ImageSharp in my use case, because ImageSharp's image rotation is quite slow and I need processing 30 images per second in a low end intel i3 processor, with as little memory allocation as possible.

My current image processing pipeline lets me take the captured image from the device, represented as an IntPtr, and crop a rotated region directly into the tensors, without creating intermediate images.

But I think for cases like this, the ideal solution would be to have a library able to process images in the same context that is running the onnx model. I mean, if for example we're using the GPU to run the onnx model, it would be desirable to be able to upload the current image to the GPU (maybe via DirectX or Vulkan buffers), do the image processing using GPU, which is probably way faster than any trick we can find, and then fill the tensors in the GPU itself. I don't know if such a thing is being considered in the OnnxRuntine APIs

[skywalkerisnull]

We tried to cast to vector3 but it did not have any measurable impact on our solution.

You may want to look into this repo: https://github.com/microsoft/onnxruntime-extensions as the goal of that repo it to assit with common pre and post processing steps.

But if you are doing rotations, why not make them part of the graph? In our embedded stuff, we also see what we can offload to the cameras ISP, there may be a rotation option built into the camera?

[vpenades]

it's not only rotation, but rotating a cropped region which is the result of another model, not sure if that's possible.

Sadly, I work exclusively with c#, so I can't use these extensions. Hopefully, something will pop up for c#

[GeorgeS2019]

related issues to exposing API:
[C#] Update onnxruntime_dependencies diagram with contributed and custom operators API

other related issues:

• [C#] ContribOperators
• [Documentation, Windows, c#] Add an entry on Build for Custom Op development
• [C#] Documentation on how to coordinate the external Onnxruntime-Extension
• [C#] Basic build instruction to generate custom_op_library.dll
• [C#] Provide custom_op_library.dll

[GeorgeS2019]

@skywalkerisnull could U add [c#] in front of your title of discussion instead of using dotnet