Best way to convert an Image buffer Uint8List to cv.Mat

[gau-nernst]

Typically we can access raw image data from other library as an Uint8List object. Assuming we know the format of the image (e.g. RGB uint8), what is the best (fast and convenient) way to convert it to OpenCV Mat? Some of the approaches I am thinking of:

• Create an empty Mat with the required size. Iterate over Uint8List and copy data over. For some formats like YUV_420sp, it might be a bit tricky to copy the data correctly.
• Encode in a lossless format (e.g. PNG) and use cv.imdecode() -> seems to be the most convenient, but there is overhead in compression-decompression.
• Allocate C memory with Dart FFI, copy data over, create Mat from C pointer. Need to release the memory after use.

[rainyl]

UPDATE 2024.8.30:

Since v1.2.0, the recommended ways are Mat.fromVec and Mat.fromList

Original:

If I'm not mistaken, the raw image data contains image header infos, just like the data read from file directly, if so, I'm afraid the only way is cv.imdecode(), Mat object in c++ opencv need to be constructed by constructors or methods like cv::imread, which is different in python (on python level, all Mat object are actually numpy.ndarray, so can be passed to opencv functions directly, but even so, it most likely a new Mat object will be created in native codes, I didn't dive into it, so just "most likely").

But if the raw image data are raw pixel values, it is possible to construct from values, for now the construction from UInt8List is not implemented, but the following procedure is ok:

// 2D  or 3D raw pix values
final data = ...;
final int rows = ...;
final int cols = ...;
final m = cv.Mat.create(rows, cols, type);
for (int i = 0; i < rows; i++){
  for (int j = 0; j < rows; j++){
    m.setValue<int>(i, j, data[i][j]);
  }
}

So, I think the best way is cv.imdecode(), please point it out if I'm wrong.

[rainyl]

I think the main problem is how to pass "ownership" of Uint8List to OpenCV, make sure memory can be garbage-collected

Did you mean that pass a dart UInt8List object to C/C++ directly without copying data? If so, I think it is neither easy nor necessary, if pass a UInt8List to c++, the list needs to be immutable, dart provides ImmutableList but it seems copying data is also unavoidable, so why not just copy to a native uint8 array, and unless your business really/extremely sensitive to performance, it is not necessary nowadays...

I have implemented mat construction from a list (Mat.fromList()), with copying to a native array, but I am sorry that I have no more time to do the above works, if you really need it, you can work on it and contributions are welcome!

[gau-nernst]

No problem at all. Indeed copy is unavoidable. Right now I have best experience with the following code

cv.Mat imageCv = cv.Mat.create(...)  // set correct width, height, and number of channels
imageCv.data.setAll(0, buffer)  // where buffer is Uint8List

[rainyl]

Okay.

BTW, I am confused why you set data with buffer, as far as I know, data is a getter of Mat and will get data from native pointer directly, so change the elements inside it won't change the data of mat.

[gau-nernst]

It works for me. You can try on your own to see. cv.Mat.data is Uint8List in Dart, which allows me to use .setAll() (or .setRange()) to copy the data to OpenCv Mat directly. This is significantly faster than using cv.Mat.setValue(), probably because .setValue() is an FFI call which has a lot of overhead.

[rainyl]

okay. I read the doc of astypedlist and noticed that it will return a reference, so it makes sense.😂