Enhancing ImageSharp for Better Deep Learning Inference Processing Support

[sdcb]

Hello ImageSharp team,

I am using your wonderful library in my project, and I must commend the simplicity and flexibility provided by it. However, I foresee that there could be potential improvements which might be of great assistance to users in the Deep Learning domain.

Deep learning model inference often requires common preprocessing and postprocessing steps. Notably, operations such as Dilate, Threshold, and FindContours are frequently employed. These operators are widely known and frequently utilized in OpenCV.

The addition of these three operators would be a significant enhancement to ImageSharp library, and it would facilitate developers who are actively involved in deep learning projects:

Dilate: This operator is crucial in image processing, specifically in expanding shapes in an image.
Threshold: This operator helps in image segmentation and simplifying visual data for analysis.
FindContours: This operator is utilized to detect and draw contours in images, which comes in handy in object detection and recognition tasks.
I am aware that maintaining the balance between simplicity and complexity in a library is challenging. However, I believe that the incorporation of these operators would significantly extend its application in the field of deep learning without compromising the overall simplicity.

I am hoping to hear your thoughts on this suggestion. Thank you for your time and consideration.

Kind Regards, Sdcb

[antonfirsov]

Threshold

ImageSharp has processors and extension methods for BinaryThreshold and AdaptiveThreshold (img.Mutate(ctx => ctx.BinaryThreshold(0.5f));). Is this what you are looking for?

FindContours

Similarly, there are several DetectEdges extensions which and various built-in kernels under KnownEdgeDetectorKernels.

Dilate

We can consider adding this to the core library if it can be implemented with a limited amount of self-contained code which does not need to pull in dependencies to robust linear algebra libraries (I'm not familiar with the algorithm). Is this something you would be interested contributing?

[ynse01]

Keep in kind that there are many more aspects to a Deep Learning library then Computer Vision alone.
However ImageSharp can be great for the Computer Vision part of your solution.

One such example in the OnnxRuntime project, where the ResNet50 C# tutorial uses ImageSharp.

[sdcb]

@antonfirsov I'm surprised that just awared ImageSharp already exists so many infrastructures to handle Computer Vision algorithms.

But I just checked

ImageSharp/src/ImageSharp/Processing/KnownEdgeDetectorKernels.cs

Line 11 in 945a83f

public static class KnownEdgeDetectorKernels

It seems DetectEdge will apply the border to the image in place, I expect get contours' as a array or something.

[antonfirsov]

array or something

Can you be more specific?

[sdcb]

I expect an array returned by Edge detection method, array of edges
This is the OpenCvSharp4's ABI for your reference: https://github.com/shimat/opencvsharp/blob/dff67281f3df2304426de4d3d42b6a41ea06d29f/src/OpenCvSharp/Cv2/Cv2_imgproc.cs#L3014

[JimBobSquarePants]

I would like, not “I expect”

[antonfirsov]

array of edges

This is beyond just applying an edge detection image filter. A contour extraction step is needed to get that information. From a quick search OpenCV uses the Suzuki and Abe algorithm to achive this. It's unlikely the core team will ever invest into studying and implementing this, since CV algorithms are outside of the primary scope of the library and our expertise.

Again, if this is something interesting and important for you and potentially for other users, we would be happy to see a community contribution solving this problem -- as long as it the added code is of a maintainable size and complexity.