Greyscale Camera support?

[sferguson-ISI]

Description

Hello, I am trying to get a monochrome camera to work with the camera example detect.py code. Specifically I have a See3CAM_CU55M grayscale camera I'm attempting to get working with the called Gstreamer pipeline code in the example. I can get the camera outputting a stream with gst-launch-1.0 -v v4l2src device=/dev/video1 ! "video/x-raw, format=GRAY8, width=1280, height=720" ! videoconvert ! autovideosink sync=false.

However when I attempt to launch the detect.py like so: python3 /home/mendel/test/examples-camera_portion/detect.py --model /home/mendel/pre_trained_model/edgetpu/test_data/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite --labels /home/mendel/pre_trained_model/edgetpu/test_data/coco_labels.txt --videosrc /dev/video1 --threshold 0.7 --top_k 10

I get this output:
Loading /home/mendel/pre_trained_model/edgetpu/test_data/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite with /home/mendel/pre_trained_model/edgetpu/test_data/coco_labels.txt labels. Neural network weight read [{'name': 'normalized_input_image_tensor', 'index': 7, 'shape': array([ 1, 300, 300, 3], dtype=int32), 'shape_signature': array([ 1, 300, 300, 3], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.0078125, 128), 'quantization_parameters': {'scales': array([0.0078125], dtype=float32), 'zero_points': array([128], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}] Input tensor size: (300, 300) Gstreamer pipeline: v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! decodebin ! glupload ! tee name=t t. ! queue ! glfilterbin filter=glbox name=glbox ! video/x-raw,width=300,height=300 ! appsink name=appsink emit-signals=true max-buffers=1 drop=true t. ! queue ! glsvgoverlaysink name=overlaysink

Error: gst-stream-error-quark: Internal data stream error. (1): gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming stopped, reason not-negotiated (-4)

I tried adding in a format=GRAY8 at the source part of the pipeline but still I get the same error. Any recommendations on what to try next? Does this have to do with the fact the camera is a monochrome camera?

Click to expand!

Issue Type

Support

Operating System

Mendel Linux

Coral Device

Dev Board

Other Devices

No response

Programming Language

Python 3.8

Relevant Log Output

Loading /home/mendel/pre_trained_model/edgetpu/test_data/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite with /home/mendel/pre_trained_model/edgetpu/test_data/coco_labels.txt labels. Neural network weight read [{'name': 'normalized_input_image_tensor', 'index': 7, 'shape': array([ 1, 300, 300, 3], dtype=int32), 'shape_signature': array([ 1, 300, 300, 3], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.0078125, 128), 'quantization_parameters': {'scales': array([0.0078125], dtype=float32), 'zero_points': array([128], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}] Input tensor size: (300, 300) Gstreamer pipeline: v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! decodebin ! glupload ! tee name=t t. ! queue ! glfilterbin filter=glbox name=glbox ! video/x-raw,width=300,height=300 ! appsink name=appsink emit-signals=true max-buffers=1 drop=true t. ! queue ! glsvgoverlaysink name=overlaysink

Error: gst-stream-error-quark: Internal data stream error. (1): gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming stopped, reason not-negotiated (-4)

[sferguson-ISI]

Just as a follow up I'm assuming it's the greyscale camera that's causing the issue as I have tried a color version and it works with no issue. It's a E-con Systems See3CAM_CU55 MHL camera. The v4l2-ctl output:

`v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'GREY' (8-bit Greyscale)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.250s (4.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.100s (10.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.042s (24.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
        [1]: 'Y12 ' (12-bit Greyscale)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.500s (2.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.200s (5.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.083s (12.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)`

[hjonnala]

I think, it only supports YUYV format: https://coral.ai/docs/dev-board/camera/#connect-a-usb-camera

[sferguson-ISI]

Thanks @hjonnala , is there possibly a way to duplicate the single grey channel to the other channels an pipe that in as an artificial YUYV format?

[hjonnala]

is there possibly a way to duplicate the single grey channel to the other channels an pipe that in as an artificial YUYV format?

Unfortunately, I am not aware of any possible way of doing that.

[sferguson-ISI]

You gave me a clue with the YUYV format. I'm able to get the object detection going now with this pipeline: GST_DEBUG=2 gst-launch-1.0 -v v4l2src device=/dev/video1 ! "video/x-raw, format=GRAY8, width=640, height=480" ! videoconvert ! video/x-raw, format=YUY2, width=640,height=480 ! autovideosink sync=false

In the gstreamer.py code I had to tack on the output by specifying the conversion string in "run_pipeline" function. Specifically the "SRC_CAPS" I had to modify with: SRC_CAPS = 'video/x-raw,format=GRAY8,width={width},height={height} ! videoconvert ! video/x-raw, format=YUY2, width=640,height=480'

Now the issue is that the framerate is sitting at 6 fps, basically unusable. Is there a different place I should be specifying this conversion string? In the 'scale_caps' perhaps?