gstreamer: Could not use gstreamer.run_pipeline with MP4 videosrc

[jleger12]

Description

I am trying to leverage the gstreamer.py pipeline definition to run inference on test MP4 data instead of via the /dev/video1 webcam.

Looking at the run_pipeline function, this appears to be supported by the following else case on line 223 with the use of qtdemux:

else:
        demux =  'avidemux' if videosrc.endswith('avi') else 'qtdemux'
        PIPELINE = """filesrc location=%s ! %s name=demux  demux.video_0
                    ! queue ! decodebin  ! videorate
                    ! videoconvert n-threads=4 ! videoscale n-threads=4
                    ! {src_caps} ! {leaky_q} """ % (videosrc, demux)

When initializing the gstreamer pipeline, the following error occurs:

Warning: gst_parse_error: Delayed linking failed. (7): ./grammar.y(510): gst_parse_no_more_pads (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0:
failed delayed linking some pad of GstDecodeBin named decodebin0 to some pad of GstVideoRate named videorate0
Error: gst-stream-error-quark: Internal data stream error. (1): qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:demux:
streaming stopped, reason not-linked (-1)

Click to expand!

Issue Type

Bug, Support

Operating System

Mendel Linux

Coral Device

Dev Board Mini

Other Devices

No response

Programming Language

Python 3.7

Relevant Log Output

No response

[thebruce87m]

Did you get to the bottom of this? Closest I've come to running my own video on the Dev Board Mini is to convert the mp4 using the following:

ffmpeg -i drop.avi -an -vf "scale=960:540,format=yuv420p" -colorspace bt470bg -color_range tv -color_primaries smpte170m -color_trc bt709 -vcodec libx264 -brand mp42 -c:v libx264 -profile:v baseline drop.mp4

and then run it via:

gst-launch-1.0 filesrc location=drop.mp4 ! decodebin ! autovideoconvert ! autovideosink

I can't quite get this to work in the gstreamer.py script however

[jleger12]

@thebruce87m : I was able to put something together with help from a RidgeRun example pipeline

The solution required a few extra components to be installed:

• Install libuav (we were missing the avdec_h264 gstreamer plugin)
  • sudo apt install gstreamer1.0-libav
• Confirm plugin is installed
  • gst-inspect-1.0 avdec_h264

I was only targeting MP4 files, so I dropped the AVI support and ended up with the following pipeline definition:

PIPELINE = """filesrc location=%s ! qtdemux name=demux  demux.video_0 !
                    h264parse ! avdec_h264 ! videorate ! videoconvert n-threads=4 ! 
                    videoscale n-threads=4 ! {src_caps} ! {leaky_q} """ % (videosrc)

Let me know if this works for you and/or if there are any changes/optimizations that can be made. I admittedly only know enough GStreamer to be dangerous so would welcome a second set of eyes!

[thebruce87m]

@jleger12

Thanks, with this edit to gstreamer.py along with changing to glimagesync I am able to run the example:

diff --git a/gstreamer/gstreamer.py b/gstreamer/gstreamer.py
index 13dcd05..adff762 100755
--- a/gstreamer/gstreamer.py
+++ b/gstreamer/gstreamer.py
@@ -223,10 +223,11 @@ def run_pipeline(user_function,
         PIPELINE = 'rtspsrc location=%s'%videosrc
     else:
         demux =  'avidemux' if videosrc.endswith('avi') else 'qtdemux'
-        PIPELINE = """filesrc location=%s ! %s name=demux  demux.video_0
-                    ! queue ! decodebin  ! videorate
-                    ! videoconvert n-threads=4 ! videoscale n-threads=4
-                    ! {src_caps} ! {leaky_q} """ % (videosrc, demux)
+
+        PIPELINE = """filesrc location=%s ! qtdemux name=demux  demux.video_0 !
+                      h264parse ! avdec_h264 ! videorate ! videoconvert n-threads=4 ! 
+                      videoscale n-threads=4 ! {src_caps} ! {leaky_q} """ % (videosrc)
+
     ''' Check for the object tracker.'''
     if trackerName != None:
         if trackerName == 'mediapipe':
@@ -254,7 +255,7 @@ def run_pipeline(user_function,
             t. ! {leaky_q} ! videoconvert ! videoscale ! {scale_caps} ! videobox name=box autocrop=true
                ! {sink_caps} ! {sink_element}
             t. ! {leaky_q} ! videoconvert
-               ! rsvgoverlay name=overlay ! videoconvert ! ximagesink sync=false
+               ! rsvgoverlay name=overlay ! videoconvert ! glimagesink sync=false
             """
     if objectOfTracker:
         mot_tracker = objectOfTracker.trackerObject.mot_tracker

Now what I don't understand is that the inference time claims to be 140ms, but the fps claims to be 0 fps - which I think really means very slow. This is reflected by the bounding boxes not updating very often compared to the video output.

Surely if the inference is 140ms I should be getting ~ 7 fps? Do you have any insights here? If not I'll add a new issue.

Thanks for your help!

[alkurop]

Hi guys. A bit of a delayed response.

I was also fighting with this one, and ended up streaming video with vlc to rtsp, and consuming it by gstreamer
Not sure if this is reasonable but I'm getting around 11 fps.

An other problem with direct video was that gstreamer played it in ff, not in correct frame rate. But now that vlc controls frame rate, I don't have any issues other then an increased inference delay compared to camera input.

vlc sample.mp4 --sout="#rtp{sdp=rtsp://:8080/stream}"

python3 detect.py --videosrc "rtsp://localhost:8080/stream latency=0 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! videorate ! video/x-raw,framerate=30/1 "