Would a GStreamer plugin be feasible?

[andrew-otiv]

The gstreamer library provides a high level API for streaming data into and out of many container formats. It's a tool that supports every Audio/Video compression or container format anyone's heard of... but not MCAP as far as I can tell.

Would it be feasible to implement GStreamer source and sink plugins that can stream audio and video to/from MCAP file, or are there limitations in either the MCAP container or Gstreamer that would prevent this integration?

For more context, we have a project that has both custom PUB/SUB telemetry, as well as compressed audio and video streams. I'm still evaluating if it makes more sense to:

• extend MCAP (with custom schemas) to also include audio and whatever compressed video format we settle on, or to
• extend an AV container like Matroska (with custom schemas) to contain our custom telemetry data, or to
• give up on container formats and write our own logger and viewer code that separately loads telemetry and AV data.

If there were a gstreamer plugin, it would give me some confidence that AV data beyond individually compressed images (starting, say, with the formats in WebRTC) can actually work well with MCAP and the rest of the foxglove platform.

This isn't a feature request; I'm just curous if someone's already thought about/evaluated this.

For completeness, a related discussion about exporting mcap to video: https://github.com/orgs/foxglove/discussions/124

Thanks!
Andrew

[andrew-otiv]

And another audio related issue (the acapelleabots) https://github.com/foxglove/community/issues/147
which got moved to a discussion: https://github.com/orgs/foxglove/discussions/173

[andrew-otiv]

There is a ROS2 gstreamer bridge: https://github.com/BrettRD/ros-gst-bridge
will study that more; maybe that can work if we're willing to take a dependency on ROS2.

[jtbandes]

I don't have an immediate full answer to your question (I don't believe anyone has created such a GStreamer plugin yet) but just to address one item:

• extend MCAP (with custom schemas) to also include audio and whatever compressed video format we settle on

First of all, we are making a foray into compressed video support with H.264: https://foxglove.dev/blog/announcing-h264-support-in-foxglove Please try it out and give us your feedback!

And also wanted to clarify that these things would not require an extension to MCAP itself, as MCAP is a generic pub/sub container format. While the Foxglove visualization tools understand a set of specific schemas (the Foxglove schemas + many standard ROS msgs), the MCAP layer does not particularly care what schemas you use, so it doesn't preclude folks from storing non-ROS/non-Foxglove data inside MCAP and building their own tooling around it.

[andrew-otiv]

Hey thanks for the reply! Good to know that even h264 support is still considered experimental to manage expectations. I didn't even consider MCAP (or pub/sub) for compressed video until I saw your schema for CompressedImage having an option for h264.

Indeed, I understand the part about the container being extensible by adding schemas and also updating the surrounding recording and playback tooling. Matroska and some other containers also claim to be totally generic regarding what they can contain, but I still expect to find important differences in their capabilities vs. MCAP that might make or break our use case. For example MCAP will already work with our telemetry schemas, but I'd be surprised if matroska does; it's probably older than flatbuffers.