encode2 (old) FFmpeg API affecting encoder latency

[bmegli]

Short Description

FFmpeg introduced "new" API around 2016

• https://ffmpeg.org/pipermail/ffmpeg-devel/2016-March/190525.html
• which decoupled encoder input/output

The old API (encode2) generally can return only single packet per-frame (unless draining).

Cold encoder init may take a lot of time (100 ms order).

By the time we get first packet from encoder:

• a few packets are are already queued and ready
• but only one of them is returned on each encode
• and the latency that sets initially is kept until end of encoding session

[bmegli]

Above is only important for low-latency streaming (real-time)

[bmegli]

Checking if you are affected

The easiest way is to dump number or ready packets waiting when FFmpeg asks for packet.

Add cerr << "ctx->packet_pools->size() = " << ctx->packet_pools->size() << endl;

Before popping next packet:

jetson-ffmpeg/nvmpi_enc.cpp

Line 431 in 2006718

packet_index= ctx->packet_pools->front();

You will see how much packets are pending

• e.g. 6 packets with 30 fps
• means added ~200 ms latency (6 * 33.3(3) ms)

[bmegli]

Side Effects

The fact that we are lagging a few packets behind makes #130 non-fatal

There is concurrency issue but it generally affects fresh data.

We are typically working on data lagging a few packets behind

[bmegli]

Workarounds

Decrease ffmpeg-jetson num_capture_buffers through FFmpeg

-num_capture_buffers <int>        E..V..... Number of buffers in the capture context (from 1 to 32) (default 10)

This limits the number of pending packets possible

In L4T 5.1 with AGX Orin lowest safe value is 4 (see below).

Beware

Nvidia nvmpi doesn't respect all values.

In L4T 5.1 with AGX Orin requesting anything <= 3 results in

ctx->enc->capture_plane.getNumBuffers() = 3 ctx->enc->output_plane.getNumBuffers() = 4

You can check what happens for you by adding

• cerr << "capture.getNumBuffers() = " << ctx->enc->capture_plane.getNumBuffers() << " output.getNumBuffers() = " << ctx->enc->output_plane.getNumBuffers() << endl;
• after

jetson-ffmpeg/nvmpi_enc.cpp

Line 341 in 2006718

ret = ctx->enc->capture_plane.setupPlane(V4L2_MEMORY_MMAP, ctx->packets_num, true, false);

Side Effects

jetson-ffmpeg

• uses getNumBuffers() in some places
• in other FFmpeg input stored in ctx->packets_num

When nvidia-nvmpi doesn't respect your request

• those values become different

Exploit

Setting num_capture_buffers to 1 results in:

• ctx->enc->capture_plane.getNumBuffers() = 3 ctx->enc->output_plane.getNumBuffers() = 4
• and fresh packet always overwritten at first buf_index due to
  

  jetson-ffmpeg/nvmpi_enc.cpp

  Line 100 in 2006718

  ctx->buf_index=(ctx->buf_index+1)%ctx->packets_num;

You will:

• get low latency this way to at least see what is possible with Jetson
• but it is not safe and has side effects (!)
• especially in view of potential concurrency issue in encoder #130

[bmegli]

Improving Latency (sane way)

Implement jetson-ffmpeg with new FFmpeg API

• send_frame and receive_packet

Which allows easily getting any pending packets (multiple packets per frame)

This task is non-trivial.

[bmegli]

Functional fix is in #132

• fixes latency issues

It has other drawback - may miss last few packets in case of EOS.

Choose your poison.

[Keylost]

No need to choose poison) I have added some measures to prevent packet loss

[bmegli]

@Keylost

Your EOS handling looks reasonable

Looks like you ventured as far as to Nvidia V4L2 extensions docs

• https://docs.nvidia.com/jetson/l4t-multimedia/group__V4L2Enc.html

EOS Handling

The following sequence must be followed for sending EOS and recieving EOS from the encoder.

1. Send EOS to encoder by queueing on the output plane a buffer with bytesused = 0 for the 0th plane (v4l2_buffer.m.planes[0].bytesused = 0).
2. Dequeues buffers on the capture plane until it gets a buffer with bytesused = 0 for the 0th plane.

Or guessed it from their 2500 lines lump of code "example"

• /usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode_main.cpp

---

Anyway, your implementation looks like the one most ahead now

• nv_buf_utils and NVUtils support (old L4T, new L4T)
• fixed decoding performance in newer L4T + Jetpack
• fixed encoder latency issue
• FFmpeg 4, 5, 6 support
• and a few other fixes