non-trivial color formats support in FFmpeg >= 4.1

[bmegli]

Before FFMpeg (avcodec) 4.1 (e.g. Ubuntu 18.04 with 3.4) VAAPI FFmpeg pipeline performs colour conversions automatically.

Example:

• bgr0 (4:4:4 chroma) format as input, h264_vaapi encoder
• H.264 profile only supports 4:2:0 chroma, the conversion happens automatically

Since FFMpeg 4.1 (e.g. Ubuntu 20.04 with 4.2.4) this is no longer possible and results in

[h264_vaapi @ 0x55af43d3ee40] No usable encoding profile found.

From FFMpeg trac the solution is to use vaapi_scale filter to perform (hopefully) hardware conversion.

f2ab50f implements proof-of-concept working for bgr0 in 4.2.4 (Ubuntu 20.04).

Related to RNHVE#25

Related to UNHVD#27

[bmegli]

Solving it in generic way is non-trivial.

There are codecs and hardware that support 4:2:0, 4:2:2 and 4:4:4.

E.g. see HEVC 4:2:2 and 4:4:4 since IceLake.

This means that ideally we should match input chroma subsampling vs specific hardware supported subsampling. All in the context of bits per pixel (e.g. for 8 bit format we could fall back to NV12, for 10 bit format to P010LE).

There are complex cases like 4:2:2 input for HEVC with free kernel build which would not support encoding. Both 4:4:4 and 4:2:0 would be supported so we could go down to 4:2:0 or up to 4:4:4. I am not even sure it is possible to easily query hardware for support as avcodec user.

---

There is easy way to solve this in non-automatic way - just let user specify intermediate format for hardware conversion.

Advantages:

• supports all scenarios (chroma upsampling, downsampling, various bits per pixel)

Disadvantages:

• needs selection and some understanding from user code
• hardcoded path in user code may work with some hardware and fail on other (see this table)

The last case is rather rare.

[bmegli]

Fixed in 59f416f

It is enough to set frames_ctx->sw_format = AV_PIX_FMT_NV12 or AV_PIX_FMT_P010LE depending on data bit depth.

Note left in the code:

// Starting from FFmpeg 4.1, avcodec will not fall back to NV12 automatically
// when using non 4:2:0 software pixel format not supported by codec.
// Here, instead of using h->sw_pix_fmt we always fall to P010LE for 10 bit
// input and NV12 otherwise which may possibly lead to some loss of information
// on modern hardware supporting 4:2:2 and 4:4:4 chroma subsampling
// (e.g. HEVC with >= IceLake)

This means that fix may not be future proof (encoding 4:4:4 chroma input with HEVC and >= IceLake platform may possibly drop chroma information to 4:2:0, needs testing).