d3d11vpp: add GPU scale option for intel and nvidia GPUs

[billxc]

usage:
vf set d3d11vpp=scale=nvidia:scale-target

PS:
The pull requests is has not been rebased to one commit yet, as I still need some changes, and will rebase to one commit once ready for merge.

[billxc]

Quate the comment from #11390

@billxc First off, it's great that you're working on this, and thank you for looking at my original feedback. However, after taking an initial look at your tree, it looks like you have written it as a separate filter. For it to be mergeable, the functionality should be added to d3d11vpp. Were you just doing it as a separate filter as an initial development step? I can't tell.
Also, the filter's configuration interface should not hard code the "Super Resolution" concept - as I understand it, it's just a generic scaling function and vendors or anyone can write and plug in a scaling algorithm (I see you have added Intel as a second example), so it should just be a scale configuration. It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?). Thanks!
Feel free to open a PR - it will be easier to discuss there.
Originally posted by @philipl in #11390 (comment)

Also to make it scalable for the future we could extend options of d3d11vpp with arbitrary GUID that user could provide. Would avoid having to recompile mpv for different variant of the same. I think AMD has some sort of super res too, which has to be enabled, because normally this interface just scales without fancy features.
Originally posted by @kasper93 in #11390 (comment)

[billxc]

@philipl

I have refacted the code to reuse d3d11vpp, also updatd the option name. There will be another change later to change SuperResolution to Scale to avoid potential copyright issue.

It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?)

RTX video do support any resolution. Idealy, we should take the actual window size, and use them for the target height/width
but I am not quite familiar with MPV code, Would you help provide some examples for getting the actual window height and width?

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

---

Also I have tried to use the intel auto HDR API, but failed, do we have examples for turning a SDR video to HDR video.

[kasper93]

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

I don't think they call their video hq scaler FSR in any documentation. Though indeed looking at it they did expose it through own interface, rather than rolling custom video processor in d3d11 video processing interface (to be confirmed if not available through it, not sure what browsers use). It could live in the same mpv filter (d3d11vpp), same as they integrated it in VLC.

See:
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_HQ_Scaler_API.md
https://code.videolan.org/videolan/vlc/-/merge_requests/4953

[ruihe774]

The AMF doc says it supports

• DirectX 11
• DirectX 12
• Vulkan
• OpenCL

FWIW I'm curious about whether it is possible to be implemented as a cross-platform filter with both d3d11 and vulkan backends.

[kasper93]

For AMD AMF stuff, it seems to be integrated in ffmpeg https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=11632 so we can skip this part and have it for free once it get merged.

[Jules-A]

For AMD AMF stuff, it seems to be integrated in ffmpeg https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=11632 so we can skip this part and have it for free once it get merged.

FFMPEG are being difficult again (https://www.phoronix.com/news/AMD-AMF-FFmpeg-Better-2024) so not really sure if it would get merged soon.

[SaberDirewolf]

@philipl

I have refacted the code to reuse d3d11vpp, also updatd the option name. There will be another change later to change SuperResolution to Scale to avoid potential copyright issue.

It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?)

RTX video do support any resolution. Idealy, we should take the actual window size, and use them for the target height/width but I am not quite familiar with MPV code, Would you help provide some examples for getting the actual window height and width?

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

Also I have tried to use the intel auto HDR API, but failed, do we have examples for turning a SDR video to HDR video.

Not sure if this is helpful, but was staring at how Chrome's Auto HDR works and is the original commit (there might be bug fixes after this commit): chromium/chromium@b2fae4c

Here is another example: Aleksoid1978/VideoRenderer#128

[kasper93]

Also needs fixes for gcc compilation and commits should be squashed into logical parts.

[srk24]

is this helpful?
https://docs.nvidia.com/rtx/ngx/programming-guide/index.html#dlvsr

[Hrxn]

@srk24 What is this NGX? Is it even supported anymore?

[philipl]

@srk24 What is this NGX? Is it even supported anymore?

NGX is off topic for the work in this PR. It's nvidia's umbrella for some AI based image/video processing scalers - none of which are usable via the d3d11 scaling mechanism covered by this PR.

[billxc]

updated the easy parts.
and still some TODOs:

1. Seeking correct place to do the uploading/downloading
2. Support any scale ratio
3. Support getting current VO size for scaling output

update:

Tasks 1, 2, and 3 all require extensive knowledge of MPV, which I currently lack.

Therefore, I will keep the changes small:

1. Remove support for IMG_FMT_420P, maintaining the same rule as d3d11vpp.
2. Hard code the scale output window to 1440p. Considering that this feature only supports recent graphics cards, this should strike a balance between performance and quality.

[wadixx]

updated the easy parts. and still some TODOs:

1. Seeking correct place to do the uploading/downloading
2. Support any scale ratio
3. Support getting current VO size for scaling output

Hey any update on TODOs?

[SPiCa-P]

Hi, sorry for bothering you.
Thank you for your excellent work!
When I used your build, my mpv crashed using vf=d3d11sr=mode=nvidia after 1-3min. I checked both output-error logs from mpv and windows error logs, the only useful message is in the windows error log, that is:

‘d3d11 crash 0xc0000005'

Is there any other way to detect issues or logs? Thank you so much!

[billxc]

Hi, sorry for bothering you. Thank you for your excellent work! When I used your build, my mpv crashed using vf=d3d11sr=mode=nvidia after 1-3min. I checked both output-error logs from mpv and windows error logs, the only useful message is in the windows error log, that is:

‘d3d11 crash 0xc0000005'

Is there any other way to detect issues or logs? Thank you so much!

The code has changed a lot since that version, it may be a memory leak or something like that.

[billxc]

updated the easy parts. and still some TODOs:

1. Seeking correct place to do the uploading/downloading
2. Support any scale ratio
3. Support getting current VO size for scaling output

update:
Tasks 1, 2, and 3 all require extensive knowledge of MPV, which I currently lack.

Therefore, I will keep the changes small:

1. Remove support for IMG_FMT_420P, maintaining the same rule as d3d11vpp.
2. Hard code the scale output window to 1440p. Considering that this feature only supports recent graphics cards, this should strike a balance between performance and quality.

[AlawamiAZ]

using this build gives me this artifact on certain videos using the vf=d3d11sr=mode=nvidia:scale=1440p option

the videos with this issue can be found using with this bittorrent hash 27fe4e276f5b3d6997d9409ae844b32eeae278f7

[billxc]

Updated

[billxc]

using this build gives me this artifact on certain videos using the vf=d3d11sr=mode=nvidia:scale=1440p option the videos with this issue can be found using with this bittorrent hash 27fe4e276f5b3d6997d9409ae844b32eeae278f7

There is something wrong when converting the YUV420P formt to NV12.
YUV420P is not supported by d3d11vpp before, and I tried to support this. It led to many bugs, and some are beyond my capability. so in the current pr, YUV420P will not be supported by d3d11vpp just like before.

[github-actions]

Download the artifacts for this pull request:

Windows

• mpv-i686-w64-mingw32
• mpv-x86_64-w64-mingw32
• mpv-x86_64-windows-msvc

macOS

• mpv-macos-12-intel
• mpv-macos-13-intel
• mpv-macos-14-arm

[raven02]

@billxc do you mind updating the https://github.com/billxc/mpv/releases/tag/MPV-RTX-SuperRes-RC with your latest build ?

[raven02]

using this build gives me this artifact on certain videos using the vf=d3d11sr=mode=nvidia:scale=1440p option the videos with this issue can be found using with this bittorrent hash 27fe4e276f5b3d6997d9409ae844b32eeae278f7

There is something wrong when converting the YUV420P formt to NV12. YUV420P is not supported by d3d11vpp before, and I tried to support this. It led to many bugs, and some are beyond my capability. so in the current pr, YUV420P will not be supported by d3d11vpp just like before

I tried older build which use d3d11vpp but it always failed when converting YUV420P format. D3d11sr is fine but it crashed 2-3 mins aftrerward. Anyway to get d3d11vpp working without YUV420P

[srk24]

I have poor knowledge for c & mpv, but hope this ft can be merged. I tried to read the code, does this code help to get the size of the vo? use dheight & dwidth.
vo.c
void vo_get_src_dst_rects(struct vo *vo, struct mp_rect *out_src, struct mp_rect *out_dst, struct mp_osd_res *out_osd)
mp_image_params_get_dsize

vo_direct3d.c
GetClientRect(vo_w32_hwnd(priv->vo), &window_rc);

like this? :

RECT clientRect;
GetClientRect(vo_w32_hwnd(priv->vo), &clientRect);
int windowWidth = clientRect.right - clientRect.left;
int windowHeight = clientRect.bottom - clientRect.top;