[BUG] 1.9.4 HDR Regression: black screen on AMD Vulkan

[randomgitname]

Describe the bug
Connecting to PS5 with HDR enabled on vulkan renderer causes black screen, 264 or 265 without hdr functions as expected.
As for dx11va 264, it shows a corrupted green effect over the stream, 265 and HDR appear to work as expected with short test.

Debug Log
Vulkan log 265 hdr, let me know if you want full logs with other specific settings.

[2025-01-13 22:40:46:125125] [I] Switched to profile 0, resolution: 1920x1080
[2025-01-13 22:40:46:168168] [W] [libplacebo] Masking sampleable from wrapped texture because the corresponding format 'rx10' does not support PL_FMT_CAP_SAMPLEABLE
[2025-01-13 22:40:46:168168] [W] [libplacebo] Masking blit_src from wrapped texture because the corresponding format 'rx10' does not support PL_FMT_CAP_BLITTABLE
[2025-01-13 22:40:46:168168] [W] [libplacebo] Masking sampleable from wrapped texture because the corresponding format 'rxgx10' does not support PL_FMT_CAP_SAMPLEABLE
[2025-01-13 22:40:46:168168] [W] [libplacebo] Masking blit_src from wrapped texture because the corresponding format 'rxgx10' does not support PL_FMT_CAP_BLITTABLE
[2025-01-13 22:40:46:168168] [I] [libplacebo] Picked surface configuration 4: VK_FORMAT_A2B10G10R10_UNORM_PACK32 + VK_COLOR_SPACE_HDR10_ST2084_EXT
[2025-01-13 22:40:46:169169] [E] [libplacebo] Validation failed: (image->planes[i]).texture->params.sampleable (src/renderer.c:2704)
[2025-01-13 22:40:46:170170] [E] [libplacebo] Backtrace:
[2025-01-13 22:40:46:171171] [E] [libplacebo] #0 0x7ffcf46334a3 in pl_render_image+0x14f3 (C:\Program Files (x86)\Sony\Chiaki-ng-Win\libplacebo-349.dll+0x34a3) (0x1800034a3)
[2025-01-13 22:40:46:171171] [E] [libplacebo] #1 0x7ffcf4632077 in pl_render_image+0xc7 (C:\Program Files (x86)\Sony\Chiaki-ng-Win\libplacebo-349.dll+0x2077) (0x180002077)
[2025-01-13 22:40:46:171171] [E] [libplacebo] #2 0x7ff738ec484a (C:\Program Files (x86)\Sony\Chiaki-ng-Win\chiaki.exe+0x6484a) (0x14006484a)
[2025-01-13 22:40:46:188188] [E] [libplacebo] #3 0x7ffcc142af5a in QMetaCallEvent::placeMetaCall+0x3a (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0xeaf5a) (0x1800eaf5a)
[2025-01-13 22:40:46:188188] [E] [libplacebo] #4 0x7ffcc1428dec in QObject::event+0x17c (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0xe8dec) (0x1800e8dec)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #5 0x7ffcf2462a4d in QApplicationPrivate::notify_helper+0x10d (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Widgets.dll+0x12a4d) (0x180012a4d)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #6 0x7ffcf2461b70 in QApplication::notify+0x18c0 (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Widgets.dll+0x11b70) (0x180011b70)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #7 0x7ffcc13dda5b in QCoreApplication::notifyInternal2+0x11b (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0x9da5b) (0x18009da5b)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #8 0x7ffcc13e0b68 in QCoreApplicationPrivate::sendPostedEvents+0x208 (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0xa0b68) (0x1800a0b68)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #9 0x7ffcc156e5ff in QEventDispatcherWin32::processEvents+0x8f (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0x22e5ff) (0x18022e5ff)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #10 0x7ffcc13e52ee in QEventLoop::exec+0x19e (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0xa52ee) (0x1800a52ee)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #11 0x7ffcc14cce2c in QThread::exec+0xcc (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0x18ce2c) (0x18018ce2c)
[2025-01-13 22:40:46:206206] [E] [libplacebo] #12 0x7ffcc1576248 in QThread::start+0x458 (C:\Program Files (x86)\Sony\Chiaki-ng-Win\Qt6Core.dll+0x236248) (0x180236248)
[2025-01-13 22:40:46:209209] [E] [libplacebo] #13 0x7ffd2145e8d6 in BaseThreadInitThunk+0x16 (C:\WINDOWS\System32\KERNEL32.DLL+0x2e8d6) (0x18002e8d6)
[2025-01-13 22:40:46:211211] [E] [libplacebo] #14 0x7ffd2227fbcb in RtlUserThreadStart+0x2b (C:\WINDOWS\SYSTEM32\ntdll.dll+0xdfbcb) (0x1800dfbcb)
[2025-01-13 22:40:46:211211] [W] Failed to render Placebo frame!

Desktop (please complete the following information):

• OS: Windows
• Version 24H2
• Device PC
• chiaki-ng Version 1.9.4

Additional context
AMD 24.12.1, 5700xt video card

[streetpea]

@randomgitname do you still get the issue with this build: https://github.com/streetpea/chiaki-ng/actions/runs/12775627883/artifacts/2430710115 ... what changed is that vulkan was re-enabled with a patch instead of falling back to d3d11va when using HDR. This enables zero copy which is preferred for performance reasons, but it looks like the patch isn't working for you. The original patch was tested and worked for another user but then it was changed because it caused a different issue and that user never retested it so maybe that caused it to stop working. I don't have a windows pc with an amd card with vulkan support to test.

[randomgitname]

Yes, this works in vulkan with 265 hdr.

[streetpea]

@randomgitname that was just a test but it break d3d11va with hdr

[randomgitname]

Just tested and yes it does break on my system.

[D] [libplacebo] (Re)creating 1920x1080x0 texture with format rx10: ../src/utils/upload.c:252
[D] [libplacebo] Texture creation failed: not supported
[E] [libplacebo] Failed initializing plane texture!
[W] Failed to map AVFrame to Placebo frame!

[randomgitname]

I also had to restart the app to get vulkan to work again after testing d3d.

[streetpea]

@randomgitname I woudn't close this issue because it's not using vulkan zero-copy but I updated the code to not fail when choosing the vulkan hw decoder and hdr on amd on Windows in v1.9.5. Interestingly enough the patch worked fine for amd on Linux radv driver but not for the amd radeon driver on Windows apparently.

[github-actions]

This issue is stale because it has been open for 90 days with no activity.