5x latency increase with VR Headset attached (receiver)

[aivenhoe]

Description of the bug

A sender pc runs an application that sends a texture through a capture card (Decklink 8K Pro). The receiver PC has a capture card too, (Decklink 8K Pro) and receives the texture stream (using the licensed (!) AVPro DeckLink plugin) that is rendered on a geometry in the scene.

On the receiver PC, when using a regular unity camera, the latency between the texture sender and the receiving pc is around 9-10 frames (running at 30fps, see sceenshot below).

Whereas when using a Varjo Headset attached, the latency appears as around 40 to 50 frames (also running at 30fps, see Screenshot below).

I assume the the latency between the texture sender and the texture receiver does not really change, but I rather think that the Varjo plugin hooks into the renderpipeline causing a delay in the texture creation of the AVPro Decklink plugin.

My Setup:

• Unity version: 2022.3.12.f1 HRDP Renderpipeline
• AVPro DeckLink version: 1.9.7
• Operating system version: Windows 10 IoT Enterprise 22H2
• Hardware model: AMD Ryzen Threadripper PRO 5965WX 24-Cores 3.80 GH / NVIDIA GeForce RTX 4090 / Decklink 8K Pro
• Cables used: HDTV1RTX06L (good ones)
• Desktop Video version: 12.7.1
• Video mode (resolution, frame-rate, pixel-format): 8K UHD (7680 × 4320), 30fps, 10-bit YUV 4:2:2

Steps to Reproduce

• set up a sender pc, sending 8k uhd, 30fps, 10-bit YUV 4:2:2 texture
• start receiving end with Decklink 8K Pro and a Unity application with the AVPro Decklink plugin, using the provided Input demo in HDRP. Notice a low latency.
• add a Varjo headset (probably the problem persists with another vr headset like the meta quest) and a unity xr-rig with a camera. Notice 5 times latency increase.

[aivenhoe]

I did a quick test with the very old klinker plugin from keijiro and the varjo headset connected. There the latency was around 13 frames with the same video mode and unity version, see screenshot below.

the color in HDRP is off obviously, but is correct in the build-in render pipeline. any ideas?
the latency with this plugin is better, but as stated from the author, it is not recommended to use this for production.