Applicability question

[realizator]

Hello @bmegli,
You mentioned you've streamed Realsense video to the Unity. We have a similar task, and need a low-latency livestream from the stereoscopic camera (StereoPi, based on Raspberry Pi) to the Oculus Quest.
How do you think, can we use your solution? If it will have hardware-accelerated decode on Oculus Quest?

Regards,
Eugene

upd> This ticket is opened following your answer here: https://answers.unity.com/questions/1627714/stream-live-camera-view-over-udp.html

[bmegli]

Hi @realizator,

I would start by examining ALVR which claims working (alpha) support for streaming to Oculus Quest. If not as end-to-end solution (PC vs RPi) then at least to get idea how to start.

The first question you should ask yourself is - do you really need hardware acceleration.

As far as HVS is concerned

Encoding

HVE

• HVE (the encoder) currently only supports VAAPI hardware acceleration
• this means mostly Linux and mostly Intel
• this means it will probably not work with RPi
• without making modifications to the code

FFmpeg

Have a look at FFmpeg HWAccelIntro Platform API Availability:

In the context of RapberryPi you have:

• MMAL hardware acceleration
• OpenMax (OMX) hardware acceleration

Examining the table FFmpeg API Implementation Status:

• MMAL is only for decoding
• OMX works for encoding and is usable from CLI (command line)

This means that OMX would be the technology to go for hardware encoding on RPi.

I would first try with ready-made tools if hardware acceleration worked (gstreamer and FFmpeg).

Coming back to HVE

It is likely that OMX uses the some kind of interface that VAAPI does in FFmpeg and would require only minor changes in the code (see this issue).

Decoding

HVD

HVD supports multiple hardware decoding technologies but none of them is supported on Android.

FFmpeg

I know little about OculusQuest apart from the fact that it has Snapdragon (ARM) and runs Android.

My experience with FFmpeg hardware accelerated decoding on Android is bad (see this issue).

Unless something changed, based on my past experience, I would discourage you from using FFmpeg based MediaCodec hardware decoding on Android. Maybe I just have bad memories. Maybe it should be only used in zero-copy scenario on low-end Android devices (which doesn't seem to be possible with Unity)

ALVR

Which brings us back to ALVR, which as was stated claims support for streaming to OculusQuest.

Transport

Apart from hardware or software encoding/decoding you have to transport the data between the devices. Again, I would see what ALVR does as a starting point.

I have managed to run streaming pipeline to Unity application on Android but I would say that it didn't meet my expectations.

My NHVE, RNHVE and UNHVD:

• use custom protocol
• which is good for my rapid prototyping and experiments
• but would be a disaster for long term support

You may also have a look at DJI Android Video Stream Decoding Sample, which was open source as far I remember and used hardware decoding through MediaCodec (FFmpeg for parsing H.264 and MediaCodec directly from Java as it was originally designed to be used, not through FFmpeg).

Kind regards

[bmegli]

Hi @realizator

If you have no more question I am going to close the issue.

[realizator]

Hi @bmegli,
Thank you for your tips! We are now in the process of experimenting.