HLS Support for OffthreadVideo #2930

sydneypalumbo · 2023-09-25T23:04:10Z

Hi there! I'm currently trying to use m3u8 files (from mux.com) as the src of <OffthreadVideo /> with remotion v4.0.19 but getting the following error during rendering. I thought this would be supported with ffmpeg, but is it not possible to use m3u8 with OffthreadVideo? Here's an example file, which doesn't seem to be corrupt. Thanks so much in advance for your help!

>  [compositor] [FFmpeg] Statistics: 2196 bytes read, 0 seeks
>  Error occurred Error: Compositor error: Invalid data found when processing input
>     0: std::backtrace::Backtrace::force_capture 
>     1: <compositor::errors::ErrorWithBacktrace as core::convert::From<ffmpeg_next::util::error::Error>>::from
>     2: compositor::opened_stream::open_stream
>     3: compositor::opened_video_manager::OpenedVideoManager::get_video
>     4: compositor::commands::execute_command
>     5: rayon_core::thread_pool::ThreadPool::install::{{closure}}
>     6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
>     7: rayon_core::registry::WorkerThread::wait_until_cold
>     8: std::sys_common::backtrace::__rust_begin_short_backtrace
>     9: core::ops::function::FnOnce::call_once{{vtable.shim}}
>    10: std::sys::unix::thread::Thread::new::thread_start
>    11: __pthread_joiner_wake
>      at onMessage (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:56:47)
>      at processInput (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:122:9)
>      at Socket.<anonymous> (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:142:9)
>      at Socket.emit (node:events:514:28)
>      at Socket.emit (node:domain:489:12)
>      at addChunk (node:internal/streams/readable:324:12)
>      at readableAddChunk (node:internal/streams/readable:297:9)
>      at Socket.Readable.push (node:internal/streams/readable:234:10)
>      at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

The text was updated successfully, but these errors were encountered:

JonnyBurger · 2023-09-26T11:13:51Z

Thanks a lot for reporting!

Actually, there is no explicit documentation on which codecs are supported by OffthreadVideo. I am fixing that with #2934.

We'll also add a FFmpeg binary which supports HLS in the next version after which we can assess whether this file can be supported.

JonnyBurger · 2023-09-29T07:35:50Z

The new version now throws a different error and it seems like it will take significant effort to support M3U8.

Until we sort it out, I recommend to enable MP4 files on Mux and use those to render!

sydneypalumbo · 2023-09-29T18:15:17Z

@JonnyBurger sounds good, thanks again!

mjdavidson · 2023-11-01T03:11:51Z

@JonnyBurger we're also very keen to use HLS with <OffthreadVideo />, to preview video edits on our front-end. How far away do you think support for this will be?

JonnyBurger · 2023-11-01T07:29:08Z

@mjdavidson If you just need it in the preview / Remotion Player, you can probably add hls.js to get it working!

For rendering, we need to parse the HLS format and download only the specific parts before feeding it to OffthreadVideo.

The scariest part is the HLS spec which seems quite long. I don't see a suitable library for JS/Rust that we can get working quickly, so I don't see it happening soon.

mjdavidson · 2023-11-02T01:28:23Z

@JonnyBurger yep, when we render the final video we have access to a normalized mp4 video so we will use that. We already use hls.js in our React project for our normal video player. How do you propose linking hls.js to the Remotion video component?

JonnyBurger · 2023-11-02T11:31:23Z

@mjdavidson Sure, I added a snippet for it here: https://www.remotion.dev/docs/miscellaneous/snippets/hls

Only tested some very basic cases though.

rmanalan · 2023-11-10T09:47:49Z

I got this working pretty well today. I did end up adding a few optimizations to the useEffect in the guide:

  useEffect(() => {
    if (!src) {
      throw new Error('src is required')
    }

    const hls = new Hls({
      enableWorker: !isSafari(),
      lowLatencyMode: true,
      backBufferLength: 20,
      startLevel: 4,
      maxBufferLength: 5,
      maxMaxBufferLength: 5,
    })
    if (hls) {
      hls.on(Hls.Events.MANIFEST_PARSED, () => {
        hls.startLoad(startFrom)
      })
      hls.attachMedia(videoRef.current!)
      hls.loadSource(src)
    }

    return () => {
      hls.destroy()
    }
  }, [src])

Also, the guide states that audio "will not render," but in my testing, audio works just fine in preview mode.

JonnyBurger · 2023-11-11T11:00:03Z

@rmanalan Thanks for sharing! Putting this back into the docs: #3155

JonnyBurger · 2023-11-11T11:05:32Z

By "will not render", I meant actually turning the video into an MP4, which I think is true. I'll clarify it as well.

tslater · 2024-02-23T07:40:23Z

@JonnyBurger Can you give a little more detail as to what errors you're getting or what you need to feed to the component? I'd like to try my hand at getting hls to work if you can give me the context.

JonnyBurger · 2024-02-23T10:24:06Z

@tslater I don't know the errors I got specifically, I just know that the native <video> tag doesn't support HLS by default and the FFmpeg C library doesn't recognize a .m3u8 as media like other video files.

It is just unknown to them which is why it doesn't work right now.

algora-pbc · 2024-03-20T06:07:43Z

💎 $226.11 bounty • nonsense.com

Steps to solve:

Start working: Comment /attempt #2930 with your implementation plan
Submit work: Create a pull request including /claim #2930 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to remotion-dev/remotion!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @feliciien	Mar 25, 2024, 2:24:20 PM	WIP
🟢 @ChetanBhosale	Jun 30, 2024, 11:51:20 AM	WIP

urbit-pilled · 2024-03-20T06:45:18Z

Looks like something I can fix, I'll give it a try
/attempt #2930

JonnyBurger · 2024-03-20T06:55:52Z

I think this is a really difficult issue to solve (at least for me 😄)

Do you have a plan on how to proceed? @urbit-pilled

(The bounty was not created by us nor someone we know!)

feliciien · 2024-03-25T14:24:18Z

/attempt #2930

Algora profile	Completed bounties	Tech	Active attempts	Options
@feliciien	5 bounties from 1 project	MDX, Rust, JavaScript & more		Cancel attempt

feliciien · 2024-03-25T21:09:47Z

@sydneypalumbo Hello can you tell me what command? Did you run for having this "> [compositor] [FFmpeg] Statistics: 2196 bytes read, 0 seeks

Error occurred Error: Compositor error: Invalid data found when processing input
0: std::backtrace::Backtrace::force_capture
1: <compositor::errors::ErrorWithBacktrace as core::convert::From<ffmpeg_next::util::error::Error>>:: from
2: compositor::opened_stream::open_stream
3: compositor::opened_video_manager::OpenedVideoManager::get_video
4: compositor::commands::execute_command
5: rayon_core::thread_pool::ThreadPool::install::{{closure}}
6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
7: rayon_core::registry::WorkerThread::wait_until_cold
8: std::sys_common::backtrace::__rust_begin_short_backtrace
9: core::ops::function::FnOnce::call_once{{vtable.shim}}
10: std::sys::unix::thread::Thread::new::thread_start
11: __pthread_joiner_wake
at onMessage (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:56:47)
at processInput (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:122:9)
at Socket. (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:142:9)
at Socket.emit (node:events:514:28)
at Socket.emit (node:domain:489:12)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Socket.Readable.push (node:internal/streams/readable:234:10)
at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)" and where I want to reproduce the same error so it will be more easy for me to fix it

feliciien · 2024-03-25T21:10:27Z

I want to reproduce the same error on my editor

0018akhil · 2024-04-22T14:31:48Z

@JonnyBurger

Is there a time limit to a video?

There should be a time limit to the video because we can play live stream via the .m3u8 link.

I tried playing a video, it gets the packets(segments) in the network tab but the video player only plays the video once it gets all the packets(segments).

ChetanBhosale · 2024-06-30T11:51:16Z

/attempt #2930

Options

Cancel my attempt

sydneypalumbo added the bug Something isn't working label Sep 25, 2023

JonnyBurger mentioned this issue Sep 26, 2023

@remotion/renderer: Add HLS decoder and asetrate filter to FFmpeg binary #2935

Merged

JonnyBurger changed the title ~~Rendering OffthreadVideo with m3u8 results in "Compositor error: Invalid data found when processing input"~~ HLS Support for OffthreadVideo Oct 19, 2023

JonnyBurger removed the bug Something isn't working label Jan 31, 2024

algora-pbc bot added the 💎 Bounty label Mar 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLS Support for OffthreadVideo #2930

HLS Support for OffthreadVideo #2930

sydneypalumbo commented Sep 25, 2023

JonnyBurger commented Sep 26, 2023 •

edited

Loading

JonnyBurger commented Sep 29, 2023

sydneypalumbo commented Sep 29, 2023

mjdavidson commented Nov 1, 2023

JonnyBurger commented Nov 1, 2023

mjdavidson commented Nov 2, 2023

JonnyBurger commented Nov 2, 2023

rmanalan commented Nov 10, 2023 •

edited

Loading

JonnyBurger commented Nov 11, 2023

JonnyBurger commented Nov 11, 2023

tslater commented Feb 23, 2024

JonnyBurger commented Feb 23, 2024

algora-pbc bot commented Mar 20, 2024 •

edited

Loading

urbit-pilled commented Mar 20, 2024 •

edited

Loading

JonnyBurger commented Mar 20, 2024

feliciien commented Mar 25, 2024 •

edited by algora-pbc bot

Loading

feliciien commented Mar 25, 2024

feliciien commented Mar 25, 2024

0018akhil commented Apr 22, 2024

ChetanBhosale commented Jun 30, 2024 •

edited by algora-pbc bot

Loading

HLS Support for OffthreadVideo #2930

HLS Support for OffthreadVideo #2930

Comments

sydneypalumbo commented Sep 25, 2023

JonnyBurger commented Sep 26, 2023 • edited Loading

JonnyBurger commented Sep 29, 2023

sydneypalumbo commented Sep 29, 2023

mjdavidson commented Nov 1, 2023

JonnyBurger commented Nov 1, 2023

mjdavidson commented Nov 2, 2023

JonnyBurger commented Nov 2, 2023

rmanalan commented Nov 10, 2023 • edited Loading

JonnyBurger commented Nov 11, 2023

JonnyBurger commented Nov 11, 2023

tslater commented Feb 23, 2024

JonnyBurger commented Feb 23, 2024

algora-pbc bot commented Mar 20, 2024 • edited Loading

💎 $226.11 bounty • nonsense.com

Steps to solve:

urbit-pilled commented Mar 20, 2024 • edited Loading

JonnyBurger commented Mar 20, 2024

feliciien commented Mar 25, 2024 • edited by algora-pbc bot Loading

feliciien commented Mar 25, 2024

feliciien commented Mar 25, 2024

0018akhil commented Apr 22, 2024

ChetanBhosale commented Jun 30, 2024 • edited by algora-pbc bot Loading

JonnyBurger commented Sep 26, 2023 •

edited

Loading

rmanalan commented Nov 10, 2023 •

edited

Loading

algora-pbc bot commented Mar 20, 2024 •

edited

Loading

urbit-pilled commented Mar 20, 2024 •

edited

Loading

feliciien commented Mar 25, 2024 •

edited by algora-pbc bot

Loading

ChetanBhosale commented Jun 30, 2024 •

edited by algora-pbc bot

Loading