StitchingFramesToVideo while rendering is occurring, instead of after renderFrames is finished

[cachemoney]

Hi, I am trying to see if its possible to start stitching the frames after an arbitrarily number of frames have been rendered. I am going through render.ts to understand the assetsInfo object as that is a dependency for the stitchFramesToVideo function. Is there any guidance on how to refactor the code so that I can either do:

• calculate assetsInfo ahead of time so I can start stitching the frames after the 200th frame is rendered, for example?
• render 200 frames, call StitchFramesToVideo, render the next 200 frames and call stitchFramesToVideo again, and loop until it reaches the frameCount

Thanks!

[JonnyBurger]

Hey @cachemoney, very good question and indeed this is possible and already implemented in the V3 branch! There will be a new API called renderMedia() that will combine renderFrames() and stitchFramesToVideo() and do it in parallel.

[JonnyBurger]

Hi there,

you might be interested in the onFrameBuffer property in the V3 version of renderFrames(): https://v3.remotion.dev/docs/renderer/render-frames#onframebuffer

Every time a new frame gets rendered, you get a buffer that you can pipe into FFMPEG. But we have not entered the RTMP territory yet, so you'd have to create some custom code for that. Our v3 version of stitchFramesToVideo might help as a reference: https://github.com/remotion-dev/remotion/blob/lambda-testing/packages/renderer/src/stitcher.ts

You cannot change the video during render, so you'd have to cancel it and start a render again from the position you started.
(We currently do not have an easy way to cancel a render, but if this is interesting to you, we can implement this for you)

[cachemoney]

Thanks for the details. I just want to test my understanding of onframebuffer, when its enabled. I have been reading over the render-media and stitcher ts files.

• Does the Buffer object include audio along with the video?
• Do image files not get created at all? It's only buffers that are being piped into FFMPEG?
• Can this this method be recommended if one needs to make I/O requests, for example, to parameterize the render?
• If onframebuffer is not used, does rendering and and stitching run in parallel, with the stitching running XXXX frames behind the render?
  • if so, can I control the lag(# of frames) of stitcher to run behind the renderer?

You cannot change the video during render, so you'd have to cancel it and start a render again from the position you started.

I was thinking of a workaround without having to implement a cancelling render API. If I were to have an expressJS app along with a Remotion react app:

• Can the server(expressJS) send Server Side Events(SSE) to a react component, that updates its state? That way, dynamic content can be passed during rendering?
  • Would delayRender()/continueRender() be needed as a best practice?

Thanks again for your help.

[JonnyBurger]

Hi @cachemoney,

• No, the buffer does not include audio unfortunately, just a single image.
• That's right, no actual file is created, only a buffer.
• We designed it so that the parameters are supposed to defined upfront, so currently I cannot really recommend it because we never had live streams in mind 😞

I think render cancellation and dynamic prop updating can be done technically.
Cancellation seems like a useful feature to me that if someone needs it, I would implement in response.
To update props in real-time might be problematic, because unless you render with 1x concurrency, frames do not necessarily render in order.

Sorry for the late-ish response!

[cachemoney]

Thanks, it looks like a render cancellation API will help me achieve what I am trying to get to. My use case will utilize <Audio> tags so that FFMPEG will mux the audio to create the video file/stream. Hopefully the the new API will take that into account.

Thanks a lot!

[JonnyBurger]

@cachemoney: I created an issue to track it: #812