Ported rtAudio to portaudio for evaluation.

[ggarra13]

This is a first attempt at porting rtAudio to portaudio for evaluation.

[darbyjohnston]

Hi @ggarra13, very interesting PR, thanks! I was looking at the PortAudio information and like the license and CMake support. If PortAudio also supports audio conversion/resampling then I think it could be a good change.

I was looking over the code changes and noticed there are a number of places where it looks like the code has been reformatted without any functional changes (e.g., TimelinePlayer::setExternalTime()). Would you mind undoing those changes so it is easier to see the actual changes? Thanks!

[ggarra13]

I'lll try to fix the reformatting. What are your setting for formatting c++ in vim/emacs/Astyle/VS or whatever you use? That may help me in not messing up your code (I work with emacs, btw).
I found out portaudio does not seem to do resampling per se but lets the audio card do it for you. There's a:
Pa_IsFormatSupported( NULL, &outputParameters, sampleRate )
that can be used to verify he sampleRate and other outputParameters are valid, but in my experience it always returns paNoError.

[darbyjohnston]

I don't have any settings for formatting though I tend to follow the VS style since it's the IDE I use most often. Four spaces for indentation, including function calls that need to be broken up over multiple lines, and I generally try to keep line lengths to 80 chars though that's not a strict rule. The most important thing is not to reformat code that doesn't have any functional changes, if code needs formatting fixes that is best done separately.

That's unfortunate PortAudio does not support resampling, it would have been a compelling reason to replace the existing RtAudio support.

[darbyjohnston]

Hi @ggarra13, sorry for the delay! Support for resampling would have been a compelling reason to switch to PortAudio, but without it I'm not sure this is a good change. Especially given the amount of testing the RtAudio support has already gone through. You had mentioned some troubles you initially had with RtAudio but I think you were able to resolve them?

[ggarra13]

I don't understand it. I searched the net about portaudio resampling and it said it didn't do it. However, in mrViewer I resample with it just fine on Linux and macOS (I did not try Windows, perhaps that's where it fails). The issue with RtAudio was indeed resolved with changing the cache backwards to 0.5 or higher. Maybe it is worth testing it with a simple sine wave program.

[darbyjohnston]

It looks like it's system dependent:
PortAudio/portaudio#484

That's a good idea for the sine wave, I've been wanting to add more test cases like that.

Maybe there is another area you would be interested in helping out with? Investigating whether libsamplerate could be used as a generic resampling solution would be interesting, it's actually already built as a tlRender dependency though it's not currently being used.

[ggarra13]

Ideally, I would like to help you out trying to get scrubbing with audio, but my first attempts at it resulted in core dumps :(. I am waiting to see RV, xStudio and itview source code to see whether I should join some of those projects instead. Or maybe do a new rewrite of mrViewer (which I need to do to clean up all the spaguetti code and bad design decisions which don't allow me to incorporate OpenTimelineIO easily).
I was also looking at mlt and the shotcut editing tool which has a perfect playback system in place with multiple tracks, effects and dissolves. The only thing missing is OCIO and flipbooks. That's another possibility for a new mrViewer.
But the truth is I am tired of coding and I am looking forward to going back to being a Lighting TD/Compositor for VFX/CG features.

[darbyjohnston]

If you are interested in pursuing audio scrubbing it might make sense to create a proof of concept first and then we could see how it fits into the main code. The current tlRender audio code works OK but is not very extensible. Adding audio scrubbing might need some architectural changes that would be easier to try out separately first.

Maybe another idea is to use tlRender as the playback engine for mrViewer? That way you could focus on the features that make mrViewer unique, maybe tailoring it to TDs and Compositors? I would imagine there are a lot of interesting use cases that could be developed by someone with VFX pipeline experience.

[ggarra13]

I can see adding audio scrubbing by doing one of three things:

1. Reading an otio::RationalTime of audio and storing a 1/24 (or whatever) of audio. Then pray the audio engine is fast enough to play each audio segment fast enough to not perceive the audio gaps. I think this would work but it demands too much of the audio engine and if you add resampling with libswresample we may loose the speed. This is how mrViewer works.
2. Reading an otio::TimeRange of audio for 1 second as it is now. However when playback is stopped we would play 1/24 of audio by copying the audio to a temporary buffer. My guess is that that is how RV does it. The trick is knowing where in the TimeRange of 1 second we are at any point in time for shortening it to a RationalTime or a very short TimeRange equivalent to 1/24 of a second.
3. Reading an otio::TimeRange of audio for several seconds as store it in the audio cache. When stopped, play from the cache, with the same issue of knowing where in the cache we are for the current frame.

[ggarra13]

I played with the idea of replacing the playback engine of mrViewer with tlRender and it is something I might revisit. The main issue is that I don't know how do the Qt event loop and the FLTK event loop would work for tlRender. This clashes also with my lack of familiarity with C++11/14 features such as std::future/std::promise. In FLTK I would require to add a timeout of 0.001 or so to keep polling for updates to the video.

[darbyjohnston]

Interesting, so in mrViewer what do you do when the video rate is not a multiple of the audio rate? Like if you have 29.97FPS video and 44kHz audio?

The core of tlRender doesn't depend on Qt, it just needs some kind of event loop to tick various objects and an OpenGL window to render into. The "play-glfw" example does this using GLFW, it should also be possible to integrate with FLTK.

If you do continue with mrViewer and C++ I highly recommend learning the new features. Smart pointers (std::shared_ptr, std::unique_ptr), built-in threading (std::future, std::promise), lambdas, and some other syntax changes have made things so much easier I can't imagine not having them. There is no way I could have written tlRender and the new version of DJV without them.

[ggarra13]

El 14 ago. 2022, a las 22:56, Darby Johnston ***@***.***> escribió: Interesting, so in mrViewer what do you do when the video rate is not a multiple of the audio rate? Like if you have 29.97FPS video and 44kHz audio?

I store the default playback speed for the video ( say 29.97 ) and the audio ( 44 kHZ ). For each video I have three threads (the decode thread which just stores packets, the video thread which decodes such packets and audio thread which decodes audio packets. I also have a subtitle thread for subtitles but that’s hardly ever used. Stopping and starting these threads is handled by a barrier. With FLTK, I call a timeout at the play FPS to see if the video image has been updated and if so I call the draw of the view window. The audio is opened in a separate thread at the default frequency and the packets of the audio are decompressed into frame chunks ( of say 8192 bytes for a stereo stream) and played continuously. With audio APIs that rely on callbacks like rtAudio or portaudio there’s always the chance of audio underrun, but I spent a lot of time streamlining the code for avoid that. When the user changes the frame rate (to play faster or slower) I calculate a new fps and a speed factor by dividing it with the default fps of the video. When the current audio frequency does not match the previous frequency I reopen the audio with the new frequency (ie. Multiplied by the speed factor) and keep playing it at that speed in the audio thread as before. When I stop the playback and go step by step or do scrubbing I play the audio frame chunk for that frame which was stored in the cache. I also keep updating the audio packets so that once the cache does not have the audio frame chunk anymore I will call the audio decompress routine once again. The problem with my design of decompressing the packets into different threads is that it breaks when OpenTimelineIO enters the picture, as the barrier syncing all threads now does not work (the audio can stop at any time, not only at the end of the video). That’s where your design is superior.

If you do continue with mrViewer and C++ I highly recommend learning the new features. Smart pointers (std::shared_ptr, std::unique_ptr), built-in threading (std::future, std::promise), lambdas, and some other syntax changes have made things so much easier I can't imagine not having them. There is no way I could have written tlRender and the new version of DJV without them.

I am not sure yet what I’ll do with mrViewer. I recently bought Scott Meyer's Effective Modern C++ and I am reading it at night to catch up. OH! So you are writing a new version of DJV? I thought tlplay was it. — Gonzalo Garramuno ***@***.***

[darbyjohnston]

Just to follow up on this, maybe we should close this PR since you created a new issue for audio scrubbing:
#44

So you are writing a new version of DJV? I thought tlplay was it.

Sorry, I meant the current 2.x versions of DJV as opposed to the previous 1.x versions.

I think of tlplay as the reference application for the tlRender libraries. It demonstrates and exercises all of the functionality in tlRender but not necessarily in the most user friendly way.