Use Pre-Recorded Tracks as input/change active track

[edrihan]

It's really great how you can record in the command line, however there are some issues with that. Firstly, the transcription happens per-file which completely breaks the flow. Why can't it batch them once you're done recording? But that is neither here nor there.. as recording in the cli is just not what we are used to in the audio field. There are free (to use) programs like Reaper and Audacity, and paid DAWs which are enjoyed by people doing recordings.

So people have their own workflow for recording audio... and it's not that. You want control over what happens in your audio.

Description of proposed feature

I want to be able to swap audio files easily. I want to be able to pass an audio file, relative to a folder in manim/media/, for example:

• manim/media/voiceovers
• manim/media/recordings
• manim/media/audio

I think the best way for this to work would probably be to have a small json file which has the section name corresponding with the file path. That way, a check of that file can be performed to determine what needs to be done for that section. Then if I update a section later, I want to be able to change manim links to the new file, and have the manim command automatically create transcription and metadata for it.

If a speech file doesn't exist, I want to be prompted whether I want to record it, or whether I want to supply a path. If I choose a path outside the manim directory, I want the option to copy the file into the right place. Maybe it adds the hash to the end of the filename too?

Now it can rebuild the data in manim/media/voiceovers/cache.json where it needs to

How can the new feature be used?

Ideally you just have the section text, as the key, and the filepath as the value.

Additional comments

The other feature I want is related to this, but I don't think it deserves its own issue as for the moment, using your own audio seems difficult, but if this feature request goes through, I would love to have the audio not be hardcoded to mp3. Normally in audio you would use WAV, FLAC, or equivalent lossless codecs. We can let the internet destroy the audio later, but on the master copy you don't use mp3s. I know that it made sense to use mp3s as that is probably what the ai models put out. But for our own audio we do not want lossy compression.

This will be more in line with normal audio workflows and will make this so much easier to use. Then we'll have more of peoples' voices and less of AI, which is a good thing.

Issue #88 seems related

[kennykb]

It's in no fit shape to package as a module in manim_voiceover, but here's what I've been using.

It doesn't do WAV, FLAC or Ogg Vorbis - I wish it did, but haven't had the time to dive into how the audio handling works, so I've stuck to MP3.

# This is copied and pasted from my script that I copy and paste
# into all my animations, so I may have dropped an import or  three
from manim import *
from manim_voiceover import VoiceoverScene
from manim_voiceover.helper import msg_box, remove_bookmarks
from manim_voiceover.services.base import SpeechService
from tkinter import Tk, filedialog

class PrerecordedService(SpeechService):

    file_types = (
        ('MP3 Audio', '.mp3'),
        ('All Files', '*')
    )

    def __init__(
            self,
            transcription_model: str = "base",
            **kwargs):

        SpeechService.__init__(self,
                               transcription_model=transcription_model,
                               **kwargs)

    def generate_from_text(
            self, text:str, cache_dir:str = None, path: str=None, **kwargs
    ) -> dict:
        """"""

        # Remove bookmarks
        input_text = remove_bookmarks(text)

        if cache_dir is None:
            cache_dir = self.cache_dir

        input_data = {
            "input_text" : input_text,
            "service" : "prerecorded",
            }

        cached_result = self.get_cached_result(input_data, cache_dir)
        if cached_result is not None:
            return cached_result
        
        while path is None:
            box = msg_box("Voiceover:\n\n" + input_text)
            print(box)
            print('Please select an audio file for the text above.')
            path = filedialog.askopenfilename(
                title='Please select audio file',
                filetypes = self.file_types,
                )
            if path == ():
                raise RuntimeError('Operation cancelled.')
        print(f'Selected: {path}')

        json_dict = {
            "input_text" : text,
            "input_data" : input_data,
            "original_audio" : path,
        }

        return json_dict

and then I can say

speechService = PrerecordedService(transcription_model='small')

in place of GTTSService or RecorderService or whatever.

[h4ckt1v1st]

I like this. One of the reasons I want this is to sound less robotic in my delivery. Right now it's basically reading from a script. I want to record my natural tone first and then sync it using voiceover plugin.

[kennykb]

@h4ckt1v1st - Exactly. Also, I dislike the sound of lots of little 'self.voiceover' calls - it mutes the audio completely in between them, which sounds choppy. I can do my voiceovers with much longer sections, using <bookmark>. but with RecorderService, I then have to get that section in a single take. Moreover, I can preprocess the audio with Audacity, to add speech compression, do noise removal, add loudness normalisation, mute or snip foreign sounds (like the phone ringing or the cat meowing), or whatever I need to do for clean audio.

The code I posted above works well enough on my machine that I haven't really chased any improvements. I've used it with a bunch of videos now. Does it work for you, too?

[h4ckt1v1st]

@kennykb The issue I see with this approach is that you have to kinda split or audio in segments manually. Don't you? If not somehow, that would be an awesome workflow.

I am thinking if we can use openai whisper to basically convert the audio into transcription, not sure if that can help? Maybe in splitting the audio automatically in smaller segments? I will have to sit with it and figure it out.

Also @osolmaz seems inactive lately. PRs solving issues are kept pending without review.

[kennykb]

@h4ckt1v1st You have Whisper embedded in manim-voiceover already!

Have you tried using the 'bookmark' functionality? If I have a continuous narration, I can record it in one go (or splice it together in Audacity from as many takes as I need).

The attached file voiceover_example.py is a fairly complete example of a short script that has multiple actions on screen synchronized with it. Try running it with GTTSService (line 70 uncommented) and then again with PrerecordedService (line 72 uncommented) with your own voice reading the script. (Rename the file to .py - Github doesn't allow uploading Python source in comments)

If you'd rather use speech-to-text, then use Whisper (or whatever) to transcribe what you said, clean up the mistranscriptions, insert the <bookmark> markers where they need to go, and then do the animation.

An actual example is in my recent video here - the Manim segments start at 3:00 and 5:18. I don't think I sound too robotic, but I'm no actor!

voiceover-example.py.txt