VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers. With an intuitive Streamlit interface, you can transform a video link into a localized video with high-quality bilingual subtitles and dubbing in just a few clicks.
Key features:
-
🎥 YouTube video download via yt-dlp
-
🎙️ Word-level subtitle recognition with WhisperX
-
📝 NLP and GPT-based subtitle segmentation
-
📚 GPT-generated terminology for coherent translation
-
🔄 3-step direct translation, reflection, and adaptation for professional-level quality
-
✅ Netflix-standard single-line subtitles only
-
🗣️ Dubbing alignment with GPT-SoVITS and other methods
-
🚀 One-click startup and output in Streamlit
-
📝 Detailed logging with progress resumption
-
🌐 Comprehensive multi-language support
Difference from similar projects: Single-line subtitles only, superior translation quality
ru_demo.mp4 |
sovits.mp4 |
OAITTS.mp4 |
Current input language support and examples:
| Input Language | Support Level | Translation Demo |
|---|---|---|
| English | 🤩 | English to Chinese |
| Russian | 😊 | Russian to Chinese |
| French | 🤩 | French to Japanese |
| German | 🤩 | German to Chinese |
| Italian | 🤩 | Italian to Chinese |
| Spanish | 🤩 | Spanish to Chinese |
| Japanese | 😐 | Japanese to Chinese |
| Chinese* | 🤩 | Chinese to English |
*Chinese requires separate configuration of the whisperX model, only applicable for local source code installation. See the installation documentation for the configuration process, and be sure to specify the transcription language as zh in the webpage sidebar
Translation language support depends on the capabilities of the large language model used, while dubbing language depends on the chosen TTS method.
Experience VideoLingo quickly in Colab in just 5 minutes:
VideoLingo supports all hardware platforms and operating systems, but performs best with GPU acceleration. For detailed installation instructions , please refer to the documentation: English | 简体中文
VideoLingo provides a Dockerfile. Please refer to the installation documentation: English | 简体中文
Usage instructions: English | 简体中文
-
WhisperX performance varies across different devices. Version 1.7 performs demucs voice separation first, but this may result in worse transcription after separation compared to before. This is because whisper itself was trained in environments with background music - before separation it won't transcribe BGM lyrics, but after separation it might transcribe them.
-
The dubbing feature quality may not be perfect as it's still in testing and development stage, with plans to integrate MascGCT. For best results currently, it's recommended to choose TTS with similar speech rates based on the original video's speed and content characteristics. See the demo for effects.
-
Multilingual video transcription recognition will only retain the main language. This is because whisperX uses a specialized model for a single language when forcibly aligning word-level subtitles, and will delete unrecognized languages.
-
Multi-character separate dubbing is under development. While whisperX has VAD potential, specific implementation work is needed, and this feature is not yet supported.
- VAD to distinguish speakers, multi-character dubbing
- Customizable translation styles
- SaaS service
- Lip sync for dubbed videos
This project is licensed under the Apache 2.0 License. When using this project, please follow these rules:
- When publishing works, it is recommended (not mandatory) to credit VideoLingo for subtitle generation.
- Follow the terms of the large language models and TTS used for proper attribution.
- If you copy the code, please include the full copy of the Apache 2.0 License.
We sincerely thank the following open-source projects for their contributions, which provided important support for the development of VideoLingo:
- Join our Discord: https://discord.gg/9F2G92CWPp
- Submit Issues or Pull Requests on GitHub
- Follow me on Twitter: @Huanshere
- Visit the official website: docs.videolingo.io
- Email me at: [email protected]
If you find VideoLingo helpful, please give us a ⭐️!
