Transcribe (whisper) and translate (gpt) voice into LRC file.
Project description
Open-Lyrics
Open-Lyrics is a Python library that transcribes voice files using
faster-whisper, and translates/polishes the resulting text
into .lrc
files in the desired language using OpenAI-GPT.
Installation
-
Please install CUDA and cuDNN first according to https://opennmt.net/CTranslate2/installation.html to enable
faster-whisper
. -
Add your OpenAI API key to environment variable
OPENAI_API_KEY
. -
Install PyTorch:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
-
Install whisperx and (Recommended) the latest fast-whisper
pip install git+https://github.com/guillaumekln/faster-whisper pip install git+https://github.com/m-bain/whisperx.git
-
(Optional) If you want to process videos, install ffmpeg and add
bin
directory to yourPATH
. -
This project can be installed from PyPI:
pip install openlrc
or install directly from GitHub:
pip install git+https://github.com/zh-plus/Open-Lyrics
-
Go to Spacy to install the required package using
spacy download xxx
. For example, you needspacy download ja_core_news_sm
if the source language is japanese.
Usage
from openlrc import LRCer
lrcer = LRCer()
# Single file
lrcer.run('./data/test.mp3', target_lang='zh-cn') # Generate translated ./data/test.lrc with default translate prompt.
# Multiple files
lrcer.run(['./data/test1.mp3', './data/test2.mp3'], target_lang='zh-cn')
# Note we run the transcription sequentially, but run the translation concurrently for each file.
# Path can contain video
lrcer.run(['./data/test_audio.mp3', './data/test_video.mp4'], target_lang='zh-cn')
# Generate translated ./data/test_audio.lrc and ./data/test_video.srt
# Use context.yaml to improve translation
lrcer.run('./data/test.mp3', target_lang='zh-cn', context_path='./data/context.yaml')
# To skip translation process
lrcer.run('./data/test.mp3', target_lang='en', skip_trans=True)
Context
Utilize the available context to enhance the quality of your translation.
Save them as context.yaml
in the same directory as your audio file.
background: "This is a multi-line background.
This is a basic example."
audio_type: Movie
synopsis_map: {
movie_name1 (without extension): "This
is a multi-line synopsis for movie1.",
movie_name2 (without extension): "This
is a multi-line synopsis for movie2.",
movie_name3 (without extension): "This is a single-line synopsis for movie 3.",
}
Todo
- [Efficiency] Batched translate/polish for GPT request (enable contextual ability).
- [Efficiency] Concurrent support for GPT request.
- [Efficiency & Transcription Quality] Use whisperx for transcription.
- [Translation Quality] Make translate prompt more robust according to https://github.com/openai/openai-cookbook.
- [Feature] Automatically fix json encoder error using GPT.
- [Efficiency] Asynchronously perform transcription and translation for multiple audio inputs.
- [Quality] Improve batched translation/polish prompt according to gpt-subtrans.
- [Feature] Input video support.
- [Feature] Multiple output format support.
- [Feature] Align ground-truth transcription with audio.
- [Quality] Use multilingual language model to assess translation quality.
- [Quality] Speech enhancement for input audio.
- [Efficiency] Add Azure OpenAI Service support.
- [Quality] Use claude for translation.
- [Feature] Add local LLM support.
- [Feature] Multiple translate engine (Microsoft, DeepL, Google, etc.) support.
- [Feature] Build a electron + fastapi GUI for cross-platform application.
- Add fine-tuned whisper-large-v2 models for common languages.
- [Others] Add transcribed examples.
- Song
- Podcast
- Audiobook
Credits
- https://github.com/guillaumekln/faster-whisper
- https://github.com/m-bain/whisperX
- https://github.com/openai/openai-python
- https://github.com/openai/whisper
- https://github.com/machinewrapped/gpt-subtrans
- https://github.com/MicrosoftTranslator/Text-Translation-API-V3-Python
Star History
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.