tungnaa 0.1.1

tungnaa is a text-to-voice model family and musical instrument

Tungnaa Interactive Voice Instruments

Training and GUI inference for interactive artistic text-to-voice models.

Installation

pip install tungnaa[gui] (to use the instrument)

pip install tungnaa[train] (if you are installing on a server to train models)

Usage

tungnaa --help

Models from Huggingface

git clone [e-mail]:intelligent-instruments-Lab/tungnaa

Running with the Python Audio Engine

• #todo model selection from the Tungnaa gui, including block size and sample rate that match system (currently not possible to have a SR mismatch)
• #todo audio device selection from the Tungnaa gui

tungnaa run --tts models/tts/rtalign_044_jvs.ckpt --vocoder models/vocoder/rave3-jvs-warm200k-lobeta_c052f53b23_streaming.ts --audio-out default

Using SuperCollider, PureData or Max as Audio Engine

If --latent-audio switch is enabled, Tungnaa will stream RAVE latent trajectories over a single audio-rate channel, which can be piped into another audio engine running the RAVE vocoder. The piping can be done relatively easily on Linux using JACK, and on MacOS using Blackhole.

SuperCollider: sclang supercollider/rtvoice-demo.scd

tungnaa run --tts models/tts/rtalign_044_jvs.ckpt --latent_audio

Training Models

vocoder training

using victor-shepardson RAVE fork

example preprocessing with joining of short files (especially useful for datasets containing many short utterances)

rave preprocess \
--input_path /path/to/audio/directory \
--output_path /path/to/tmp/storage/myravedata \
--num_signal 150000 --sampling_rate 48000 \
--join_short_files

example transfer learning using IIL rave-models:

rave train --name 001-my-vocoder-name \
--config rave-models/voice_multi_b2048_r48000/config.gin --config transfer \
--db_path /path/to/tmp/storage/myravedata \
--out_path /path/to/rave/runs \
--transfer_ckpt rave-models/voice_multi_b2048_r48000/version_0/checkpoints/last.ckpt \
--n_signal 150000
--gpu 0

example export using sign normalization (latents correlate with louder/brighter sounds):

rave export --run /path/to/rave/runs/001-my-vocoder-name \
--streaming --normalize_sign --latent_size ...

Tungnaá preprocessing

see tungnaa prep --help.

To use datasets other than vctk or hifitts, it may be necessary to add an adapter function in prep.py.

example:

tungnaa prep \
--datasets '{kind:"vctk", path:"/path/to/VCTK"}' \
--rave-path /path/to/rave_streaming.ts \
--out-path /path/to/tmp/dataset_name

training

see tungnaa trainer --help

example:

tungnaa trainer --experiment 001-my-tts-name \
--model-dir /path/for/checkpoints \
--log-dir /path/for/logs \
--manifest /path/to/tmp/dataset_name/manifest.json \
--rave-model /path/to/rave_streaming.ts 
--lr 3e-4 --lr-text 3e-5 --epoch-size 200 --save-epochs 20 \
--device cuda:0 \
train 

resume a stopped training: add --checkpoint /path/to/checkpoint

transfer learning: add --checkpoint /path/to/checkpoint --resume False

in-text annotations

--speaker_annotate prepends the speaker id determined during prepreprocessing. With --speaker_dataset, it includes the dataset.

--csv /path/to/file.csv accepts a jvs_labels_encoder_k7.csv-style CSV file. First column contains the audio filename without extension, second column contains an annotation to be prepended to text.

If you were to use all three options, you would get:

"csvval:[dataset:speaker] original text"

Developing

Poetry is used for packaging and dependency management. Conda is used for environments and Python version management, and may be replaced by virtualenv or similar.

1. cd tungnaa
2. conda create -n tungnaa python=3.12 ffmpeg
3. conda activate tungnaa
4. poetry install

Note that poetry should not be installed in the project environment, but rather from the system package manager, with pipx, or in a separate environment.

To add a dependency, use poetry add, or edit pyproject.toml and then run poetry lock; poetry install.

To add a model, use dvc add /path/to/model, then git add /path/to/model.dvc. Tungnaá models should go in models/tts/, and be accompanied by a model.md file. Vocoders should go in models/vocoders.

docs

run mkdocs serve to build and view documentation

run mkdocs gh-deploy to deploy to github pages

Non-Python Dependencies

• Poetry (https://python-poetry.org/) for packaging

Python Dependencies

See pyproject.toml