VITS - TTS 0.22.0 documentation (2024)

VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) is an End-to-End (encoder -> vocoder together) TTS model that takes advantage of SOTA DL techniques like GANs, VAE,Normalizing Flows. It does not require external alignment annotations and learns the text-to-audio alignmentusing MAS, as explained in the paper. The model architecture is a combination of GlowTTS encoder and HiFiGAN vocoder.It is a feed-forward model with x67.12 real-time factor on a GPU.

🐸 YourTTS is a multi-speaker and multi-lingual TTS model that can perform voice conversion and zero-shot speaker adaptation.It can also learn a new language or voice with a ~ 1 minute long audio clip. This is a big open gate for trainingTTS models in low-resources languages. 🐸 YourTTS uses VITS as the backbone architecture coupled with a speaker encoder model.

Important resources & papers#

🐸 YourTTS: https://arxiv.org/abs/2112.02418
VITS: https://arxiv.org/pdf/2106.06103.pdf
Neural Spline Flows: https://arxiv.org/abs/1906.04032
Variational Autoencoder: https://arxiv.org/pdf/1312.6114.pdf
Generative Adversarial Networks: https://arxiv.org/abs/1406.2661
HiFiGAN: https://arxiv.org/abs/2010.05646
Normalizing Flows: https://blog.evjang.com/2018/01/nf1.html

VitsConfig#

class TTS.tts.configs.vits_config.VitsConfig(output_path='output', logger_uri=None, run_name='run', project_name=None, run_description='🐸Coqui trainer run.', print_step=25, plot_step=100, model_param_stats=False, wandb_entity=None, dashboard_logger='tensorboard', save_on_interrupt=True, log_model_step=None, save_step=10000, save_n_checkpoints=5, save_checkpoints=True, save_all_best=False, save_best_after=10000, target_loss=None, print_eval=False, test_delay_epochs=0, run_eval=True, run_eval_steps=None, distributed_backend='nccl', distributed_url='tcp://localhost:54321', mixed_precision=False, precision='fp16', epochs=1000, batch_size=32, eval_batch_size=16, grad_clip=<factory>, scheduler_after_epoch=True, lr=0.001, optimizer='AdamW', optimizer_params=<factory>, lr_scheduler=None, lr_scheduler_params=<factory>, use_grad_scaler=False, allow_tf32=False, cudnn_enable=True, cudnn_deterministic=False, cudnn_benchmark=False, training_seed=54321, model='vits', num_loader_workers=0, num_eval_loader_workers=0, use_noise_augment=False, audio=<factory>, use_phonemes=False, phonemizer=None, phoneme_language=None, compute_input_seq_cache=False, text_cleaner=None, enable_eos_bos_chars=False, test_sentences_file='', phoneme_cache_path=None, characters=None, add_blank=True, batch_group_size=0, loss_masking=None, min_audio_len=1, max_audio_len=inf, min_text_len=1, max_text_len=inf, compute_f0=False, compute_energy=False, compute_linear_spec=True, precompute_num_workers=0, start_by_longest=False, shuffle=False, drop_last=False, datasets=<factory>, test_sentences=<factory>, eval_split_max_size=None, eval_split_size=0.01, use_speaker_weighted_sampler=False, speaker_weighted_sampler_alpha=1.0, use_language_weighted_sampler=False, language_weighted_sampler_alpha=1.0, use_length_weighted_sampler=False, length_weighted_sampler_alpha=1.0, model_args=<factory>, lr_gen=0.0002, lr_disc=0.0002, lr_scheduler_gen='ExponentialLR', lr_scheduler_gen_params=<factory>, lr_scheduler_disc='ExponentialLR', lr_scheduler_disc_params=<factory>, kl_loss_alpha=1.0, disc_loss_alpha=1.0, gen_loss_alpha=1.0, feat_loss_alpha=1.0, mel_loss_alpha=45.0, dur_loss_alpha=1.0, speaker_encoder_loss_alpha=1.0, return_wav=True, use_weighted_sampler=False, weighted_sampler_attrs=<factory>, weighted_sampler_multipliers=<factory>, r=1, num_speakers=0, use_speaker_embedding=False, speakers_file=None, speaker_embedding_channels=256, language_ids_file=None, use_language_embedding=False, use_d_vector_file=False, d_vector_file=None, d_vector_dim=None)[source]#

Defines parameters for VITS End2End TTS model.

Parameters:

model (str) – Model name. Do not change unless you know what you are doing.
model_args (VitsArgs) – Model architecture arguments. Defaults to VitsArgs().
audio (VitsAudioConfig) – Audio processing configuration. Defaults to VitsAudioConfig().
grad_clip (List) – Gradient clipping thresholds for each optimizer. Defaults to [1000.0, 1000.0].
lr_gen (float) – Initial learning rate for the generator. Defaults to 0.0002.
lr_disc (float) – Initial learning rate for the discriminator. Defaults to 0.0002.
lr_scheduler_gen (str) – Name of the learning rate scheduler for the generator. One of the torch.optim.lr_scheduler.*. Defaults toExponentialLR.
lr_scheduler_gen_params (dict) – Parameters for the learning rate scheduler of the generator. Defaults to {‘gamma’: 0.999875, “last_epoch”:-1}.
lr_scheduler_disc (str) – Name of the learning rate scheduler for the discriminator. One of the torch.optim.lr_scheduler.*. Defaults toExponentialLR.
lr_scheduler_disc_params (dict) – Parameters for the learning rate scheduler of the discriminator. Defaults to {‘gamma’: 0.999875, “last_epoch”:-1}.
scheduler_after_epoch (bool) – If true, step the schedulers after each epoch else after each step. Defaults to False.
optimizer (str) – Name of the optimizer to use with both the generator and the discriminator networks. One of thetorch.optim.*. Defaults to AdamW.
kl_loss_alpha (float) – Loss weight for KL loss. Defaults to 1.0.
disc_loss_alpha (float) – Loss weight for the discriminator loss. Defaults to 1.0.
gen_loss_alpha (float) – Loss weight for the generator loss. Defaults to 1.0.
feat_loss_alpha (float) – Loss weight for the feature matching loss. Defaults to 1.0.
mel_loss_alpha (float) – Loss weight for the mel loss. Defaults to 45.0.
return_wav (bool) – If true, data loader returns the waveform as well as the other outputs. Do not change. Defaults to True.
compute_linear_spec (bool) – If true, the linear spectrogram is computed and returned alongside the mel output. Do not change. Defaults to True.
use_weighted_sampler (bool) – If true, use weighted sampler with bucketing for balancing samples between datasets used in training. Defaults to False.
weighted_sampler_attrs (dict) – Key retuned by the formatter to be used for weighted sampler. For example {“root_path”: 2.0, “speaker_name”: 1.0} sets sample probabilitiesby overweighting root_path by 2.0. Defaults to {}.
weighted_sampler_multipliers (dict) – Weight each unique value of a key returned by the formatter for weighted sampling.For example {“root_path”:{“/raid/datasets/libritts-clean-16khz-bwe-coqui_44khz/LibriTTS/train-clean-100/”:1.0, “/raid/datasets/libritts-clean-16khz-bwe-coqui_44khz/LibriTTS/train-clean-360/”: 0.5}.It will sample instances from train-clean-100 2 times more than train-clean-360. Defaults to {}.
r (int) – Number of spectrogram frames to be generated at a time. Do not change. Defaults to 1.
add_blank (bool) – If true, a blank token is added in between every character. Defaults to True.
test_sentences (List[List]) – List of sentences with speaker and language information to be used for testing.
language_ids_file (str) – Path to the language ids file.
use_language_embedding (bool) – If true, language embedding is used. Defaults to False.

Note

Check TTS.tts.configs.shared_configs.BaseTTSConfig for the inherited parameters.

Example

>>> from TTS.tts.configs.vits_config import VitsConfig>>> config = VitsConfig()

VitsArgs#

class TTS.tts.models.vits.VitsArgs(num_chars=100, out_channels=513, spec_segment_size=32, hidden_channels=192, hidden_channels_ffn_text_encoder=768, num_heads_text_encoder=2, num_layers_text_encoder=6, kernel_size_text_encoder=3, dropout_p_text_encoder=0.1, dropout_p_duration_predictor=0.5, kernel_size_posterior_encoder=5, dilation_rate_posterior_encoder=1, num_layers_posterior_encoder=16, kernel_size_flow=5, dilation_rate_flow=1, num_layers_flow=4, resblock_type_decoder='1', resblock_kernel_sizes_decoder=<factory>, resblock_dilation_sizes_decoder=<factory>, upsample_rates_decoder=<factory>, upsample_initial_channel_decoder=512, upsample_kernel_sizes_decoder=<factory>, periods_multi_period_discriminator=<factory>, use_sdp=True, noise_scale=1.0, inference_noise_scale=0.667, length_scale=1, noise_scale_dp=1.0, inference_noise_scale_dp=1.0, max_inference_len=None, init_discriminator=True, use_spectral_norm_disriminator=False, use_speaker_embedding=False, num_speakers=0, speakers_file=None, d_vector_file=None, speaker_embedding_channels=256, use_d_vector_file=False, d_vector_dim=0, detach_dp_input=True, use_language_embedding=False, embedded_language_dim=4, num_languages=0, language_ids_file=None, use_speaker_encoder_as_loss=False, speaker_encoder_config_path='', speaker_encoder_model_path='', condition_dp_on_speaker=True, freeze_encoder=False, freeze_DP=False, freeze_PE=False, freeze_flow_decoder=False, freeze_waveform_decoder=False, encoder_sample_rate=None, interpolate_z=True, reinit_DP=False, reinit_text_encoder=False)[source]#

VITS model arguments.

Parameters:

num_chars (int) – Number of characters in the vocabulary. Defaults to 100.
out_channels (int) – Number of output channels of the decoder. Defaults to 513.
spec_segment_size (int) – Decoder input segment size. Defaults to 32 (32 * hoplength = waveform length).
hidden_channels (int) – Number of hidden channels of the model. Defaults to 192.
hidden_channels_ffn_text_encoder (int) – Number of hidden channels of the feed-forward layers of the text encoder transformer. Defaults to 256.
num_heads_text_encoder (int) – Number of attention heads of the text encoder transformer. Defaults to 2.
num_layers_text_encoder (int) – Number of transformer layers in the text encoder. Defaults to 6.
kernel_size_text_encoder (int) – Kernel size of the text encoder transformer FFN layers. Defaults to 3.
dropout_p_text_encoder (float) – Dropout rate of the text encoder. Defaults to 0.1.
See Also
Explicit Abnormality Extraction for Unsupervised Motion Artifact Reduction in Magnetic Resonance Imaging Multi-Omic Integration of Blood-Based Tumor-Associated Genomic and Lipidomic Profiles Using Machine Learning Models in Metastatic Prostate Cancer Causal effects of omega‐6 and LDL‐C on androgenetic alopecia: A Mendelian randomization study US Patent Application for METHODS AND COMPOSITIONS FOR MODULATING STING SIGNALING AND INNATE IMMUNE RESPONSES Patent Application (Application #20240271138 issued August 15, 2024)
dropout_p_duration_predictor (float) – Dropout rate of the duration predictor. Defaults to 0.1.
kernel_size_posterior_encoder (int) – Kernel size of the posterior encoder’s WaveNet layers. Defaults to 5.
dilatation_posterior_encoder (int) – Dilation rate of the posterior encoder’s WaveNet layers. Defaults to 1.
num_layers_posterior_encoder (int) – Number of posterior encoder’s WaveNet layers. Defaults to 16.
kernel_size_flow (int) – Kernel size of the Residual Coupling layers of the flow network. Defaults to 5.
dilatation_flow (int) – Dilation rate of the Residual Coupling WaveNet layers of the flow network. Defaults to 1.
num_layers_flow (int) – Number of Residual Coupling WaveNet layers of the flow network. Defaults to 6.
resblock_type_decoder (str) – Type of the residual block in the decoder network. Defaults to “1”.
resblock_kernel_sizes_decoder (List[int]) – Kernel sizes of the residual blocks in the decoder network. Defaults to [3, 7, 11].
resblock_dilation_sizes_decoder (List[List[int]]) – Dilation sizes of the residual blocks in the decoder network. Defaults to [[1, 3, 5], [1, 3, 5], [1, 3, 5]].
upsample_rates_decoder (List[int]) – Upsampling rates for each concecutive upsampling layer in the decoder network. The multiply of thesevalues must be equal to the kop length used for computing spectrograms. Defaults to [8, 8, 2, 2].
upsample_initial_channel_decoder (int) – Number of hidden channels of the first upsampling convolution layer of the decoder network. Defaults to 512.
upsample_kernel_sizes_decoder (List[int]) – Kernel sizes for each upsampling layer of the decoder network. Defaults to [16, 16, 4, 4].
periods_multi_period_discriminator (List[int]) – Periods values for Vits Multi-Period Discriminator. Defaults to [2, 3, 5, 7, 11].
use_sdp (bool) – Use Stochastic Duration Predictor. Defaults to True.
noise_scale (float) – Noise scale used for the sample noise tensor in training. Defaults to 1.0.
inference_noise_scale (float) – Noise scale used for the sample noise tensor in inference. Defaults to 0.667.
length_scale (float) – Scale factor for the predicted duration values. Smaller values result faster speech. Defaults to 1.
noise_scale_dp (float) – Noise scale used by the Stochastic Duration Predictor sample noise in training. Defaults to 1.0.
inference_noise_scale_dp (float) – Noise scale for the Stochastic Duration Predictor in inference. Defaults to 0.8.
max_inference_len (int) – Maximum inference length to limit the memory use. Defaults to None.
init_discriminator (bool) – Initialize the disciminator network if set True. Set False for inference. Defaults to True.
use_spectral_norm_disriminator (bool) – Use spectral normalization over weight norm in the discriminator. Defaults to False.
use_speaker_embedding (bool) – Enable/Disable speaker embedding for multi-speaker models. Defaults to False.
num_speakers (int) – Number of speakers for the speaker embedding layer. Defaults to 0.
speakers_file (str) – Path to the speaker mapping file for the Speaker Manager. Defaults to None.
speaker_embedding_channels (int) – Number of speaker embedding channels. Defaults to 256.
use_d_vector_file (bool) – Enable/Disable the use of d-vectors for multi-speaker training. Defaults to False.
d_vector_file (List[str]) – List of paths to the files including pre-computed speaker embeddings. Defaults to None.
d_vector_dim (int) – Number of d-vector channels. Defaults to 0.
detach_dp_input (bool) – Detach duration predictor’s input from the network for stopping the gradients. Defaults to True.
use_language_embedding (bool) – Enable/Disable language embedding for multilingual models. Defaults to False.
embedded_language_dim (int) – Number of language embedding channels. Defaults to 4.
num_languages (int) – Number of languages for the language embedding layer. Defaults to 0.
language_ids_file (str) – Path to the language mapping file for the Language Manager. Defaults to None.
use_speaker_encoder_as_loss (bool) – Enable/Disable Speaker Consistency Loss (SCL). Defaults to False.
speaker_encoder_config_path (str) – Path to the file speaker encoder config file, to use for SCL. Defaults to “”.
speaker_encoder_model_path (str) – Path to the file speaker encoder checkpoint file, to use for SCL. Defaults to “”.
condition_dp_on_speaker (bool) – Condition the duration predictor on the speaker embedding. Defaults to True.
freeze_encoder (bool) – Freeze the encoder weigths during training. Defaults to False.
freeze_DP (bool) – Freeze the duration predictor weigths during training. Defaults to False.
freeze_PE (bool) – Freeze the posterior encoder weigths during training. Defaults to False.
freeze_flow_encoder (bool) – Freeze the flow encoder weigths during training. Defaults to False.
freeze_waveform_decoder (bool) – Freeze the waveform decoder weigths during training. Defaults to False.
encoder_sample_rate (int) – If not None this sample rate will be used for training the Posterior Encoder,flow, text_encoder and duration predictor. The decoder part (vocoder) will betrained with the config.audio.sample_rate. Defaults to None.
interpolate_z (bool) – If encoder_sample_rate not None and this parameter True the nearest interpolationwill be used to upsampling the latent variable z with the sampling rate encoder_sample_rateto the config.audio.sample_rate. If it is False you will need to add extraupsample_rates_decoder to match the shape. Defaults to True.

Vits Model#

class TTS.tts.models.vits.Vits(config, ap=None, tokenizer=None, speaker_manager=None, language_manager=None)[source]#

VITS TTS model

Paper::: https://arxiv.org/pdf/2106.06103.pdf
Paper Abstract::: Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallelsampling have been proposed, but their sample quality does not match that of two-stage TTS systems.In this work, we present a parallel endto-end TTS method that generates more natural sounding audio thancurrent two-stage models. Our method adopts variational inference augmented with normalizing flows andan adversarial training process, which improves the expressive power of generative modeling. We also propose astochastic duration predictor to synthesize speech with diverse rhythms from input text. With theuncertainty modeling over latent variables and the stochastic duration predictor, our method expresses thenatural one-to-many relationship in which a text input can be spoken in multiple wayswith different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS)on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publiclyavailable TTS systems and achieves a MOS comparable to ground truth.

Check TTS.tts.configs.vits_config.VitsConfig for class arguments.

Examples

>>> from TTS.tts.configs.vits_config import VitsConfig>>> from TTS.tts.models.vits import Vits>>> config = VitsConfig()>>> model = Vits(config)

VITS - TTS 0.22.0 documentation (2024)

Important resources & papers#

VitsConfig#

VitsArgs#

Vits Model#

References