5.6k sao GitHub và vẫn tăng — huggingface/parler-tts là dự án Python mà TopGit đang theo dõi trên nền tảng. Inference and training library for high-quality TTS models.
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
Parler-TTS is a lightweight text-to-speech (TTS) model that can generate high-quality, natural sounding speech in the style of a given speaker (gender, pitch, speaking style, etc). It is a reproduction of work from the paper Natural language guidance of high-fidelity text-to-speech with synthetic annotations by Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively.
Contrarily to other TTS models, Parler-TTS is a fully open-source release. All of the datasets, pre-processing, training code and weights are released publicly under permissive license, enabling the community to build on our work and develop their own powerful TTS models.
This repository contains the inference and training code for Parler-TTS. It is designed to accompany the Data-Speech repository for dataset annotation.
[!IMPORTANT]
08/08/2024: We are proud to release two new Parler-TTS checkpoints:
Parler-TTS Mini, an 880M parameter model.
Parler-TTS Large, a 2.3B parameter model.
These checkpoints have been trained on 45k hours of audiobook data.
In addition, the code is optimized for much faster generation: we've added SDPA and Flash Attention 2 compatibility, as well as the ability to compile the model.
📖 Quick Index
Installation
Usage
🎲 Using a random voice
🎯 Using a specific speaker
Training
Demo
Model weights and datasets
Optimizing inference
Installation
Parler-TTS has light-weight dependencies and can be installed in one line:
Parler-TTS has been trained to generate speech with features that can be controlled with a simple text prompt, for example:
import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-v1").to(device)
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-v1")
prompt = "Hey, how are you doing today?"
description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)
🎯 Using a specific speaker
To ensure speaker consistency across generations, this checkpoint was also trained on 34 speakers, characterized by name. The full list of available speakers includes:
Laura, Gary, Jon, Lea, Karen, Rick, Brenda, David, Eileen, Jordan, Mike, Yann, Joy, James, Eric, Lauren, Rose, Will, Jason, Aaron, Naomie, Alisa, Patrick, Jerry, Tina, Jenna, Bill, Tom, Carol, Barbara, Rebecca, Anna, Bruce, Emily.
To take advantage of this, simply adapt your text description to specify which speaker to use: Jon's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.
You can replace "Jon" with any of the names from the list above to utilize different speaker characteristics. Each speaker has unique vocal qualities that can be leveraged to suit your specific needs. For more detailed information on speaker performance with voice consistency, please refer inference guide.
import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-v1").to(device)
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-v1")
prompt = "Hey, how are you doing today?"
description = "Jon's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."
input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)
Tips:
Include the term "very clear audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
The remaining speech features (gender, speaking rate, pitch and reverberation) can be controlled directly through the prompt
✨ Optimizing Inference Speed
We've set up an inference guide to make generation faster. Think SDPA, torch.compile and streaming!
The training folder contains all the information to train or fine-tune your own Parler-TTS model. It consists of:
1. An introduction to the Parler-TTS architecture
2. The first steps to get started
3. A training guide
[!IMPORTANT]
TL;DR: After having followed the installation steps, you can reproduce the Parler-TTS Mini v1 training recipe with the following command line:
[!IMPORTANT]
You can also follow this fine-tuning guide on a mono-speaker dataset example.
Acknowledgements
This library builds on top of a number of open-source giants, to whom we'd like to extend our warmest thanks for providing these tools!
Special thanks to:
Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively, for publishing such a promising and clear research paper: Natural language guidance of high-fidelity text-to-speech with synthetic annotations.
the many libraries used, namely 🤗 datasets, 🤗 accelerate, jiwer, wandb, and 🤗 transformers.
Descript for the DAC codec model
Hugging Face 🤗 for providing compute resources and time to explore!
Citation
If you found this repository useful, please consider citing this work and also the original Stability AI paper:
@misc{lacombe-etal-2024-parler-tts,
author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
title = {Parler-TTS},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/parler-tts}}
}
@misc{lyth2024natural,
title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
author={Dan Lyth and Simon King},
year={2024},
eprint={2402.01912},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
Contribution
Contributions are welcome, as the project offers many possibilities for improvement and exploration.
Namely, we're looking at ways to improve both quality and speed:
Datasets:
Train on more data
Add more features such as accents
Training:
Add PEFT compatibility to do Lora fine-tuning.
Add possibility to train without description column.
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/huggingface/parler-tts là nguồn chính thức.
huggingface/parler-tts có bao nhiêu sao?
huggingface/parler-tts có 5.6k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/huggingface/parler-tts. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
huggingface/parler-tts có tag gì không?
Bản đồng bộ chưa ghi nhận topic GitHub nào cho huggingface/parler-tts. GitHub topics hiển thị ở thanh bên phải trang repo — đó là nơi đáng kiểm tra nhất.
huggingface/parler-tts còn đang phát triển không?
Commit gần nhất trên huggingface/parler-tts là 1.9 năm trước (theo timestamp GitHub). Repo có 591 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
huggingface/parler-tts viết bằng ngôn ngữ gì?
huggingface/parler-tts chủ yếu viết bằng Python. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Đọc đầy đủ README ở tab phía trên.
Muốn nghe thêm một ý kiến về parler-tts?
Hỏi một AI đọc được trang này — một cú bấm là có ngay nhận định về parler-tts.