Vall-E

Vall-E is a neural text-to-speech (TTS) model that treats speech synthesis as a conditional language modeling task over discrete audio tokens rather than continuous waveform regression. Built on top of an off-the-shelf neural audio codec, Vall-E first encodes speech into discrete codes, then learns to generate these codes conditioned on input text and a short acoustic prompt. Trained on approximately 60,000 hours of English speech, it is designed for zero-shot TTS, enabling high-quality personalized voice generation from only a three-second recording of an unseen speaker.

Vall-E can reproduce speaker identity, prosody, and even environmental characteristics such as background noise or recording conditions. It also shows in-context learning capabilities, adapting to new speakers and styles without fine-tuning.