Multilingual Speech Synthesis

Introduction

This project explores neural text-to-speech (TTS) technologies to produce natural, expressive, and multilingual speech synthesis. We evaluate models such as Coqui TTS, XTTS, Edge TTS, and specialized cloud APIs.

The focus is on French voice quality, emotion and intonation management, as well as the ability to generate synthetic voice in real time with latency under 100ms.

The study also includes voice cloning customization and adaptation to specific domains via fine-tuning on specialized corpora.

Neural speech synthesis has undergone a radical transformation in recent years. Where older TTS systems produced a metallic, artificial voice — that of GPS and early voice assistants — modern architectures generate speech nearly indistinguishable from the human voice. This revolution is driven by deep neural models: Tacotron, FastSpeech, VITS, and more recently architectures based on transformers and diffusion models. Unlike parametric or concatenative synthesis that dominated until the 2010s, neural TTS directly learns the mapping between phonemes, acoustic features, and raw audio samples. The result is unprecedented fluency, expressiveness, and vocal fidelity, opening the door to previously inaccessible use cases.

Voice Quality

Naturalness of a synthetic voice is the most fundamental quality criterion. A natural voice is not limited to correct word pronunciation: it incorporates rhythm, emphasis, pitch variations, micro-pauses, breathing, and emotional modulations. These prosodic features are what distinguish a living voice from a mechanical one. Modern neural models capture these nuances by learning from thousands of hours of human voice recordings. The XTTS model, for example, uses a sequence-to-sequence architecture with attention mechanisms capable of learning a specific speaker's prosody from just a few seconds of reference audio. The quality achieved is such that untrained listeners cannot distinguish synthetic voice from natural voice under standard listening conditions.

Mean Opinion Score (MOS) is the standard for subjective voice quality evaluation. The protocol involves having a panel of human listeners rate audio samples from 1 (poor) to 5 (excellent). Samples are presented in random order, with built-in quality controls to detect inattentive evaluators. A MOS of 4.5 or higher is considered equivalent to natural human voice quality. Our systematic evaluation campaigns measure MOS for each TTS engine across multiple languages, text types, and voice configurations. Results show that the best neural models — XTTS v2, Edge TTS, and ElevenLabs — achieve MOS scores between 4.2 and 4.7 depending on language and content type. French is the language where gaps between models are most pronounced, reflecting the additional difficulty of its prosody and complex liaisons.

The multilingual dimension is at the heart of this project. Not all TTS engines perform equally depending on the target language. Models predominantly trained on English often produce artificial voices in other languages, with a perceptible accent, unsuitable prosody, or incorrect pronunciation of specific sounds. French presents particular challenges: vowel nasalization, mandatory and optional liaisons, elision, distinction between open and close vowels, the typical descending melody of declarative sentences, and management of silent final e. XTTS excels in French thanks to a balanced multilingual training corpus and a per-language conditioning mechanism that preserves the phonetic characteristics of each language. Microsoft's Edge TTS offers comparable quality on standard spoken French, while Coqui TTS requires extensive fine-tuning to match these results. Asian languages — Mandarin, Japanese, Korean — present additional challenges related to tones and complex writing systems, areas where Western models still struggle.

Latency is a critical factor for interactive applications. Natural conversation requires the voice response to be produced in under 100 milliseconds, a threshold below which the delay is imperceptible to the user. This technical constraint is difficult to satisfy for neural models, whose inference is inherently more costly than traditional methods. The standard architecture of a neural TTS pipeline — text encoding, mel spectrogram generation, neural vocoder decoding (HiFi-GAN, WaveGlow) — adds incompressible latency. Our optimizations focus on three levers: parallelization of generation steps, use of efficient vocoders like HiFi-GAN that convert spectrograms to audio signals in real time, and audio streaming via WebRTC that starts playback as soon as the first samples are available, without waiting for synthesis to complete. By combining these techniques, we achieve end-to-end latency of 60 to 90 ms on a modern GPU, compatible with real-time requirements.

Voice cloning is one of the most impressive capabilities of modern neural TTS. The technique enables reproducing a specific speaker's voice from an audio sample of a few seconds to a few minutes. XTTS implements this mechanism via a speaker encoder that extracts a vector representation of the voice (speaker embedding), then used to condition generation. Unlike fine-tuning which requires retraining the model on hours of recordings, XTTS's few-shot approach enables instant cloning with surprising quality from just 5 to 10 seconds of reference. Voice clone fidelity strongly depends on source sample quality: a clean recording, without background noise or reverberation, with a calm voice and clear diction, produces the best results. Emotion and expressivity, however, are more difficult to transfer: a voice clone captures timbre and general prosody, but reproducing fine emotional variations remains an open challenge.

Voice customization goes beyond simple cloning. It encompasses the ability to modify a synthetic voice's attributes independently of the source speaker: pitch adjustment, speech rate, emotional energy (joy, sadness, anger, neutrality), and even perceived age. Modern models expose these parameters via style embeddings or explicit prosodic controllers. In our architecture, these customization parameters are accessible via a REST API that accepts semantic descriptors: a client can request a "female, young, cheerful voice speaking French at moderate speed" and obtain a corresponding synthesis without technical knowledge of the underlying models. This abstraction layer is crucial for adoption of speech synthesis by applications without audio processing expertise.

Benchmarking local versus cloud solutions answers a pragmatic question: where to run speech synthesis? Cloud APIs like Azure Speech, Google Cloud TTS, ElevenLabs, and Amazon Polly offer immediate quality, managed infrastructure, and usage-based pricing. They excel in synthesis quality and language coverage, with high MOS scores and zero maintenance. Their drawbacks are recurring cost at high volume, incompressible network latency (50 to 200 ms round-trip), dependence on an internet connection, and data sovereignty — the text to be synthesized leaves the local infrastructure. Local solutions — Coqui TTS, self-hosted XTTS, VITS — reverse the equation: no network latency, no API cost, total data confidentiality, possibility of fine-tuning on business corpora. Their limitations are sometimes lower initial quality (except fine-tuning), deployment complexity, and GPU dependence for the most performant models. Our benchmarks systematically compare both approaches: for moderate usage (under 100,000 requests per month), cloud is economically advantageous. Beyond that, local becomes more cost-effective, with a return on investment of 6 to 12 months depending on volume.

Applications of neural speech synthesis cover a broad and rapidly expanding spectrum. Interactive telephony (IVR) is one of the most transformed domains: traditional interactive voice servers, with robotic timbre and rigid menus, are giving way to natural voice assistants capable of understanding and responding in spoken language, with voices customized by brand and context. Digital accessibility is an equally important driver: high-quality speech synthesis enables visually impaired or dyslexic people to access written information fluidly and pleasantly, far superior to historical synthesis voices. Content creation — audiobooks, podcasts, training videos, e-learning — benefits from reduced production costs: a professional-quality voice can be generated in seconds, without needing an actor or recording studio.

E-health opens fascinating perspectives: personalized voices for patients who have lost the ability to speak due to neurological conditions can be reconstructed from previous recordings. This voice cloning application — sometimes called "voice banking" — allows a patient to keep their natural voice even after losing the ability to speak. Personal voice assistants, integrated into connected objects and smart speakers, benefit from more natural voices that improve user engagement and trust. In the automotive domain, high-quality speech synthesis enriches the driving experience: navigation, traffic information, vehicle interactions — with a natural voice that does not distract the driver. Education and training leverage multilingual synthesis to produce educational content in multiple languages from a single text, accelerating localization and reducing production costs.

Technical Challenges

Remaining technical challenges guide the continuation of our research. Emotion and expressivity management remains imperfect: current models struggle to finely modulate emotion mid-sentence, for example shifting from a neutral to an interrogative intonation on a specific word to express doubt. Long text coherence is another challenge: generating a 30-minute speech synthesis without prosodic drift, with a stable voice from start to finish, requires context tracking mechanisms that are not yet mature in current architectures. Fine pronunciation control — proper names, acronyms, technical terms — requires a sophisticated text normalization (front-end) module, an area where solutions differ considerably in quality. Model size reduction to enable synthesis on mobile or embedded devices, without significant quality degradation, is a priority optimization axis.

Architecture

The internal architecture of our evaluation platform reflects the complexity of the TTS pipeline. The system is structured into independent, interchangeable layers. The input layer normalizes the source text: language detection (for multilingual), abbreviation resolution, number transcription, punctuation, and special symbol handling. The synthesis layer itself encapsulates different TTS engines behind a unified interface, allowing switching between Coqui TTS, XTTS, Edge TTS, or a cloud API without modifying application logic. The post-processing layer applies audio filters: volume normalization, residual noise reduction, equalization adjustment, insertion of natural silences between sentences. The streaming layer encapsulates the audio output in WebRTC channels for real-time broadcast, with adaptive buffering to compensate for latency variations. Everything is orchestrated by a central service that manages sessions, frequent synthesis caching, and quality logging.

Acoustic analysis with Librosa plays a central role in our evaluation methodology. We use this library to automatically extract objective voice quality metrics: Perceptual Evaluation of Speech Quality (PESQ), spectral distortion, formants, jitter, and shimmer — indicators of voice stability. These objective measurements are correlated with subjective MOS evaluations to build a predictive model of perceived quality. The ultimate goal is to be able to estimate MOS of a synthesis without systematically resorting to a human panel, which is costly and slow to mobilize. Initial indications suggest promising correlation between certain spectral metrics and MOS scores, paving the way for a reliable automatic evaluation tool.

Perspectives

The project roadmap is organized around several priority axes. Fine-tuning XTTS on specialized corpora — medical voices, educational voices, children's voices — to improve quality in domains where generalist models struggle. Integration of an input emotion detector, which would analyze the source text to infer the appropriate emotion and parameterize synthesis accordingly. Reduction of XTTS model size through distillation and quantization to enable real-time CPU deployment, without a dedicated GPU — a major technological barrier for democratizing neural speech synthesis. Extension to new languages, particularly French regional languages (Breton, Alsatian, Corsican) and African languages, where TTS offerings are virtually nonexistent. Finally, creation of an in-house speech synthesis service, combining an XTTS model fine-tuned on high-quality French voices with a front-end normalization module optimized for the French language and a low-latency vocoder for real-time.

Obtained results suggest significant potential of neural speech synthesis for professional applications. The quality achieved — MOS above 4.5 in the best configurations — makes synthetic voice indistinguishable from human voice for the majority of uses. Latency under 100 ms is now achievable on modern GPU infrastructure, and progress in model optimization suggests CPU operation in the near future. Coqui TTS and XTTS technologies, in particular, offer a solid open source foundation for building custom solutions without proprietary dependency. Voice customization, from simple speed adjustment to full voice cloning, opens user experience possibilities that were unthinkable five years ago. This project will contribute to the democratization of natural speech synthesis in French-language professional applications, by providing a rigorous evaluation methodology and concrete architectural recommendations.