Low-Rate Tokens Improve Controllable Speech Synthesis

Modern zero-shot TTS systems are being pulled in several directions at once: they need to preserve the speaker, keep the text correct, sound natural, follow style instructions, and stay usable under noisy enrollment speech. Qwen-Audio-3.0-TTS is Alibaba’s attempt to optimize that whole production envelope rather than one clean-speech benchmark. The paper’s central bet is that a lower-rate discrete speech interface can reduce autoregressive cost if the acoustic renderer and training curriculum compensate for the lost temporal detail.

Core Contribution

The system combines a Qwen language model for semantic speech-token prediction, a flow-matching acoustic model conditioned on LM hidden states, and a BigVGAN vocoder. Its 12.5 Hz tokenizer is the main architectural change from CosyVoice-style systems: the token rate is halved from 25 Hz, while a larger finite scalar quantization codebook and multi-task supervision are used to retain content, speaker, language, emotion, and acoustic-event information.

The authors also put substantial weight on controllability. The model accepts natural-language instructions for role, emotion, speaking style, rate, timbre, and accent, plus 86 fine-grained inline tags for phrase- or word-level control, including non-verbal events such as laughter, breathing, coughing, and sighing. That makes the paper less a single-model benchmark report than a recipe for a deployable voice-generation system.

Technical Approach

The training pipeline has five stages. First, the LM and flow-matching model are pretrained independently, preserving the division between semantic planning and acoustic reconstruction. Second, joint LM–FM training couples the two through continuous LM hidden states and applies high-quality data annealing, shifting later training toward cleaner, more expressive speech. Third, LM reinforcement learning uses GRPO with a composite reward for content consistency, duration, generation diversity, and prosody. Fourth, the LM is frozen while the flow-matching model is trained on degraded prompts with additive noise, reverberation, codecs, packet loss, echo, far-field recordings, and device artifacts. Fifth, flow-matching reinforcement learning optimizes speaker similarity, ASR intelligibility, and DNSMOS perceptual quality.

The paper reports the leaderboard context explicitly: Figure 1 shows the Artificial Analysis Text-to-Speech Arena snapshot where Qwen-Audio-3.0-TTS-Plus ranks first on July 16, 2026, with an Elo score of 1,237 from 1,427 samples.

Results and Analysis

The tokenizer ablation supports the low-rate design, but with a qualification. Dropping from 25 Hz to 12.5 Hz with the same 6,561-code vocabulary hurts content consistency and speaker similarity; scaling the 12.5 Hz tokenizer to 59,049 codes recovers much of that loss. On SEED-TTS-Eval, the final tokenizer reaches 1.23 CER and 83.09 SIM on Chinese, 2.37 WER and 77.49 SIM on English, and 6.68 CER with 80.61 SIM on the hard set.

On SEED-TTS-Eval against recent systems, Qwen-Audio-3.0-TTS is competitive rather than uniformly dominant. It reports 0.84 CER on test-zh, better than the human transcription row at 1.26 and below CosyVoice3-1.5B at 1.12, while its test-en WER is 1.54 and test-hard CER is 7.00. The paper’s own interpretation is sensible: the model aims for a balance between content accuracy and speaker similarity rather than only minimizing WER or CER.

The broader evaluations are stronger. On CV3-Eval cross-lingual voice cloning, the paper reports an average error reduction from 10.09% for CosyVoice3-1.5B to 4.05%, about a 60% relative reduction, across 12 transfer directions. In the degraded-reference benchmark, Qwen-Audio-3.0-TTS reaches DNSMOS 3.962, 3.925, and 3.305 under noisy, reverberant, and unclear prompts, while keeping speaker similarity at 76.14%, 74.12%, and 76.53%. That is the most convincing part of the evaluation because it targets a failure mode that clean zero-shot TTS benchmarks often miss.

Instruction following is mixed but useful. On the bilingual zero-shot voice-cloning benchmark, Qwen-Audio-3.0-TTS scores 78.94 overall in Chinese and 80.45 in English, above CosyVoice3-1.5B at 75.91 and 64.09. Speaker similarity is slightly lower than CosyVoice3-1.5B, however, at 73.27 versus 75.60 in Chinese and 66.66 versus 68.45 in English. The trade-off is visible: stronger style control appears to cost a little prompt-voice preservation.

Caveats

The evidence is broad, but it is still a technical report from the model’s builders. Some comparisons use commercial APIs and public releases rather than fully controlled training-matched baselines. The paper also emphasizes aggregate wins across many axes, which can hide task-specific regressions such as the speaker-similarity drop in instruction-following tests. There is no code or model-release link in the provided paper pages, so reproducibility rests mainly on reported methodology and benchmark tables.