UltraVoice Improves Fine-Grained Control in Spoken Dialogue Models

Spoken dialogue models can answer questions while still sounding poorly matched to the situation: an instruction to speak quietly, quickly, in a particular accent, or with a specified emotion is often treated as an afterthought. UltraVoice targets that gap with a large speech-dialogue training set organized around explicit style instructions rather than treating expressive speech as a single label. The paper fine-tunes SLAM-Omni and VocalNet on the dataset and evaluates whether the resulting systems follow detailed spoken-style requests while retaining general dialogue skills.

Core Contribution

The central contribution is UltraVoice, a dataset of more than 830 hours of speech dialogues covering six control dimensions: emotion, speed, volume, language, accent, and composite styles. The composite category matters because real requests frequently combine attributes—for example, changing pace, loudness, and affect at once—rather than isolating one factor. The authors frame the dataset as the first large-scale resource engineered for this multi-axis form of speech-style control.

The paper's distinction is not merely adding emotional labels to existing speech corpora. Each example pairs a dialogue context and response with instructions intended to specify how that response should be spoken. This turns style control into an instruction-following problem for spoken dialogue models, allowing evaluation through Instruction Following Rate (IFR) as well as subjective speech-quality judgments.

Technical Approach

Figure 1 outlines a four-stage construction pipeline: text-corpus curation, style injection and response generation, stylized speech synthesis, and quality control and filtering. The visual taxonomy separates the six broad dimensions from finer sub-dimensions, while the examples emphasize that the training distribution includes both isolated controls and combinations. That design is a practical response to a common failure mode in expressive TTS and speech agents: a system can produce an emotional utterance in isolation yet fail when an instruction asks for several compatible attributes at once.

The authors use UltraVoice for supervised fine-tuning of leading spoken dialogue models, specifically SLAM-Omni and VocalNet. They then test style controllability with MOS and IFR, including multi-dimensional requests. The evaluation also checks broader conversational competence on URO-Bench, whose Basic and Pro settings cover understanding, reasoning, and conversation. Finally, the work applies the dataset to controllable text-to-speech training, positioning the corpus as a training resource rather than a model-specific recipe.

Figure 1 also makes clear that the data pipeline relies on generated stylized speech followed by filtering. This enables scale and coverage across accents, languages, and compositional instructions, but it means the resulting models are trained against the fidelity of the synthesis and quality-control stages rather than a fully natural conversational recording distribution.

Results and Analysis

On the paper's multi-dimensional control tasks, UltraVoice fine-tuning improves MOS by 29.12–42.33% and IFR by 14.61–40.09 percentage points for the evaluated models. These are large changes under the reported control-task condition, particularly because IFR measures whether the model followed the requested attributes rather than whether it merely produced intelligible audio. The radar-chart evaluation compares each base model with its SFT variant across the six fine-grained dimensions, using GPT-4o as a reference; its purpose is to show that the gains are distributed across categories rather than being confined to a single style type.

The paper also reports average URO-Bench gains of +10.84% on Basic and +7.87% on Pro after fine-tuning. That is useful evidence against the usual trade-off in which specialized expressive tuning weakens general instruction handling. Still, these benchmark improvements should be read as compatibility evidence, not proof that the same outcome will hold for unconstrained live conversations, unfamiliar languages, or deployment conditions with noisy audio.

The controllable-TTS experiment broadens the case for the dataset, but the strongest evidence remains the reported fine-tuning results on the two named dialogue models and the paper's own control tasks. For practitioners building voice agents, the practical takeaway is concrete: style supervision should include compositional instructions and be evaluated as instruction following, not only with generic naturalness scores. The paper supports that training direction; it does not yet establish that 830 hours of synthesized and filtered dialogue covers the full range of socially appropriate speech behavior.

Limits in Practice

The evaluation is centered on SLAM-Omni, VocalNet, URO-Bench, and UltraVoice-designed control tasks, so generalization to other model families and naturally occurring multi-party dialogue remains unmeasured. MOS is also a subjective metric, and the reported aggregate ranges do not identify which individual styles, accents, languages, or combinations remain difficult. The dataset construction pipeline provides scale, but synthesized speech and automated filtering can preserve artifacts or biases from the upstream generation systems.