End-to-End Audio Generation Cuts Speech Errors

Text-to-audio systems can produce isolated sound events, speech, or music, but a coherent scene may require all three at once: intelligible dialogue under music, environmental effects that fit a caption, and timing that remains plausible over a variable duration. The paper argues that the usual division between a frozen text encoder and a separate audio decoder limits joint optimization, with speech intelligibility the clearest casualty. MiDashengLM-Gen instead couples language modeling and audio generation in one end-to-end-trained system.

Core Contribution

The central claim is architectural rather than a new benchmark alone. MiDashengLM-Gen uses a pre-trained large language model to model an audio scene as a sequence of latent tokens, then applies conditional flow matching at each token to turn those predictions into audio representations. That makes the generator autoregressive and variable-length while avoiding a fully separate, frozen text-conditioning stage.

The distinction matters because text alignment and waveform-level fidelity impose different demands. A language model can carry the long-range semantic and temporal context of a prompt; a conditional flow component can model the continuous detail required by the corresponding audio latent. The authors position this coupling as an end-to-end alternative to decoupled text-encoder/audio-decoder pipelines, and as a general text-to-audio model rather than a dedicated speech synthesizer.

Technical Approach

The reported pipeline is built around a pre-trained LLM and per-token conditional flow matching. At generation time, the LLM autoregressively predicts the context for successive audio-latent tokens, while the flow-matching component generates the token-level latent detail conditioned on that context. The resulting sequence can represent speech, music, sound effects, or their combination, and its length need not be fixed in advance.

The available figure captions provide two useful implementation-level cues. Figure 1 distinguishes training and inference pipelines, which is consistent with the paper's separation of autoregressive context prediction from token-level generation. Figure 2 describes audio-text alignment in the DashengTokenizer latent space under general caption supervision. This suggests that the audio representation is intended to preserve semantic alignment across mixed audio, rather than being optimized solely for transcription or a single sound class.

Results and Analysis

The most concrete result is on Seed-TTS. The paper reports an English word error rate of 2.79% for MiDashengLM-Gen, compared with 12.15% for existing unified models. That is a 9.36-percentage-point absolute reduction, and it narrows the distance to the 1.24% reported for dedicated text-to-speech systems. For a model that must also generate music and sound effects, this is the result that gives the end-to-end design practical weight: unified generation no longer appears to require accepting severely degraded speech recognition accuracy.

The comparison should still be read carefully. A dedicated TTS system remains lower at 1.24% WER, so the paper does not establish that a general audio-scene generator replaces specialized speech synthesis when transcription accuracy is the sole objective. Its contribution is a more favorable trade-off: substantially better intelligibility than prior unified systems while retaining a broader output space.

The abstract further reports competitive multilingual WERs and competitive mixed-audio quality on MECAT. Those findings support the intended scope, but the supplied material does not provide the corresponding numerical scores, baselines, sample counts, or uncertainty estimates. The evidence therefore clearly supports the English Seed-TTS improvement, while the breadth of the multilingual and mixed-scene claims depends on results not numerically specified here.

Caveats in Practice

The evaluation described here emphasizes WER and benchmark competitiveness. It does not establish performance for long-form scene consistency, speaker control, prompt safety, compute cost, latency, or difficult real-world mixtures outside the named benchmarks. Nor does the supplied evidence show whether gains hold across languages and audio categories at the same magnitude as the English Seed-TTS result. The method is promising where mixed-audio generation and intelligible speech must coexist; its advantage over specialized systems remains task-dependent.