Full-Context Rendering Reduces Music Codec Exposure Bias

Hybrid music generators split song creation between an autoregressive language model that plans discrete codec tokens and an acoustic renderer that produces the waveform. The split creates a training-deployment mismatch: renderers learn from clean, target-derived token plans but must operate on language-model predictions containing substitutions and omissions. FullDiT reframes that downstream task as generative recovery from an imperfect plan rather than reconstruction of the plan itself.

Core Contribution

The paper's central claim is that an acoustic renderer should retain the clean waveform target while deliberately corrupting only its codec-token condition during training. The proposed Error-Matched Distractor Conditioning (EMDC) is intended to approximate the mistakes made by the planning language model, so the renderer learns to use text and global acoustic context when the discrete plan is unreliable.

This differs from a standard conditional reconstruction setup in two ways. First, the model sees the complete song latent sequence through non-causal self-attention rather than predicting with only past context. Second, lyrics and captions are independently encoded conditions, not information implicitly assumed to be preserved by the codec plan. The intended payoff is resilience to plan errors without changing the acoustic target that defines the desired song.

Technical Approach

FullDiT is a conditional diffusion transformer operating over VAE acoustic latents. Its input condition combines eight frame-aligned residual vector-quantization (RVQ) codebook streams with separately encoded lyric and caption text. A frozen VAE decoder maps the generated latent sequence to waveform after denoising.

EMDC corrupts codec conditioning according to per-codebook teacher-forced top-1 replacement rates. Instead of sampling arbitrary replacement tokens, it draws plausible near misses from cosine-nearest-neighbor token neighborhoods. The clean target waveform and target VAE latent are left unchanged. That detail matters: the model is not trained to reproduce a corrupted sequence, but to infer a full latent realization compatible with a noisy plan and the textual conditions.

Figure 1 depicts this division of labor: the codec plan can contain clean tokens and near-miss substitutions, while the full-context renderer uses text and non-causal song-level context to recover the clean acoustic latent sequence.

At inference, four-way classifier-free guidance independently controls the increments associated with codec, lyric, and caption conditioning. The paper reports screening seven guidance tuples with Audiobox PQ, then selecting candidates for a blind three-way forced-choice evaluation. This is a useful design choice because a single aggregate guidance scale would obscure trade-offs between adherence to the planned music, vocal lyrics, and descriptive text.

Results and Analysis

The clearest matched ablation concerns EMDC. Under synthetic codec corruption, the method improves ViSQOL by 0.77. The authors also report that, when language-model tokens are fixed, EMDC variants are clearly preferred in non-tied human comparisons. Together, those results support the narrower claim that error-aware conditioning helps when the plan is wrong; they do not establish that every real planning error follows the synthetic corruption process.

Additional ablations attribute gains to full-song context and renderer-side text conditioning. This is consistent with the mechanism: long-range musical structure and words provide alternative signals when a local RVQ token is unreliable. The paper does not present the renderer as independently composing a song from scratch; its contribution is recovering acoustics from a plan that remains informative but imperfect.

For the complete system, the authors report outperforming five commercial systems on 15 of 18 automatic metrics and placing among the top three entries on the Artificial Analysis Music with Vocals Leaderboard snapshot. Figure 3 identifies the evaluated system as Lucky-Dolphin-Music-WQ-0618. The breadth of automatic comparisons is encouraging, but the aggregate count conceals which three metrics did not favor FullDiT and how large individual margins were. The reported evidence is strongest for interface-mismatch mitigation and less decisive as a general ranking of music-generation quality.

Caveats in Practice

The evaluation relies partly on synthetic corruption calibrated to teacher-forced top-1 errors, which may not capture sequence-level failures from a deployed planner. The supplied results also do not establish behavior across languages, musical genres, song lengths, or adversarially poor plans. Finally, leaderboard placement is tied to an archived snapshot and should be interpreted as a time-specific comparison rather than a permanent ordering.