Explicit Planning Improves Long-Form Music Generation Scores

Long-form music generation has to preserve an arrangement over time while rendering vocals, instruments, rhythm, and production detail. StepAudio 3 Music tackles that split problem with a symbolic planning stage before music-token generation, then uses a separate continuous acoustic renderer to produce 48-kHz waveform audio. The paper’s central argument is that an editable arrangement representation makes generated music easier to guide than direct text-to-token generation alone.

Core Contribution

The system combines three components that are usually treated separately: a music representation, an explicit musical plan, and acoustic synthesis. Its tokenizer emits a single 65,536-entry codebook at 50 Hz, intended to retain both temporal musical structure and reconstruction-relevant information. An autoregressive mixture-of-experts model then produces an intermediate ABC notation plan, termed ABC-CoT, before predicting music tokens. That plan is meant to expose melody, harmony, rhythm, and form as generation context rather than leaving them implicit in a text prompt.

The distinction is practical. ABC notation gives a text-based LLM a readable object to revise: it can reorganize rhythm, alter chord progressions, or change instrumentation before the acoustic model commits those choices to audio. The authors position this as a route to longer and more controllable outputs, including song and instrumental generation, dry-vocal accompaniment, and covers up to 5 minutes 30 seconds.

Technical Approach

The paper uses a discrete-continuous pipeline. A semantically informed self-supervised and multi-task-trained tokenizer maps audio into music tokens; a flow-matching diffusion Transformer predicts continuous StepAudio VAE latents conditioned on those tokens; and the VAE decoder renders the resulting 48-kHz audio. The reported tokenizer comparison separates direct VAE reconstruction from token-conditioned detokenization. In direct reconstruction, the 50-Hz StepAudio VAE reports MCD 3.59 ± 1.32 and SDR 8.57 ± 3.53, compared with MCD 4.79 ± 1.31 and SDR 5.54 ± 2.96 for its 25-Hz version.

For planning, the ablation compares direct generation, ABC-CoT, and ABC-CoT followed by LLM editing using the same DPO model. This isolates a meaningful question: whether the improvement comes from supplying a plan, or from revising that plan before the model renders music. The preference-optimization step is also tested independently against supervised fine-tuning.

Results and Analysis

On the common objective subsets, StepAudio 3 Music leads the reported AudioBox-Aesthetics axes: Content Enjoyment 7.7086, Content Usefulness 8.0052, and Production Quality 8.3868. It also records MuQ-MuLan music-text similarity of 0.4465, above Suno V5.5’s 0.4437, Suno V5’s 0.4287, Mureka V9’s 0.3752, and MiniMax Music 3’s 0.3703. The picture is less uniform on SongBench: Mureka V9 leads each individual listed dimension, while StepAudio reports a 5.7465 musicality score versus Mureka’s 5.9382. The evidence therefore supports better scores on the paper’s audio-quality and caption-alignment measures, but not a clean lead across all benchmarks.

DPO yields small but consistent gains over supervised fine-tuning: SongBench musicality rises from 5.6395 to 5.7465, AudioBox Content Enjoyment from 7.6259 to 7.7086, Production Quality from 8.2700 to 8.3868, and MuQ-MuLan from 0.4157 to 0.4465. More revealing is the planning ablation. ABC-CoT raises the seven-dimension SongBench mean from 6.6438 to 6.6713; LLM editing raises it further to 6.7220. The absolute improvement over direct generation is 0.0781, so the planning result is promising rather than decisive, but it is directionally consistent across melody, arrangement, vocal, instrumental, mixing, structure, and musicality.

The preliminary Artificial Analysis Music Arena Vocals snapshot gives the model a Quality Elo of 1105 from 2,119 samples. It ranks fourth in the displayed list, behind Suno V5.5 at 1170, Mureka V9 at 1159, and Mureka V8 at 1140, while ahead of Suno V5 at 1097. That external signal complements the internal benchmarks, though it is not an ablation test and uses different prompts, outputs, and voters.

Limits in Practice

The authors explicitly note that SongBench filters the supervised-fine-tuning data, so evaluation is not fully independent of data selection, and DPO uses expert pairwise preferences while automatic metrics are used for both filtering and evaluation. The objective comparison uses one generation per condition, leaving variation across random seeds unmeasured. Hosted baselines cannot be reproduced with identical service-side settings. Finally, the reported subsets emphasize vocal and lyrics-conditioned music; they do not establish comparative performance for instrumental, cover-song, or vocal-to-mix generation. Complex arrangements, long temporal dependencies, vocal naturalness, and high-frequency reconstruction remain stated areas for improvement.