Unified SSL Encoders Broaden Audio LLM Coverage

Audio large language models often start from supervised encoders tuned for particular domains: speech recognition, music tagging, environmental sound, or paralinguistic classification. That specialization creates a practical tension. A model can do well on one subset of hearing tasks while leaving gaps on others, especially when the downstream prompt asks for temporal localization, spoofing cues, or speech quality judgments rather than plain transcription. SALMONN-2 tests a different premise: a single self-supervised audio encoder can provide the foundation for a broader audio-language model if the interface to the LLM preserves enough of the encoder hierarchy.

The paper introduces SALMONN-2 around SPEAR, a unified SSL encoder, and pairs it with a multi-layer feature fusion adapter. The authors also study multimodal in-context learning for audio, arguing that it does not appear automatically from ordinary audio instruction tuning and instead needs targeted contextual biasing training.

Figure 1 summarizes the authors' positioning of SALMONN-2 against other open-weight audio LLMs across conventional audio understanding benchmarks and less-covered analysis tasks.

Core Contribution

The main contribution is not another task-specific audio front end. It is the claim that a unified SSL representation can replace mixtures of supervised encoders without collapsing performance on specialized tasks. The paper's ablations support that reading: SPEAR is evaluated against dual-encoder and single supervised-encoder configurations across speech, audio, music, and paralinguistic tasks, and the best SPEAR configuration is competitive across most columns rather than excelling only where the encoder was pretrained.

The second contribution is the treatment of temporal and contextual conditioning as trainable capabilities. SALMONN-2 injects timestamp information for temporally grounded audio understanding, and it trains contextual ASR through multimodal in-context examples rather than assuming the LLM will infer that behavior from scale.

Technical Approach

SALMONN-2 uses SPEAR as the audio encoder and adds an MLF adapter between the encoder and the language model. The adapter aggregates representations from all encoder layers before projection. That design choice matters because SSL encoders distribute information across depth: lower and middle layers may carry phonetic or acoustic detail, while later layers may become more task-shaped. A final-layer projection is simpler, but it risks throwing away useful intermediate information.

The paper compares three SPEAR usage patterns in Table IX: final-layer features, a weighted sum over layers, and MLF. The weighted-sum variant can combine layers, but the authors argue that it learns a single set of layer weights shared across all downstream tasks. MLF instead preserves layer-wise information before projection, giving the LLM a more structured view of the encoder hierarchy.

For temporal grounding, SALMONN-2 interleaves timestamp embeddings with audio embeddings. The ablation is deliberately narrow: the authors remove timestamp injection and test the change on sound event detection plus three general audio-language benchmarks. The reported gain is small on the general benchmarks, but it is consistent, and the SED drop is larger.

Results and Analysis

The clearest evidence comes from the encoder comparison. With SPEAR plus MLF, SALMONN-2 reports 68.1 on AudioCaps, 5.8/22.6 on MusicCaps, 1.6/3.2 WER on LibriSpeech ASR clean/other, 9.6 WER on GigaSpeech, 45.9 on CoVoST2, 2.7 PER on LibriSpeech phoneme recognition, 9.7 on LibriMix, 70.5 on IEMOCAP, and 94.7 on VoxCeleb. Against the dual-encoder Whisper + BEATs setup, MLF is slightly lower on AudioCaps and MusicCaps but better on LibriSpeech ASR, CoVoST2, LibriSpeech PR, LibriMix, IEMOCAP, and VoxCeleb. That is the paper's strongest empirical point: the unified SSL route does not simply trade breadth for uniformly weaker numbers.

The comparison with supervised single encoders is also instructive. AF-Whisper reaches 73.0 on IEMOCAP, above SPEAR + MLF's 70.5, but is weaker on several other tasks including AudioCaps, LibriMix, and VoxCeleb. Qwen2.5-Omni Encoder trails on AudioCaps at 63.6 and has a 9.0 PER on LibriSpeech PR. The pattern fits the authors' claim that supervised encoders can retain domain peaks while giving a less balanced profile.

The feature-fusion ablation is more modest but still useful. Compared with the SPEAR final-layer baseline, MLF improves LibriSpeech ASR at 1.6/3.2 versus 1.7/3.3, matches GigaSpeech at 9.6 versus 9.6 and CoVoST2 at 45.9 versus 45.9, and improves LibriMix at 9.7 versus 10.0, IEMOCAP at 70.5 versus 69.8, and VoxCeleb at 94.7 versus 91.9. Weighted-sum fusion helps some metrics but does not match MLF overall.

Timestamp injection shows a smaller effect. Removing it lowers SED from 70.15 to 68.74, MMAU-Pro from 58.5 to 58.2, MMAR from 64.5 to 64.3, and MMSU from 69.5 to 69.3. The SED result supports the mechanism directly; the benchmark gains are too small to carry the paper by themselves, but they suggest timestamp tokens do not hurt general audio-language performance.

Limitations

The evidence is broad across tasks, but the visible ablations mostly report aggregate scores rather than error analysis. The MICL claim is plausible from the abstract, yet the provided pages give fewer concrete numbers for contextual biasing than for encoder fusion and timestamp injection. The paper also positions SALMONN-2 among comparable-scale open-weight models, so its conclusions should not be read as a claim against closed systems or much larger proprietary models.