FOA Tokens Improve Spatial Audio Reasoning Without Replacing Audio Encoders

Most audio-language models consume monaural waveforms, which preserves sound content but removes the directional and scene-level cues needed to answer questions such as where an event occurs or how sources relate. Spatial-Omni addresses that omission by treating First-Order Ambisonics (FOA) as an additional modality rather than rebuilding the host model's audio stack. The paper's central design choice is conservative: preserve the pretrained audio encoder's semantic behavior and add a separate path for spatial evidence.

Core Contribution

The authors introduce Spatial-Omni, a lightweight integration method built around the Spatial-Omni Encoder (SO-Encoder). Instead of adapting the original audio encoder to ingest spatial signals, SO-Encoder extracts spatial cues from FOA features and turns them into spatial latents. A small projector maps those latents into the LLM token space, where they are learned jointly with audio, visual, and text tokens. This separation is the meaningful departure from approaches that bind spatial modeling to modifying or retraining the original audio encoder.

The work also contributes a training and evaluation substrate sized for the task. SO-Dataset contains 400K FOA clips and 2.1M question-answer pairs, assembled from open-source material, real recordings, and simulation. SO-Bench contains 7K clips and 7K QA pairs in FOA format, with coverage for localization, spatial relations, and motion in addition to broader spatial reasoning. The appendix comparison makes the scope distinction clear: prior datasets often cover only part of that task set, while the authors mark SO-Dataset and SO-Bench as supporting location, relation, and motion together.

Technical Approach

Figure 1 shows the architecture's parallel structure. The original audio encoder remains unchanged; the SO-Encoder supplies a dedicated sequence of spatial tokens; and a lightweight projector aligns those tokens with the multimodal LLM. The design aims to avoid exchanging ordinary audio understanding for directional information, a practical concern when spatial capability is added to an already trained Omni model.

Training is staged rather than presented as a single end-to-end retraining pass. The paper describes the spatial branch as being jointly learned with audio, vision, and text tokens while limiting added context cost. Its dataset construction also tries to bridge controlled and less controlled acoustics: the recorded subset has 3.5K FOA clips across 15 scene types and 23 recording scenes, while simulation uses SoundSpace 2.0 with room data drawn from HM3D, MP3D, and Replica. Annotation includes event category, active interval, track ID, azimuth, elevation, and distance where the evidence supports them; uncertain 3D positions are deliberately represented more coarsely rather than forced into precise labels.

Results and Analysis

The clearest retention evidence appears in Table 17, which compares each base model with its Spatial-Omni variant on MMAU and MMAU-Pro. For Qwen3-Omni, the SO-30B (MIX) variant reaches 69.80 MMAU average versus 68.40 for the base model. On MMAU-Pro, its average rises from 62.84 to 63.96, and the spatial-audio column rises from 36.62 to 46.12. That 9.50-point spatial gain is much larger than the 1.12-point overall MMAU-Pro change, which is consistent with the paper's claim that the extra branch targets a missing capability rather than broadly rewriting audio behavior.

The pattern is not exclusive to Qwen3-Omni. SO-AF3 (MIX) improves Audio Flamingo 3's MMAU-Pro spatial score from 54.77 to 61.23, while SO-Kimi (MIX) changes Kimi-Audio from 57.85 to 38.46. The latter reversal matters: the method is not uniformly beneficial across every backbone and configuration. Phi-4-mm also falls from 62.50 to 58.60 on MMAU average after its SO-4B (MIX) integration, and its MMAU-Pro spatial score slips from 32.62 to 32.31. The evidence therefore supports targeted gains for some models, especially Qwen3-Omni and Audio Flamingo 3, but not a backbone-independent guarantee.

The evaluation is broader than a single localization test, and the real-recording protocol helps move beyond clean simulation. Still, the results shown here are aggregate benchmark scores. The paper provides less evidence about deployment under microphone-array mismatch, live reverberation outside its recorded scenes, or how much the added token stream affects latency and context budgets in production systems.