Open AV-LLM Extends Reasoning to Long Videos

Most audio-visual language models are still evaluated on short clips, where the task is often to recognize a sound, identify an object, or answer a question about a localized event. AV-Flamingo targets a harder setting: long real-world videos where speech, ambient audio, visual actions, and scene changes need to be tied together over time. The paper introduces an open audio-visual LLM family, a large training corpus called Audio-Visual-Skills, and a timestamp-grounded reasoning recipe designed for long-context audio-visual QA.

The central claim is not only that adding audio helps video QA. The stronger claim is that long videos require training examples that force temporal synthesis: comparing distant moments, counting repeated events, explaining topic-level intent, and grounding intermediate reasoning steps to timestamps rather than answering from a single clip.

Core Contribution

The paper’s main contribution is a full training stack for long audio-visual understanding. Audio-Visual-Skills contains about 7M caption and question-answer instances drawn from real-world videos, with categories that explicitly push models beyond short-window perception. The appendix pages show the data-generation prompts for comparative reasoning, context understanding, counting, topic-level reasoning, and AV-Think reasoning triplets. These prompts ask the generator to require both modalities, avoid obvious textual hints, verify noisy audio captions against visual evidence, and place questions at the start, middle, or end of the video rather than only in the opening segment.

That design matters because many existing audio-visual datasets can be solved by one modality or by a short local cue. AV-Flamingo’s data recipe instead tries to make the supervision match the failure mode: long videos contain dispersed evidence, speech may be noisy or incomplete, and the relevant event may only become clear when the visual sequence and audio track are interpreted together.

Technical Approach

AV-Flamingo starts from OmniVinci and trains in three stages: short-context AV-Skills pre-training, long-context AV-Skills mid-training, and chain-of-thought post-training. Figure 2 summarizes the model path and architecture. Separate visual and audio encoders feed adaptors, the resulting audio-visual embeddings are temporally grouped, and Rotary Time Embedding aligns them with prompt text tokens before the LLM processes the sequence.

The paper produces two instruction-tuned variants. AVF-Instruct is trained for direct audio-visual instruction following. AVF-Think adds Temporal Audio-Visual Interleaved Chain-of-Thought, where the model is taught to interleave intermediate reasoning with timestamps. The visible appendix examples show the authors asking the generator to produce open-ended QA, multiple-choice QA, a timestamped thinking chain, and an audio-visual caption. The prompt explicitly rejects questions answerable from a single modality and requires grounding events with tags such as <t>...</t>.

Figure 3 gives concrete examples of Audio-Visual-Skills question types, while the later appendix pages expose the construction logic behind those examples. The data-generation prompts are unusually specific: they ask for contrast or similarity across separate video moments, broader scene context inferred from sound and visuals, and counting over the full video while checking noisy captions against other cues.

Results and Analysis

The paper reports evaluation across more than 15 audio-visual, omni-modal, audio, and vision benchmarks. The abstract states that AV-Flamingo beats similarly sized open models by clear margins and remains competitive with, and sometimes ahead of, larger open-weight and closed models, especially on long and complex real-world audio-visual understanding. The available visual sheets do not expose the full benchmark tables, so the safest reading is qualitative rather than per-benchmark: the evidence presented by the paper supports a broad evaluation claim, but the exact margin should be checked in the main results tables before using any single number as a headline result.

The more defensible technical takeaway is about task construction. The appendix pages show that the authors are not merely scaling generic captions; they are shaping supervision around temporal and cross-modal reasoning failure cases. For practitioners building AV assistants, that is the useful part: the training curriculum and data prompts define a reproducible recipe for moving from short clip recognition toward long-video event reasoning.

Caveats

The paper depends heavily on synthetic QA and reasoning generation. The prompts try to control hallucination by requiring modality checks, timestamp grounding, and fallback to None when a question is not possible, but prompt rules are not the same as verified human labels. The work is also benchmark-driven: strong scores across 15+ datasets are meaningful, yet real deployment would still need stress tests for noisy speech, missing audio, ambiguous timestamps, and videos where audio and visuals conflict rather than reinforce each other.