Time Markers Ground Audio Answers Across Two Hours

Long audio remains awkward for audio-conditioned language models because the model must do two things at once: understand the speech and preserve where evidence occurred in time. Meeting recordings, podcasts, lectures, and call-center logs often require answers such as “what happened?” and “when did it happen?” The paper introduces GigaChat Audio, a time-aware audio LLM designed to answer with explicit timestamps over inputs up to 120 minutes.

The central design is simple: keep the audio stream continuous, but insert periodic timing anchors so the language model has temporal reference points in its input sequence. Figure 1 shows the architecture: text tokens, projected audio tokens, and timing tokens are interleaved before entering the LLM.

Core Contribution

The paper’s main contribution is not a new audio encoder. It is a practical recipe for making long-audio generation verifiably time-aware: add inter-timing markers, train on duration mixtures rather than a single window length, and generate synthetic supervision from timestamped transcripts at scale.

The authors release an open-weight model for up to 120-minute inputs and a dataset described as more than 10k hours spanning seconds-to-hours recordings. They also frame temporal grounding as the measurable target. Instead of relying only on free-form answer quality, they score interval overlap, median absolute timestamp error, and judge-based metrics for timed descriptions and summaries.

Technical Approach

GigaChat Audio attaches an audio front end to a 10B-parameter LLM with a 256k-token context. The audio stack uses an encoder, subsampler, and projector to produce continuous audio embeddings aligned to the text embedding space at 160 ms frame rate. The paper states that the encoder uses a HuBERT-like setup from the authors’ previous work, with KMeans distributions from a previously trained audio LLM encoder as distillation targets over 2 million hours of unlabeled multilingual audio.

For time awareness, the input sequence includes explicit audio tokens and periodic inter-timings. The timestamps can be written as plain text, such as hh:mm:ss, or as dedicated timing tokens. The strongest format in the ablation is the plain-text hh:mm:ss representation, which reaches 50.9 mIoU at one marker per 60 seconds, compared with 47.1 for m:0, 43.8 for m:, 44.5 for m, and 20.9 for sec.

The data pipeline is also central. Figure 2 shows the synthetic supervision process: audio is transcribed and aligned into timestamped transcripts; a QA generator samples roughly 10-minute fragments; a verifier checks generated question-answer pairs against the full transcript; and evaluation questions are aggregated from five answer variants.

This matters because the target behavior is global consistency, not just local transcript lookup. The verifier step is meant to reduce front-loading bias and remove examples where generated intervals disagree with the full timestamped transcript.

Results and Analysis

The strongest evidence is in the temporal-grounding tables and length extrapolation plot. On the main comparison, the 60-second-anchor model reaches 53.8 mIoU on 20–40 minute temporal grounding, compared with 3.6 for Qwen3-Omni-30B and 56.1 for Gemini 3 Flash. With denser 7-second anchors, the same family improves to 65.2 mIoU on 20–40 minute grounding and 64.4 mIoU on 2–5 minute grounding. The trade-off is cost: Table 3 reports an added-token ratio of 16.0% at 7-second anchors versus 1.9% at 60-second anchors.

The ablation without inter-timings is the cleanest support for the paper’s claim. Removing temporal anchors worsens 20–40 minute grounding from 53.8 to 14.2 mIoU, timed description quality from 3.83 to 2.67 on the 20–40 minute bucket, and timed summarization round-segment error from 16.9 to 48.4, where lower is better. That is a large degradation, and it isolates the anchor mechanism more directly than comparisons against proprietary systems.

Length extrapolation is more nuanced. Figure 3 shows that training only on short audio does not transfer to long recordings, while training only on long audio hurts short-duration performance. The best long-range row is the 0–120 minute duration mixture, which reaches 59.6 mIoU on 80–120 minute evaluation and 65.5 mIoU on 60–80 minute evaluation, while retaining 40.6 mIoU on 0–1 minute audio. The paper’s interpretation is credible: duration mixture training is necessary because the answer often appears after multiple 10-minute fragments have passed.

The short-audio results are mixed. The model does not dominate AudioGrounding: TimeAudio reports 58.8 on the 7–10 second setting, while the authors’ 60-second model reports 45.1 and the 7-second variant reports 39.7. That caveat matters. GigaChat Audio is primarily a long-audio temporal grounding system, not a universal winner across short event localization.

Limitations

The evaluation depends heavily on synthetic QA and LLM-as-judge scoring for descriptions and summaries, even though the paper uses interval metrics where possible. Some baselines are incomplete across task columns, and proprietary systems limit reproducibility of head-to-head comparisons. The method also pays for denser anchors with extra tokens: 7-second timing improves mIoU but adds 16.0% timing tokens relative to a 160 ms audio-token frame rate.