Streaming ASR Unifies Transcription and Speaker Attribution
VibeVoice-ASR-Streaming interleaves audio, lookahead, and prior text so one model can produce speaker-attributed transcripts as speech arrives.
Underlying Paper
VibeVoice-ASR-Streaming Technical Report
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Speaker-attributed automatic speech recognition has traditionally combined transcription with a separate speaker-diarization system. That division can be awkward for live applications, where a system must produce both words and an indication of who said them. VibeVoice-ASR-Streaming instead uses a single language-model-based system to generate speaker-attributed text as audio arrives.
The paper presents streaming speaker-attributed ASR as a unified task rather than a pipeline of recognition followed by diarization. Its 7B model is evaluated on AliMeeting, AISHELL-4, AMI-SDM, AMI-IHM, and the multilingual MLC-Challenge. The authors report the lowest average recognition error across those five evaluation sets, along with the best or tied-best speaker-attribution result in 12 of 13 settings.
Core Contribution
VibeVoice-ASR-Streaming constructs one autoregressive context from fixed-size audio chunks, a small amount of future audio, and previously generated speaker-attributed text. This lets the model use earlier conversational context while producing text for incoming speech, without a separate diarization stage.
The design includes a bounded lookahead window. The reported configuration uses four frames, or 0.5 seconds, of lookahead after each audio chunk before generating that chunk's text. This is a deliberate trade-off: the system is designed for streaming use rather than full-recording offline processing, while still using limited future context to support recognition and speaker attribution.
Technical Approach
Figure 2 illustrates the alternating context structure. Speech chunks and speaker-attributed text chunks are interleaved in a single autoregressive sequence. A fixed -frame lookahead follows each speech chunk before the associated text is generated.
By keeping prior speaker-attributed text in context, the model can carry conversational history forward as it processes new audio. The resulting output directly represents both the transcript and speaker attribution, rather than requiring a downstream component to align a separate diarization result with recognized words.
Results and Analysis
Figure 1 compares recognition error for VibeVoice-ASR-Streaming-7B with four deployed streaming ASR systems across four meeting benchmarks and MLC-Challenge. AliMeeting and AISHELL-4 are evaluated with character error rate, while AMI-SDM and AMI-IHM use word error rate. For MLC-Challenge, the reported score is a macro average across nine evaluated languages.
The paper's central empirical result is broad rather than limited to one benchmark: the 7B model achieves the lowest average WER/CER across the five evaluation sets. It also reports the best or tied-best speaker-attribution outcome in 12 of 13 evaluation settings. These results support the paper's claim that a unified streaming model can jointly handle recognition and speaker attribution across meeting and multilingual evaluation conditions.
The available evidence is strongest for the selected benchmarks and metrics. It does not, by itself, establish performance for every deployment setting, hardware configuration, or conversational domain. Production use would still require evaluation on the target language mix, acoustic conditions, and latency requirements.
Practical Limits
The system uses 0.5 seconds of lookahead, so it is not a zero-lookahead recognizer. Its reported results also concern the paper's chosen meeting and multilingual benchmarks. For real-time meeting transcription or voice-agent logging, the main architectural benefit is that one model produces both transcript text and speaker attribution; deployment decisions still depend on application-specific latency, reliability, and cost measurements.
Evidence Box
strongKey Claims
- •End-to-end streaming ASR can jointly emit transcript text and speaker attribution
- •Interleaving audio, lookahead, and prior text supports streaming speaker-attributed recognition
- •A 7B model achieves strong recognition and speaker-attribution results on selected benchmarks
Key Results
- •Lowest average WER/CER for the 7B model across five evaluation sets
- •Best or tied-best speaker attribution in 12 of 13 evaluation settings
- •Four frames of lookahead correspond to 0.5 seconds before text generation
Limitations & Caveats
- •The reported evidence is limited to the selected meeting benchmarks and MLC-Challenge
- •The supplied material does not establish deployment latency, serving cost, or performance in all domains
- •The streaming design uses bounded lookahead rather than zero-lookahead recognition
