TurnBench Exposes Turn-Taking Failures Across Conversation Types

Voice agents must decide whether a silence marks a finished turn, a mid-turn pause, or a moment for a short backchannel. Existing evaluations often collapse those cases or use incompatible labels and metrics, making it difficult to compare systems on the social timing that makes dialogue feel natural. TurnBench introduces a shared evaluation protocol and a hand-labeled, multi-domain test set intended to make those errors visible.

The paper evaluates 14 systems across six conversation types: Argumentative, Casual, Collaborative, Instructional, Narrative, and Task-Oriented dialogue. Its central result is not that one model solves turn-taking. End-of-turn recall remains comparatively stable across interaction styles, whereas interruption false positives vary sharply with conversational style and are concentrated where backchannels are common. The benchmark therefore shifts the question from whether a model can detect speech boundaries to whether it can distinguish an actual floor transfer from speech that should not trigger an interruption.

Core Contribution

TurnBench pairs a 30-hour corpus of English, studio-recorded dyadic conversations with triple annotation and a standardized scoring procedure for end-of-turn (EOT) and interruption (INT) events. The authors also release a 104-hour training set, while holding test labels back to limit contamination. Rather than treating all speech onset as a turn-taking event, the protocol scores each decision in positive and negative temporal windows around annotated events. Predictions outside those windows are ignored; firing inside a scored negative window counts as a false positive.

That framing matters because a quick interruption detector can appear responsive merely by speaking over pauses and backchannels. TurnBench makes that trade-off explicit through recall, false-positive rate (FPR), and signed latency relative to the gold event. Submissions are ranked subject to an FPR ceiling, with a looser 0.1 budget used for development-oriented comparison. The design gives an operational meaning to an error that aggregate endpointing scores can hide.

Technical Approach

The benchmark covers rule-based voice activity detection, commercial and open tools, learned turn-taking predictors, and full-duplex conversational models. Rule-based RMS VAD fires on channel energy and has no linguistic information. OpenAI Realtime's server and semantic VAD modes, Kyutai SVAD, and SmartTurn v3 represent deployed endpointing or turn-detection tools. ESPnet Switchboard and Mini-base predictors provide learned baselines, while WavLM-Large variants combine acoustic representations with EOT and interruption heads.

For systems emitting continuous probabilities, the authors tune EOT and INT thresholds independently on the development set. The selected operating point is the threshold with the highest recall inside the 0.1 FPR budget, then fixed for test evaluation. Figure 2 illustrates the consequence for VAP interruption decisions: increasing the threshold lowers FPR but raises median latency. At the selected $\theta=0.86$, recall is maximized under the stated development-set budget rather than optimized for unconstrained responsiveness.

Full-duplex models are placed in live conversations and scored only on EOT, since their model-generated speech can make the absence of an interruption hard to guarantee once an event occurs. This is a sensible scope decision, but it leaves the benchmark's most difficult distinction only partially covered by the model class most likely to act conversationally.

Results and Analysis

VAP's pooled results report 0.884 EOT recall at 0.055 FPR with 368 ms median latency, and 0.945 interruption recall at 0.107 FPR with 994 ms median latency. The interruption FPR is slightly above the 0.1 development budget used to select its threshold, illustrating that a development-set operating constraint does not guarantee the same result on pooled test evaluation. The interruption result is high-recall but slow: nearly one second after the relevant onset is a poor fit for many interactive speaking settings. Its EOT performance is more practically balanced, though it still does not reproduce human timing.

The human reference is the more demanding comparison. In smooth floor transfers, human listeners begin speaking a median 151 ms before the current turn ends. No evaluated system is reported as matching that anticipatory behavior without excessive false positives. SmartTurn v3 and ESPnet predictors commit earlier, at 159 ms and 210 ms respectively, but the paper reports that their early decisions carry much higher interruption FPRs. VAP, Mini-EP, and WavLM-Large delay past speech onset to discriminate more carefully, reaching 0.87–0.95 interruption recall at 0.05–0.11 FPR but with 559–1,076 ms median latency.

The evidence supports TurnBench as a useful diagnostic benchmark: it exposes a concrete latency-selectivity frontier and shows that conversation type changes measured interruption quality. It does not yet establish a general solution for real-world spoken agents. The evaluation is English-only, studio-recorded, and dyadic, and its full-duplex methodology does not assess interruption behavior directly.