Emotionally Expressive Attacks Expose Speech Deepfake Detection Failures
A 260-hour benchmark spanning 21 attacks and five emotions shows conventional detectors can approach chance-level performance under emotional spoofing.
Underlying Paper
AffectDF: The Most Comprehensive Benchmark for Speech Deepfake Detection against Emotionally Expressive Attacks
Speech deepfake detection (SDD) systems achieve strong performance on conventional benchmarks; however, existing datasets provide limited coverage of emotionally expressive and recent large audio-language model (LALM)-based attacks. Existing emotional spoofing datasets are also limited in scale and attack diversity, typically covering only voice conversion (VC) or text-to-speech (TTS) attacks. We introduce AffectDF, the most comprehensive benchmark for emotionally expressive speech deepfakes, spanning TTS, VC, emotional VC, and LALM-based spoofing attacks across both acted and spontaneous emotional speech. AffectDF contains approximately 260 hours of speech generated using 21 spoofing attacks across five emotional states. We benchmark state-of-the-art SDD systems under conventional and emotional spoofing conditions, including LALM-based detectors evaluated with both inference-only prompting and supervised fine-tuning. Our experiments reveal severe robustness degradation when models trained on conventional benchmarks are evaluated on AffectDF, with several systems approaching near-random performance. Surprisingly, even large-scale emotional training does not consistently improve cross-domain robustness, indicating that current SDD systems fail to learn generalized spoof representations under emotional and prosodic variability. Robustness further varies substantially across emotional states, attack families, and acted vs spontaneous emotional speech conditions. These findings expose fundamental limitations of current SDD systems and establish AffectDF as a benchmark for developing more robust spoof detection models.
Speech deepfake detectors are commonly assessed on benchmarks dominated by conventional text-to-speech and voice-conversion attacks. That setting can conceal a harder question: whether a detector recognizes spoofing itself or relies on artifacts that disappear when a generated voice carries varied prosody and emotion. AffectDF targets that gap with emotionally expressive synthetic speech from both acted and spontaneous source material, including recent large audio-language-model attacks.
Core Contribution
AffectDF is a benchmark rather than a new detector. Its contribution is breadth across generation families and emotional conditions: approximately 260 hours, 86,999 training samples, 23,330 development samples, and 175,468 test samples. It covers TTS, voice conversion, emotional voice conversion, and LALM-based emotional voice conversion, split across five emotional states. The test set contains 14 attack systems, while five and two systems appear in the training and development splits respectively.
The distinction from earlier emotional spoofing datasets is concrete. EmoFake contains 42 hours and five attacks; EmoSpoof-TTS contains 29 hours and three. AffectDF also extends beyond a single generation paradigm, and its held-out attack configuration is meant to test whether training produces a transferable spoof representation rather than familiarity with a known synthesizer. Figure 1 summarizes this dataset-to-evaluation design, including conventional and emotional evaluation conditions.
Technical Approach
The authors generate samples using multiple TTS, VC, EVC, and LALM-EVC systems. For the LALM condition, they use Qwen2.5-Omni and Kimi-Audio in ordinary prompt-controlled generation and in emotion-steered variants. The steered examples use emotion-sensitive-neuron masks identified from successful emotional voice-conversion instances; the mask is activated at inference time without updating model parameters. This creates a targeted stress test for detectors faced with explicit affect control rather than only prompt variation.
The benchmark evaluates conventional and self-supervised-learning detectors alongside LALM-based systems. The latter are tested through inference-only prompting and supervised fine-tuning, separating a detector's out-of-the-box behavior from performance after exposure to the benchmark. The paper also analyzes emotion with Emotion2Vec+-large. That is an automatic estimate rather than ground-truth relabeling: its seven output classes and an unknown class do not match AffectDF's five categories exactly.
Figure 2 shows why the benchmark is materially different from ASVspoof2019 and ASVspoof5. AffectDF has far less neutral speech and visibly larger happy and sad portions, while fear, disgust, and contempt remain comparatively small. The distribution supports the intended emotional diversity, but it does not establish emotion-label accuracy by itself.
Results and Analysis
The central result is a severe train–test mismatch. On ASVspoof2019 evaluation, AASIST trained on ASV19 records 0.83% EER and 0.027 minimum t-DCF; trained on AffectDF, its ASVspoof2019 EER rises to 44.52% with minimum t-DCF 1.00. The reverse direction is also damaging: the same detector trained on ASV19 reaches 23.16% EER on AffectDF, while an ASVspoof5-trained version reaches 23.16% EER. Other conventional systems deteriorate further on AffectDF: XLSR-SLS reaches 64.83% EER and RawNet2 43.07% when trained on AffectDF and evaluated on ASVspoof2019.
These results support the narrower conclusion that conventional benchmark success does not reliably transfer across emotional and attack-domain shifts. They do not show that every approach fails equally. Voxtral is an exception worth watching: its inference-only EER is 19.52% on AffectDF, versus 3.05% on ASV19, substantially better than the conventional systems in the reported AffectDF condition but still far from its conventional-benchmark result. Low-level differential cues also appear less separable: a logistic-regression probe trained on AffectDF gives 53.16% EER on AffectDF test data, compared with 33.24% on ASVspoof2019 evaluation. The evidence makes AffectDF useful as a failure-oriented evaluation set, though it leaves open which training objective can produce genuinely transferable detection.
Evidence Box
strongKey Claims
- •Emotionally expressive spoofing exposes poor transfer from conventional SDD benchmarks
- •Diverse attack families and emotion conditions require more general spoof representations
- •LALM-based detection can be evaluated under prompting and supervised fine-tuning
Key Results
- •AffectDF contains about 260 hours, 21 attacks, and 5 emotional states
- •AASIST reaches 23.16% EER on AffectDF after ASV19 or ASV5 training
- •XLSR-SLS records 64.83% EER on ASVspoof2019 after AffectDF training
- •Voxtral inference records 19.52% EER on AffectDF versus 3.05% on ASV19
Limitations & Caveats
- •Emotion2Vec+-large emotion analysis is an approximation, not ground-truth labeling
- •Emotion2Vec+ output categories do not exactly match AffectDF’s five emotions
- •Fear, disgust, and contempt occupy small portions of the reported emotion distribution
- •Benchmark results do not identify a training objective that delivers cross-domain transfer
