Does Vocal Emotion Help Zero-Shot Depression Detection? A Negative Result on DAIC-WOZ

Overview

DAIC-WOZ depression detection is usually posed as binary classification (or PHQ-8 regression) over patient-only transcripts, since interviewer turns leak diagnostic signal [1]. Zero-shot LLM prompting with DSM-anchored criteria has turned out to be a competitive baseline here, occasionally beating fine-tuned models [2,3,4]. Given the well-established link between depression and emotion regulation [5], the natural next question is whether adding an explicit emotion signal to the prompt improves this baseline. I tested this by running emotion2vec over every patient utterance and inserting the detected label into the transcript before feeding it to Qwen3-4B-Instruct (Qwen/Qwen3-4B-Instruct-2507), zero-shot, under a prompt that explicitly warns against equating emotion with diagnosis. Three conditions: no emotion, emotion label before the utterance, emotion label after the utterance.

Results

Adding emotion labels hurt on every metric except recall, which was already near ceiling. The confusion matrices make clear this isn’t noise: true negatives collapse while true positives barely move, so the model isn’t getting worse at the task in general — it’s shifting wholesale toward predicting “depressed.” That’s the exact behavior the prompt tells it not to do (emotion is supporting evidence only, not diagnostic). The instruction is in the prompt; it just isn’t governing the model’s actual output. Placement made a real difference too — putting the label before the utterance hurt roughly twice as much as putting it after, suggesting the tag is acting less like “evidence to weigh” and more like a frame that colors how the rest of the sentence gets read, an effect that fades once the sentence content is already processed.

Discussion

I don’t think this is specific to depression detection — it looks like ordinary LLM behavior under distraction. Shi et al. found that a single irrelevant sentence added to an otherwise-solvable math problem can collapse accuracy, and that telling the model to ignore irrelevant information only partially fixes it [6]; here the caveat did roughly the same — present, but insufficient. It’s also consistent with shortcut learning more broadly: given a hard, diffuse judgment (does this transcript show persistent depressive symptoms) next to a cheap, salient feature (an explicit emotion word repeated on nearly every line), the model has an easy substitute available and takes it. The caveat is stated once, at the top of a long prompt; the emotion tags recur on every line of the transcript that follows — repetition and recency of the concrete cue plausibly beats a single abstract instruction. On top of that, a model this size (4B) has less prompt-following headroom than a frontier model, so I’d expect this effect to be worse here than it would be on GPT-4-class models, though I haven’t tested that.

The comparison to Wu, Wu & Yu is useful because they got the opposite result using the same underlying signal. Their “climate and weather” argument is that depression is a session-level, long-run state while emotion is transient and utterance-level, and that conflating the two is the central risk of using emotion for this task [7]. But their fix isn’t a caveat — it’s architectural: they extract continuous emotion embeddings from a pretrained emotion model and train a bi-LSTM whose job is specifically to learn how much a given emotional moment should count toward the session-level label, and that learned emotion modality beats both audio-only and text-only baselines on its own before even being fused. The weighting is fit from labeled data. My setup asks a zero-shot model to invent that same weighting on the fly, per transcript, from a paragraph of prose telling it not to overreact to emotion — and it overreacts to emotion anyway. The gap between the two results looks less like “emotion doesn’t help depression detection” and more like “emotion helps when something learns how to weight it, and doesn’t when a prompt just asserts how it should be weighted.”

Conclusion

If I were to push this further, I’d try getting the model to produce a session-level emotion summary first (e.g., “mostly neutral, three sad moments, one angry moment”) and only then reason about depression from that summary plus the transcript — forcing the aggregation step Wu et al. learn explicitly to happen before the diagnostic judgment, instead of asking the model to do both at once, inline, per utterance.

References

[1] DeVault, David, et al. “SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support.” Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS ‘14), International Foundation for Autonomous Agents and Multiagent Systems, 2014, pp. 1061–1068. [2] Chen, Feng, et al. “Detecting PTSD in Clinical Interviews: A Comparative Analysis of NLP Methods and Large Language Models.” arXiv, arXiv:2504.01216, 2025, arxiv.org/abs/2504.01216. [3] Teferra, Bazen Gashaw, et al. “Leveraging Large Language Models for Automated Depression Screening.” PLOS Digital Health, vol. 4, no. 7, 2025, e0000943, doi:10.1371/journal.pdig.0000943. [4] Merzougui, Dhia Eddine, et al. “Evaluating Large Language Models for Depression Symptom Estimation.” Artificial Intelligence in Medicine (AIME 2025), edited by Riccardo Bellazzi et al., Lecture Notes in Computer Science, vol. 15735, Springer, 2025, pp. 272–276. [5] Joormann, Jutta, and Meghan E. Quinn. “Cognitive Processes and Emotion Regulation in Depression.” Depression and Anxiety, vol. 31, no. 4, 2014, pp. 308–315. [6] Shi, Freda, et al. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” Proceedings of the 40th International Conference on Machine Learning, PMLR, 2023. [7] Wu, Wen, et al. “Climate and Weather: Inspecting Depression Detection via Emotion Recognition.” arXiv, arXiv:2204.14099, 2022, arxiv.org/abs/2204.14099.