Multimodal Speech Emotion Recognition on IEMOCAP using Wav2Vec2 and BERT

Overview

This project investigates multimodal speech emotion recognition (SER) by combining pretrained speech and text encoders on the IEMOCAP dataset. The speech modality is represented using Wav2Vec2, while the textual modality is encoded using BERT. To focus on the effectiveness of different fusion strategies rather than feature learning, both pretrained encoders were kept frozen throughout all experiments, and only the fusion and classification layers were trained.

Experiments were conducted using the 5-class IEMOCAP setting consisting of anger (ang), happiness (hap), excitement (exc), neutral (neu), and sadness (sad). Four architectural variants were evaluated to study the contributions of two components:

• Attention Pooling, which replaces simple mean/CLS pooling with learnable attention-based aggregation.
• Cross-Modal Attention, which enables interaction between speech and text representations before classification.

The baseline model (FF) uses mean pooling for Wav2Vec2 features, the BERT [CLS] token as the text representation, and simple feature concatenation. The remaining configurations progressively introduce attention pooling and cross-modal attention.

Experimental Results

WA: Weighted Accuracy, UA: Unweighted Accuracy (macro recall), F1: Macro F1-score.

Discussion

The baseline model achieved a weighted accuracy of 60.33%, demonstrating that frozen pretrained representations from Wav2Vec2 and BERT already provide a strong foundation for emotion recognition. However, the simple fusion strategy leaves considerable room for improvement.

Introducing attention pooling alone (TF) improved all evaluation metrics by approximately 5 percentage points, increasing weighted accuracy from 60.33% to 66.18%. This suggests that learnable pooling is more effective than uniform averaging, as it allows the model to focus on emotionally informative speech frames and important textual tokens.

A substantially larger improvement was obtained by incorporating cross-modal attention (FT). Weighted accuracy increased to 71.10%, while both Macro F1 and Unweighted Accuracy exceeded 70%. These results indicate that allowing the speech and text modalities to interact before classification produces richer multimodal representations than simple concatenation. Emotion recognition is inherently multimodal, and cross-modal attention enables the model to exploit complementary information between acoustic cues and linguistic content.

The complete model (TT), combining both attention pooling and cross-modal attention, achieved the highest Macro F1 (70.51%) and Unweighted Accuracy (70.14%), while weighted accuracy (71.03%) remained essentially identical to the cross-modal-only configuration. The negligible difference between FT and TT suggests that, when pretrained encoders are frozen, cross-modal interaction contributes most of the performance gain, whereas attention pooling provides only a marginal additional benefit.

Conclusion

This study demonstrates that the choice of multimodal fusion architecture has a significant impact on speech emotion recognition performance, even when the underlying pretrained encoders remain fixed. Cross-modal attention proved to be the most influential architectural component, improving weighted accuracy by more than 10 percentage points over the baseline. Attention pooling also contributed positively, although its impact was smaller once cross-modal attention was introduced.

These findings indicate that pretrained Wav2Vec2 and BERT embeddings already contain rich emotional information, and the primary challenge lies in effectively integrating the two modalities. Future work will compare this Wav2Vec2–BERT framework with CLAP-based multimodal representations under the same experimental protocol to further investigate the effectiveness of joint audio-text representation learning for speech emotion recognition.