MAPFusion: Enhancing RoBERTa for News Classification with Multi-Head Attention Pooling and Feature Fusion

Abstract

This paper proposes MAPFusion, a novel approach to enhance RoBERTa for news text classification using multi-head attention pooling. Traditional methods often rely on static pooling or simple averaging of [CLS] tokens, which can overlook subtle contextual information across token positions. The proposed method addresses this limitation with multi-head attention pooling (MAP), which captures context-aware representations by aggregating token-level embeddings with learned attention weights across multiple representation subspaces. The outputs from multiple attention heads are then integrated through a feature fusion layer to create a comprehensive sentence representation. This component is seamlessly integrated into the output layer of RoBERTa, preserving its pre-trained weights and requiring minimal additional parameters during fine-tuning. Experiments on benchmark datasets demonstrate that MAP-Fusion consistently outperforms baseline models, achieving significant improvements in classification accuracy. The framework is computationally efficient, broadly applicable to various text classification tasks, and provides a principled approach to more effectively utilize the latent representations of RoBERTa. Our work highlights the importance of adaptive feature aggregation in Transformer-based models, providing insights for future representation learning research. The code for this paper is available by contacting the respective authors.