北京航空航天大学学报 ›› 2021, Vol. 47 ›› Issue (3): 469-477.doi: 10.13700/j.bh.1001-5965.2020.0451

• 论文 • 上一篇    下一篇

基于语义相关的多模态社交情感分析

胡慧君1,2, 冯梦媛1,2, 曹梦丽1,2, 刘茂福1,2   

  1. 1. 武汉科技大学 计算机科学与技术学院, 武汉 430065;
    2. 智能信息处理与实时工业系统湖北省重点实验室, 武汉 430081
  • 收稿日期:2020-08-24 发布日期:2021-04-08
  • 通讯作者: 胡慧君 E-mail:huhuijun@wust.edu.cn
  • 作者简介:胡慧君,女,博士,副教授,硕士生导师。主要研究方向:智能信息处理、图像分析与理解;冯梦媛,女,硕士研究生。主要研究方向:多模态情感分析;曹梦丽,女,硕士研究生。主要研究方向:多模态情感分析;刘茂福,男,博士,教授,博士生导师。主要研究方向:自然语言处理、图像分析与理解。
  • 基金资助:
    国家社科基金重大研究计划(11&ZD189);全军共用信息系统装备预先研究项目(31502030502)

Multimodal social sentiment analysis based on semantic correlation

HU Huijun1,2, FENG Mengyuan1,2, CAO Mengli1,2, LIU Maofu1,2   

  1. 1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan 430065, China;
    2. Hubei Provincial Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan 430081, China
  • Received:2020-08-24 Published:2021-04-08
  • Supported by:
    Major Projects of National Social Science Foundation of China (11&ZD189); Pre-research Foundation of Whole Army Shared Information System Equipment (31502030502)

摘要: 社交平台允许用户采用多种信息模态发表意见与观点,多模态语义信息融合能够更有效地预测用户所表达的情感倾向。因此,多模态情感分析近年来受到了广泛关注。然而,多模态情感分析中视觉与文本存在的语义无关问题,导致情感分析效果不佳。针对这一问题,提出了基于语义相关的多模态社交情感分析(MSSA-SC)方法。采用图文语义相关性分类模型,对图文社交信息进行语义相关性识别,若图文语义相关,则对图文社交信息使用图文语义对齐多模态模型进行图文特征融合的情感分析;若图文语义无关,则仅对文本模态进行情感分析。在真实社交媒体数据集上进行了实验,由实验结果可知,所提方法能够有效降低图文语义无关情况对多模态社交媒体情感分析的影响。与此同时,所提方法的Accuracy和Macro-F1指标分别为75.23%和70.18%,均高于基准模型。

关键词: 多模态, 社交媒体, 情感分析, 语义相关, 图文特征融合

Abstract: Social platforms allow users to express opinions in a variety of information modalities, and multi-modal semantic information fusion can more effectively predict the emotional tendencies expressed by users. Therefore, multimodal sentiment analysis has received extensive attention in recent years. However, in multi-modal sentiment analysis, there is a problem of unrelated semantics between vision and text, resulting in poor sentiment analysis. In order to solve this problem, this paper proposes the Multimodal Social Sentiment Analysis based on Semantic Correlation (MSSA-SC) method. The MSSA-SC firstly adopts the semantic relevance classification model of image and text to identify the semantic relevance of the image-text social media. If the image and text are semantically related, the image and text semantic alignment multimodal model is used for the image-text feature fusion for the image-text social media sentiment analysis. When the image and text semantics are irrelevant, only the sentiment analysis is performed on the text modality. The experimental results on real social media datasets show that the MSSA-SC method can effectively reduce the influence of unrelated image and text semantics on multimodal social sentiment analysis. Moreover, the Accuracy and Macro-F1 of the MSSA-SC method are 75.23% and 70.18%, respectively, and outperform those of the benchmark model.

Key words: multimodal, social media, sentiment analysis, semantic correlation, image-text feature fusion

中图分类号: 


版权所有 © 《北京航空航天大学学报》编辑部
通讯地址:北京市海淀区学院路37号 北京航空航天大学学报编辑部 邮编:100191 E-mail:jbuaa@buaa.edu.cn
本系统由北京玛格泰克科技发展有限公司设计开发