留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于语义相关的多模态社交情感分析

胡慧君 冯梦媛 曹梦丽 刘茂福

胡慧君, 冯梦媛, 曹梦丽, 等 . 基于语义相关的多模态社交情感分析[J]. 北京航空航天大学学报, 2021, 47(3): 469-477. doi: 10.13700/j.bh.1001-5965.2020.0451
引用本文: 胡慧君, 冯梦媛, 曹梦丽, 等 . 基于语义相关的多模态社交情感分析[J]. 北京航空航天大学学报, 2021, 47(3): 469-477. doi: 10.13700/j.bh.1001-5965.2020.0451
HU Huijun, FENG Mengyuan, CAO Mengli, et al. Multimodal social sentiment analysis based on semantic correlation[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 469-477. doi: 10.13700/j.bh.1001-5965.2020.0451(in Chinese)
Citation: HU Huijun, FENG Mengyuan, CAO Mengli, et al. Multimodal social sentiment analysis based on semantic correlation[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 469-477. doi: 10.13700/j.bh.1001-5965.2020.0451(in Chinese)

基于语义相关的多模态社交情感分析

doi: 10.13700/j.bh.1001-5965.2020.0451
基金项目: 

国家社科基金重大研究计划 11&ZD189

全军共用信息系统装备预先研究项目 31502030502

详细信息
    作者简介:

    胡慧君   女,博士,副教授,硕士生导师。主要研究方向:智能信息处理、图像分析与理解

    冯梦媛   女,硕士研究生。主要研究方向:多模态情感分析

    曹梦丽   女,硕士研究生。主要研究方向:多模态情感分析

    刘茂福   男,博士,教授,博士生导师。主要研究方向:自然语言处理、图像分析与理解

    通讯作者:

    胡慧君, E-mail: huhuijun@wust.edu.cn

  • 中图分类号: TP37

Multimodal social sentiment analysis based on semantic correlation

Funds: 

Major Projects of National Social Science Foundation of China 11&ZD189

Pre-research Foundation of Whole Army Shared Information System Equipment 31502030502

More Information
  • 摘要:

    社交平台允许用户采用多种信息模态发表意见与观点,多模态语义信息融合能够更有效地预测用户所表达的情感倾向。因此,多模态情感分析近年来受到了广泛关注。然而,多模态情感分析中视觉与文本存在的语义无关问题,导致情感分析效果不佳。针对这一问题,提出了基于语义相关的多模态社交情感分析(MSSA-SC)方法。采用图文语义相关性分类模型,对图文社交信息进行语义相关性识别,若图文语义相关,则对图文社交信息使用图文语义对齐多模态模型进行图文特征融合的情感分析;若图文语义无关,则仅对文本模态进行情感分析。在真实社交媒体数据集上进行了实验,由实验结果可知,所提方法能够有效降低图文语义无关情况对多模态社交媒体情感分析的影响。与此同时,所提方法的Accuracy和Macro-F1指标分别为75.23%和70.18%,均高于基准模型。

     

  • 图 1  MSSA-SC方法示意

    Figure 1.  Schematic diagram of MSSA-SC method

    图 2  图文关联的文本语义单元提取方法架构

    Figure 2.  Architecture of text semantic unit extraction method based on image-text correlation

    图 3  图像分类模型提取的语义标签

    Figure 3.  Semantic tag extracted by image classification model

    图 4  SAMRoBERTa和SRMRoBERTa模型结构

    Figure 4.  Structure of SAMRoBERTa and SRMRoBERTa model

    图 5  图文社交信息的图像特征权重热力图

    Figure 5.  Heat map of image feature weight of

    表  1  图文语义相关性数据集的数据分布

    Table  1.   Data distribution of image-text semantic correlation datasets

    标签 微博图文对数
    P 7 442
    N 2 375
    下载: 导出CSV

    表  2  微博图文情感分析数据集的数据分布

    Table  2.   Data distribution of image-text microblog sentiment analysis datasets

    标签 测试样例 训练样例
    积极 981 4 660
    中性 2 404 9 567
    消极 615 1 903
    下载: 导出CSV

    表  3  实验参数设置

    Table  3.   Experimental parameter setting

    参数 数值
    最大句子长度 256
    批处理个数 12
    学习率 2×10-5
    自注意力机制头数 6
    预热学习率 0.1
    全部样本训练次数 4.0
    下载: 导出CSV

    表  4  微博图文情感分类实验结果

    Table  4.   Experimental results of image-text microblog sentiment classification

    模态 方法 Accuracy/% Macro-F1/%
    图像 ResNet 58.91 45.27
    文本 BiLSTM 70.92 63.48
    BERT 73.25 68.43
    RoBERTa 73.77 69.59
    图文 Multi-CNN 68.31 62.46
    MBERT 74.52 68.70
    SAMRoBERTa 74.70 69.65
    MSSA-SC 75.23 70.18
    下载: 导出CSV

    表  5  图文情感分析数据集中的图文数据实例

    Table  5.   Image-text samples from image-text sentiment analysis dataset

    图文数据实例 实际情感标签 SRMRoBERTa图文语义相关性分类
    消极 图文语义不相关
    下载: 导出CSV
  • [1] 王英, 龚花萍. 基于情感维度的大数据网络舆情情感倾向性分析研究-以"南昌大学自主保洁"微博舆情事件为例[J]. 情报科学, 2017, 35(4): 40-45. https://www.cnki.com.cn/Article/CJFDTOTAL-QBKX201704007.htm

    WANG Y, GONG H P. Research on the sentiment tendency of big data network public opinion sentiment based on the sentiment dimension-Taking the "Nanchang University Independent Cleaning" Weibo public opinion event as an example[J]. Information Science, 2017, 35(4): 40-45(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-QBKX201704007.htm
    [2] SENG J K, ANG K L. Multimodal emotion and sentiment modeling from unstructured big data: Challenges, architecture & techniques[J]. IEEE Access, 2019, 7: 90982-90998. doi: 10.1109/ACCESS.2019.2926751
    [3] GHOSAL D, AKHTAR S, CHAUHAN D, et al. Contextual inter-modal attention for multi-modal sentiment analysis[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL Press, 2018: 3454-3466.
    [4] NIE L, WANG W, HONG R, et al. Multimodal dialog system: Generating responses via adaptive decoders[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York: ACM Press, 2019: 1098-1106.
    [5] LIU Y, OTT M, GOYAL N, et al. RoBERTa: A robustly optimized BERT pretraining approach[EB/OL]. (2019-07-26)[2020-08-01]. https://arxiv.org/abs/1907.11692.
    [6] LIU M, ZHANG L, LIU Y, et al. Recognizing semantic correlation in image-text Weibo via feature space mapping[J]. Computer Vision and Image Understanding, 2017, 163: 58-66. doi: 10.1016/j.cviu.2017.04.012
    [7] XIAO Z, LIANG P. Chinese sentiment analysis using bidirectional LSTM with word embedding[C]//International Conference on Cloud Computing and Security. Berlin: Springer, 2016: 601-610.
    [8] ZHENG J, CHEN X, DU Y, et al. Short text sentiment analysis of micro-blog based on BERT[C]//MUE 2019, FutureTech 2019: Advanced Multimedia and Ubiquitous Engineering. Berlin: Springer, 2019: 390-396.
    [9] DEVLIN J, CHANG M, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-10-11)[2020-08-01]. https: //arxiv.org/abs/1810.04805.
    [10] MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM Press, 2010: 83-92.
    [11] JINDAL S, SINGH S. Image sentiment analysis using deep convolutional neural networks with domain specific fine tuning[C]//2015 International Conference on Information Processing. Piscataway: IEEE Press, 2015: 447-451.
    [12] YOU Q, LUO J, JIN H, et al. Joint visual-textual sentiment analysis with deep neural networks[C]//Proceedings of the 23rd ACM International Conference on Multimedia. New York: ACM Press, 2015: 1071-1074.
    [13] ZHU X, CAO B, XU S, et al. Joint visual-textual sentiment analysis based on cross-modality attention mechanism[C]//Proceedings of the International Conference on Multimedia Modeling. Berlin: Springer, 2019: 264-276.
    [14] 缪裕青, 汪俊宏, 刘同来, 等. 图文融合的微博情感分析方法[J]. 计算机工程与设计, 2019, 40(4): 1099-1105. https://www.cnki.com.cn/Article/CJFDTOTAL-SJSJ201904032.htm

    MIAO Y Q, WANG J H, LIU T L, et al. Joint visual-textual approach for microblog sentiment analysis[J]. Computer Engineering and Design, 2019, 40(4): 1099-1105(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-SJSJ201904032.htm
    [15] ZHAO Z Y, ZHU H Y, XUE Z H, et al. An image-text consistency driven multimodal sentiment analysis approach for social media[J]. Information Processing and Management, 2019, 56(6): 102097. doi: 10.1016/j.ipm.2019.102097
    [16] 蔡国永, 吕光瑞, 徐智. 基于层次化深度关联融合网络的社交媒体情感分类[J]. 计算机研究与发展, 2019, 56(6): 1312-1324. https://www.cnki.com.cn/Article/CJFDTOTAL-JFYZ201906019.htm

    CAI G Y, LV G R, XU Z. A hierarchical deep correlation fusion network for sentiment classification in social media[J]. Journal of Computer Research and Development, 2019, 56(6): 1312-1324(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-JFYZ201906019.htm
    [17] LI W, ZHAO J. TextRank algorithm by exploiting Wikipedia for short text keywords extraction[C]//2016 3rd International Conference on Information Science and Control Engineering. Piscataway: IEEE Press, 2016: 683-686.
    [18] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2016: 770-778.
    [19] CHOLLET F. Xception: Deep learning with depthwise separable convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 1251-1258.
    [20] MIKOLOV T, CHEN K, CORRADO G S, et al. Efficient estimation of word representations in vector space[EB/OL]. (2013-01-16)[2020-08-01]. https://arxiv.org/abs/1301.3781.
    [21] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5998-6008.
    [22] CAI G, XIA B. Convolutional neural networks for multimedia sentiment analysis[C]//NLPCC 2015: Natural Language Processing and Chinese Computing. Berlin: Springer, 2015: 159-167.
    [23] YU J, JIANG J. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2019: 5408-5414.
  • 加载中
图(5) / 表(5)
计量
  • 文章访问数:  365
  • HTML全文浏览量:  12
  • PDF下载量:  233
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-08-24
  • 录用日期:  2020-09-11
  • 刊出日期:  2021-03-20

目录

    /

    返回文章
    返回
    常见问答