基于注意力机制与特征相关性的人脸表情识别

doi:10.13700/j.bh.1001-5965.2020.0507

北京航空航天大学学报 ›› 2022, Vol. 48 ›› Issue (1): 147-155.doi: 10.13700/j.bh.1001-5965.2020.0507

基于注意力机制与特征相关性的人脸表情识别

兰凌强, 刘淇缘, 卢树华

中国人民公安大学信息网络安全学院, 北京 102600

收稿日期:2020-09-09 发布日期:2022-01-26
通讯作者: 卢树华 E-mail:lushuhua@ppsuc.edu.cn
基金资助:
国家重点研发计划（2016YFC0801005）；中央高校基本科研业务经费项目（2019JKF225）；公共安全行为科学实验室开放课题（2020SYS16）

Facial expression recognition based on attention mechanism and feature correlation

LAN Lingqiang, LIU Qiyuan, LU Shuhua

College of Information and Cyber Security, People's Public Security University of China, Beijing 102600, China

Received:2020-09-09 Published:2022-01-26

摘要/Abstract

摘要： 针对自然条件下人脸表情识别面临遮挡、光照、姿势变化等挑战，存在识别准确率低的问题，提出了一种新的深度学习网络模型用于人脸表情识别。以ResNet为基础网络，融合了瓶颈注意力机制及全局二阶池化层，其中瓶颈注意力机制专注于表情重要特征的提取，全局二阶池化层度量表情特征之间的相关性，在此基础上通过联合正则化策略，平衡和改善特征数据分布情况，提高表情识别准确率。所提方法在2个公开数据集FER2013和CK+进行了测试及验证，最高准确率分别达到了74.227%和95.8%，性能优于诸多现存的主流方法，表明所提模型具有较好的准确性和鲁棒性。

关键词: 表情识别, 深度学习, 瓶颈注意力机制, 全局二阶池化层, 联合正则化策略

Abstract: There are many challenges including occlusion, illumination and posture variation in the facial expression recognition under natural conditions, leading to the low accuracy. This paper proposes a new deep learning network model for facial expression recognition. This model uses ResNet as backbone, and introduces the bottleneck attention module and the global second-order pooling layer. The bottleneck attention module focuses on the extraction of important expression features, and the global second-order pooling layer aims to measure the correlation among expression features. On this basis, the joint normalization strategies are used balance and improve the distribution of feature data, which improves the accuracy of expression recognition. The test and validation of the proposed method have been carried out on the two public datasets FER2013 and CK+, resulting in the highest accuracy rates of 74.227% and 95.8%, respectively. The performance is better than most of the latest facial expression recognition methods. The results indicate that this model has better accuracy and robustness.

Key words: expression recognition, deep learning, bottleneck attention module, global second-order pooling layer, joint normalization strategies

中图分类号:

TP391

兰凌强, 刘淇缘, 卢树华. 基于注意力机制与特征相关性的人脸表情识别[J]. 北京航空航天大学学报, 2022, 48(1): 147-155.

LAN Lingqiang, LIU Qiyuan, LU Shuhua. Facial expression recognition based on attention mechanism and feature correlation[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(1): 147-155.

参考文献

[1] TAHA B,HATZINAKOS D.Emotion recognition from 2D facial expressions[C]//2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE).Piscataway: IEEE Press,2019:1-4.
[2] LIU K C,HSU C C,WANG W Y,et al.Real-time facial expression recognition based on CNN[C]//2019 International Conference on System Science and Engineering (ICSSE).Piscataway:IEEE Press,2019:120-123.
[3] CHA H S,CHOI S J,IM C H.Real-time recognition of facial expressions using facial electromyograms recorded around the eyes for social virtual reality applications[J].IEEE Access,2020,8:62065-62075.
[4] PANG L,LI N Q,ZHAO L,et al.Facial expression recognition based on Gabor feature and neural network[C]//2018 International Conference on Security,Pattern Analysis and Cybernetics (SPAC).Piscataway:IEEE Press,2018:489-493.
[5] KUSHWAH K,SHARMA V,SINGH U.Neural network method through facial expression recognition[C]//2017 International conference of Electronics,Communication and Aerospace Technology (ICECA).Piscataway:IEEE Press,2017:532-537.
[6] LI J,JIN K,ZHOU D L,et al.Attention mechanism-based CNN for facial expression recognition[J].Neurocomputing,2020,411:340-350.
[7] XIANG J,ZHU G M.Joint face detection and facial expression recognition with MTCNN[C]//2017 4th International Conference on Information Science and Control Engineering (ICISCE).Piscataway:IEEE Press,2017:424-427.
[8] ZHOU Y,FENG Y Y,ZENG S Y,et al.Facial expression recognition based on convolutional neural network[C]//2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS).Piscataway:IEEE Press,2019:410-413.
[9] ZENG G H,ZHOU J C,JIA X,et al.Hand-crafted feature guided deep learning for facial expression recognition[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).Piscataway:IEEE Press,2018:423-430.
[10] PARK J C,WOO S,LEE J Y,et al.BAM:Bottleneck attention module[EB/OL].(2018-07-18)[2020-09-02].https://arxiv.org/pdf/1807.06514.
[11] GAO Z L,XIE J T,WANG Q L,et al.Global second-order pooling convolutional networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway:IEEE Press,2019:3019-3028.
[12] 兰凌强,李欣,刘淇缘,等.基于联合正则化策略的人脸表情识别方法[J].北京航空航天大学学报,2020,46(9):1797-1806.LAN L Q,LI X,LIU Q Y,et al.Facial expression recognition method based on a joint normalization strategy[J].Journal of Beijing University of Aeronautics and Astronautics,2020,46(9):1797-1806(in Chinese).
[13] GOODFELLOW I J,ERHAN D,CARRIER P L,et al.Challenges in representation learning:A report on three machine learning contests[C]//International Conference on Neural Information Processing.Berlin:Springer,2013:117-124.
[14] LUCEY P,COHN J F,KANADE T,et al.The extended Cohn-Kanade dataset (CK+):A complete dataset for action unit and emotion-specified expression[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-workshops.Piscataway:IEEE Press,2010:94-101.
[15] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].(2015-10-04)[2020-09-02].https://arxiv.org/abs/1409.1556.
[16] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[17] HE K M,ZHANG X Y,REN S Q,et al.Identity mappings in deep residual networks[M]//Computer Vision ECCV 2016.Berlin:Springer,2016:630-645.
[18] CHENG S,ZHOU G H.Facial expression recognition method based on improved VGG convolutional neural network[J].International Journal of Pattern Recognition and Artificial Intelligence,2020,34(7):2056003.
[19] ZHONG Y X,QIU S H,LUO X S,et al.Facial expression recognition based on optimized ResNet[C]//2020 2nd World Symposium on Artificial Intelligence (WSAI).Piscataway:IEEE Press,2020:84-91.
[20] HU J,SHEN L,ALBANIE S,et al.Squeeze-and-excitation networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,42(8):2011-2023.
[21] CHEN Y Z,HU H F.Facial expression recognition by inter-class relational learning[J].IEEE Access,2019,7:94106-94117.
[22] WANG F J,SHEN L P.Expression recognition using region features and facial action units[C]//2019 15th International Conference on Intelligent Environments.Piscataway:IEEE Press,2019:9-15.
[23] XU Q T,ZHAO N J.A facial expression recognition algorithm based on CNN and LBP feature[C]//2020 IEEE 4th Information Technology,Networking,Electronic and Automation Control Conference (ITNEC).Piscataway:IEEE Press,2020:2304-2308.
[24] WANG K,PENG X J,YANG J F,et al.Region attention networks for pose and occlusion robust facial expression recognition[J].IEEE Transactions on Image Processing,2020,29:4057-4069.
[25] LI Y,ZENG J B,SHAN S G,et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing,2019,28(5):2439-2450.
[26] GAN Y L,CHEN J Y,YANG Z K,et al.Multiple attention network for facial expression recognition[J].IEEE Access,2020,8:7383-7393.
[27] YU K C,SALZMANN M.Second-order convolutional neural networks[EO/OL].(2017-03-20)[2020-09-02].http://export.arxiv.org/abs/1703.06817.
[28] LI P H,XIE J T,WANG Q L,et al.Is second-order information helpful for large-scale visual recognition?[C]//2017 IEEE International Conference on Computer Vision (ICCV).Piscataway:IEEE Press,2017:2089-2097.
[29] LI P H,XIE J T,WANG Q L,et al.Towards faster training of global covariance pooling networks by iterative matrix square root normalization[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2018:947-955.
[30] FEI Z X,YANG E F,LI D,et al.Combining deep neural network with traditional classifier to recognize facial expressions[C]//2019 25th International Conference on Automation and Computing (ICAC).Piscataway:IEEE Press,2019:1-6.
[31] ADIL B,NADJIB K M,YACINE L.A novel approach for facial expression recognition[C]//2019 International Conference on Networking and Advanced Systems (ICNAS).Piscataway:IEEE Press,2019:1-5.
[32] SUN X,XIA P P,ZHANG L M,et al.A ROI-guided deep architecture for robust facial expressions recognition[J].Information Sciences,2020,522:35-48.
[33] XIE S Y,HU H F,WU Y B.Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J].Pattern Recognition,2019,92:177-191.
[34] HASANI B,NEGI P S,MAHOOR M.BReG-NeXt:Facial affect computing using adaptive residual networks with bounded gradient[J].IEEE Transactions on Affective Computing,2020,99:1.
[35] SHAO J,QIAN Y S.Three convolutional neural network models for facial expression recognition in the wild[J].Neurocomputing,2019,355:82-92.
[36] XIE W C,SHEN L L,DUAN J M.Adaptive weighting of handcrafted feature losses for facial expression recognition[J].IEEE Transactions on Cybernetics,2021,51(5):2787-2800.

基于注意力机制与特征相关性的人脸表情识别

Facial expression recognition based on attention mechanism and feature correlation

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	包壮壮, 赵学军. 基于EfficientDet的无预训练SAR图像船舶检测器[J]. 北京航空航天大学学报, 2021, 47(8): 1664-1672.
[2]	陆婷婷, 张尧, 阎岩, 杨利民, 杨卫东. 一种基于自动特征学习的陨石坑区域检测算法[J]. 北京航空航天大学学报, 2021, 47(5): 939-952.
[3]	徐晓华, 钱平, 王一达, 周昕悦, 徐汉麟, 徐李冰. 面向电力系统的多粒度隐患检测方法[J]. 北京航空航天大学学报, 2021, 47(3): 520-530.
[4]	李昊, 管荑, 王杉, 石玮, 刘子鑫, 刘晓川. 电力系统厂站接线图拓扑关系检测技术[J]. 北京航空航天大学学报, 2021, 47(3): 531-538.
[5]	李昊, 王杉, 耿玉杰, 王黎, 孙文昌, 苗纯源. 基于深度学习和图匹配的接线图检测与校核[J]. 北京航空航天大学学报, 2021, 47(3): 539-548.
[6]	谢彭宇, 徐新. 基于多尺度联合学习的行人重识别[J]. 北京航空航天大学学报, 2021, 47(3): 613-622.
[7]	马腾宇, 李孜, 刘日升, 樊鑫, 罗钟铉. 基于无监督学习的多模态可变形配准[J]. 北京航空航天大学学报, 2021, 47(3): 658-664.
[8]	茅剑, 刘泰康, 刘培国. 一种基于深度学习的电磁信息泄漏检测方法[J]. 北京航空航天大学学报, 2021, 47(11): 2200-2207.
[9]	张敏, 李赵红, 刘金豆, 张珍珍. 基于多尺度残差卷积网络的HEVC视频隐写分析[J]. 北京航空航天大学学报, 2021, 47(11): 2226-2233.
[10]	崔家礼, 曹衡, 张亚明, 罗嗣梧, 李锦涛, 王华峰. 一种复杂场景下的人眼检测算法[J]. 北京航空航天大学学报, 2021, 47(1): 38-44.
[11]	陈猛夫. 基于迁移学习的暴恐图像自动识别[J]. 北京航空航天大学学报, 2020, 46(9): 1677-1681.
[12]	杜雨佳, 李海生, 姚春莲, 蔡强. 基于三元组网络的单图三维模型检索[J]. 北京航空航天大学学报, 2020, 46(9): 1691-1700.
[13]	陶一宁, 苏峰, 袁培江, 王田苗, 钟涛, 郝静如. 联合足迹识别与监控视频分析的智能刑侦系统[J]. 北京航空航天大学学报, 2020, 46(9): 1730-1738.
[14]	张洁茹, 苏峰, 袁培江, 王田苗, 陶一宁, 丁东. 双光谱智能体温检测与健康大数据管理系统[J]. 北京航空航天大学学报, 2020, 46(9): 1739-1746.
[15]	张子昊, 周千里, 王蓉. 基于空间注意力机制的行人再识别方法[J]. 北京航空航天大学学报, 2020, 46(9): 1747-1755.