基于级联森林和多模态融合的脑力疲劳识别算法

邓浩伟; 侯月皎; 张朝月; 徐慕华; 朱玲玲; 赵永岐

doi:10.13700/j.bh.1001-5965.2023.0030

基于级联森林和多模态融合的脑力疲劳识别算法

doi: 10.13700/j.bh.1001-5965.2023.0030

1.
中国人民解放军军事科学院军事医学研究院，北京 100850
2.
合肥工业大学仪器科学与光电工程学院，合肥 230009

基金项目: 基础科研项目(JKCY2019548B001)

详细信息

通讯作者:
E-mail：yqzhaoprc@sina.com

中图分类号: TP301.6
计量
- 文章访问数: 251
- HTML全文浏览量: 103
- PDF下载量: 6
- 被引次数: 0
出版历程
- 收稿日期: 2023-01-31
- 录用日期: 2023-05-23
- 网络出版日期: 2023-06-30
- 整期出版日期: 2025-02-28

Mental fatigue recognition algorithm based on cascade forest and multi-modal fusion

1.
Beijing Institute of Basic Medical Sciences，Beijing 100850，China
2.
School of Instrument Science and Opto-Electronics Engineering，Hefei University of Technology，Hefei 230009，China

Funds: Industrial Technology Development Program (JKCY2019548B001)

More Information

Corresponding author: E-mail：yqzhaoprc@sina.com

摘要

摘要:
脑力疲劳是影响人的认知功能和工作效率的重要因素，但目前没有公开的与脑力疲劳相关的多模态融合数据库，且常用于识别脑力疲劳的脑电信号在采集过程中易对人体造成负担和活动限制，因此，提出一种基于多模态生理信号的脑力疲劳识别算法。实验采用连续认知任务诱发受试者的脑力疲劳，同步采集脑电和心电2种生理信号。采用4导联(Fp1，F7，F8，Fp2)脑电信号和心电信号构建多模态融合特征，输入级联森林模型完成脑力疲劳识别任务。最终获得14份有效脑力疲劳多模态数据集，并实现了99.60%的平均识别率。通过引入级联森林和多模态融合技术，有效提高了脑力疲劳识别的准确性和鲁棒性，为脑力疲劳监测与干预提供了技术支持。
- 脑力疲劳 /
- 多模态融合 /
- 级联森林 /
- 脑电图 /
- 心电图
Abstract:
Mental fatigue is an important factor affecting human cognitive function and work efficiency, but there is no publicly available multi-modal fusion database related to mental fatigue, and the EEG signals commonly used to identify mental fatigue are prone to burden and activity limitation during the acquisition process, which led to the proposal of a mental fatigue identification algorithm based on multi-modal physiological signals. The experiment used a continuous cognitive task to induce mental fatigue in the subjects, and two physiological signals, EEG and ECG, were acquired simultaneously. The 4-lead (Fp1, F7, F8, Fp2) EEG and ECG signals were used to construct the multi-modal fusion features, and inputted into the cascade forest model to complete the mental fatigue recognition task. Finally, 14 valid mental fatigue multi-modal datasets were obtained and an average recognition rate of 99.60% was achieved. By introducing the cascade forest and multi-modal fusion technology, the accuracy and robustness of mental fatigue recognition are effectively improved, which provides technical support for mental fatigue monitoring and intervention.
- mental fatigue /
- multi-modal fusion /
- cascade forest /
- electroencephalogram /
- electrocardiogram

HTML全文

图 1 本文算法流程图

Figure 1. Proposed algorithm flowchart

下载: 全尺寸图片幻灯片

图 2 级联森林结构图

Figure 2. Structure diagram of cascade forest

下载: 全尺寸图片幻灯片

图 3 实验现场

Figure 3. Experimental site

下载: 全尺寸图片幻灯片

图 4 4个特定电极点在10-20系统电极放置法中的位置

Figure 4. Location of four specific electrode points in 10-20 international standard system

下载: 全尺寸图片幻灯片

图 5 单模态特征与多模态融合特征的分类结果对比

Figure 5. Comparison of classification results of single-modal features and multi-modal fusion features

下载: 全尺寸图片幻灯片

图 6 3种不同方法中14位被试各自的平均识别率

Figure 6. Average recognition rate of 14 subjects by three different methods

下载: 全尺寸图片幻灯片

表 1 实验环境

Table 1. Experimental environment

硬件/软件	版本/配置
OS	Ubuntu 16.04 LTS，64位
Python	3.7
处理器	IntelR Xeon(R) Gold 6248 CPU @ 2.50 GHz x 46
图形	llvmpipe (LLVM 6.0,256 bits)
内存	125.6 GiB

下载: 导出CSV

表 2 主观脑力疲劳自评量表

Table 2. Subjective mental fatigue self-rating scale

题项	说明及描述
困倦感	对自己的困倦程度进行评价，是否存在主观上想睡、渴望休息的感觉；症状表现为困倦、瞌睡打盹儿、想躺卧休息、提不起精神、不想干活做事
注意力状态	对自己的注意力集中程度进行评价，是否存在注意涣散，精神难以集中的情况
思维状态	对自己的思维状态进行评价，是否存在意识模糊不清，思维混乱的症状
情绪不安感	对自己的情绪状态进行评分，是否存在心烦意乱、焦躁不安、易怒的情况及其严重程度
视觉疲劳感	对自己的视觉疲劳感受进行评分，是否存在眼晴发涩、发干、眼晴疼痛、视物模糊等症状及其严重程度

下载: 导出CSV

表 3 主观脑力疲劳自评分及相对应等级

Table 3. Subjective mental fatigue self-rating score and corresponding level

自评分	脑力疲劳等级
≤2	不疲劳
2～4	轻度疲劳
4～6	中度疲劳
6～8	重度疲劳
≥8	极度疲劳

下载: 导出CSV

表 4 单模态特征与多模态融合特征的分类结果对比

Table 4. Comparison of classification results of single-modal features and multi-modal fusion features

特征类型	平均识别率/%	标准差/%
ECG特征	95.89	7.49
EEG特征	97.37	2.40
多模态融合特征（EEG+ECG）	99.60	0.90

下载: 导出CSV

表 5 2种方法与级联森林的分类结果对比

Table 5. Comparison of classification results of two methods and cascade forest

分类方法	平均识别率/%	标准差/%
SVM	79.91	14.25
CNN	87.50	10.51
级联森林	99.60	0.90

下载: 导出CSV

参考文献(33)

[1]	VAN CUTSEM J, MARCORA S, DE PAUW K, et al. The effects of mental fatigue on physical performance: A systematic review[J]. Sports Medicine, 2017, 47(8): 1569-1588. doi: 10.1007/s40279-016-0672-0
[2]	HOPSTAKEN J F, VAN DER LINDEN D, BAKKER A B, et al. A multifaceted investigation of the link between mental fatigue and task disengagement[J]. Psychophysiology, 2015, 52(3): 305-315. doi: 10.1111/psyp.12339
[3]	VAN DEN BERG M J, WU L J, GANDER P H. Subjective measurements of in-flight sleep, circadian variation, and their relationship with fatigue[J]. Aerospace Medicine and Human Performance, 2016, 87(10): 869-875. doi: 10.3357/AMHP.4587.2016
[4]	FULCHER B D, PHILLIPS A J K, ROBINSON P A. Quantitative physiologically based modeling of subjective fatigue during sleep deprivation[J]. Journal of Theoretical Biology, 2010, 264(2): 407-419. doi: 10.1016/j.jtbi.2010.02.028
[5]	HEAD J, TENAN M S, TWEEDELL A J, et al. Prior mental fatigue impairs marksmanship decision performance[J]. Frontiers in Physiology, 2017, 8: 680. doi: 10.3389/fphys.2017.00680
[6]	SAHAYADHAS A, SUNDARAJ K, MURUGAPPAN M. Detecting driver drowsiness based on sensors: A review[J]. Sensors, 2012, 12(12): 16937-16953. doi: 10.3390/s121216937
[7]	LAL S K L, CRAIG A. Driver fatigue: Electroencephalography and psychological assessment[J]. Psychophysiology, 2002, 39(3): 313-321. doi: 10.1017/S0048577201393095
[8]	TREJO L J, KUBITZ K, ROSIPAL R, et al. EEG-based estimation and classification of mental fatigue[J]. Psychology, 2015, 6(5): 572-589. doi: 10.4236/psych.2015.65055
[9]	TRAN Y, WIJESURIYA N, TARVAINEN M, et al. The relationship between spectral changes in heart rate variability and fatigue[J]. Journal of Psychophysiology, 2009, 23(3): 143-151. doi: 10.1027/0269-8803.23.3.143
[10]	HU J F. Comparison of different features and classifiers for driver fatigue detection based on a single EEG channel[J]. Computational and Mathematical Methods in Medicine, 2017, 2017: 5109530.
[11]	BUTKEVIČIŪTĖ E, MICHALKOVIČ A, BIKULČIENĖ L. ECG signal features classification for the mental fatigue recognition[J]. Mathematics, 2022, 10(18): 3395. doi: 10.3390/math10183395
[12]	ZHAO C L, ZHAO M, LIU J P, et al. Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator[J]. Accident Analysis & Prevention, 2012, 45: 83-90.
[13]	MONTEIRO T G, SKOURUP C, ZHANG H X. Optimizing CNN hyperparameters for mental fatigue assessment in demanding maritime operations[J]. IEEE Access, 2020, 8: 40402-40412. doi: 10.1109/ACCESS.2020.2976601
[14]	DEHAIS F, DUPRÈS A, BLUM S, et al. Monitoring pilot’s mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions[J]. Sensors, 2019, 19(6): 1324. doi: 10.3390/s19061324
[15]	胡淑燕, 郑钢铁. 基于EEG频谱特征的驾驶员疲劳监测研究[J]. 中国安全生产科学技术, 2010, 6(3): 90-94. doi: 10.3969/j.issn.1673-193X.2010.03.017 HU S Y, ZHENG G T. Study on driver fatigue detection based on EEG spectrum-related features[J]. Journal of Safety Science and Technology, 2010, 6(3): 90-94(in Chinese). doi: 10.3969/j.issn.1673-193X.2010.03.017
[16]	SHEN K Q, LI X P, ONG C J, et al. EEG-based mental fatigue measurement using multi-class support vector machines with confidence estimate[J]. Clinical Neurophysiology, 2008, 119(7): 1524-1533. doi: 10.1016/j.clinph.2008.03.012
[17]	HAN S, WEI C F, WU Q. Recognition of fatigue status of pilots based on deep sparse auto-encoding network[C]//Proceedings of the 36th Chinese Control Conference. Piscataway: IEEE Press, 2017: 10945-10950.
[18]	JEONG J H, YU B W, LEE D H, et al. Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional LSTM network using electroencephalography signals[J]. Brain Sciences, 2019, 9(12): 348. doi: 10.3390/brainsci9120348
[19]	HAN S Y, KIM J W, LEE S W. Recognition of pilot’s cognitive states based on combination of physiological signals[C]//Proceedings of the 7th International Winter Conference on Brain-Computer Interface. Piscataway: IEEE Press, 2019: 1-5.
[20]	ZHOU Z H, FENG J. Deep forest[J]. National Science Review, 2019, 6(1): 74-86. doi: 10.1093/nsr/nwy108
[21]	CHENG J, CHEN M Y, LI C, et al. Emotion recognition from multi-channel EEG via deep forest[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(2): 453-464.
[22]	FANG Y F, YANG H Y, ZHANG X G, et al. Multi-feature input deep forest for EEG-based emotion recognition[J]. Frontiers in Neurorobotics, 2021, 14: 617531. doi: 10.3389/fnbot.2020.617531
[23]	姚娟娟, 路堃, 许金秀. 基于便携式脑电设备的快乐和悲伤情绪分类[J]. 计算机系统应用, 2020, 29(5): 233-238. doi: 10.3969/j.issn.1003-3254.2020.05.034 YAO J J, LU K, XU J X. Classification of happiness and sadness based on portable EEG devices[J]. Computer Systems & Applications, 2020, 29(5): 233-238(in Chinese). doi: 10.3969/j.issn.1003-3254.2020.05.034
[24]	仲伟峰, 李志, 刘燕, 等. 基于少导联脑电和时频深度网络的帕金森病伴快速眼动睡眠行为障碍智能辅助诊断方法[J]. 生物医学工程学杂志, 2021, 38(6): 1043-1053. doi: 10.7507/1001-5515.202009067 ZHONG W F, LI Z, LIU Y, et al. Intelligence-aided diagnosis of Parkinson’s disease with rapid eye movement sleep behavior disorder based on few-channel electroencephalogram and time-frequency deep network[J]. Journal of Biomedical Engineering, 2021, 38(6): 1043-1053(in Chinese). doi: 10.7507/1001-5515.202009067
[25]	WASCHER E, RASCH B, SÄNGER J, et al. Frontal theta activity reflects distinct aspects of mental fatigue[J]. Biological Psychology, 2014, 96: 57-65. doi: 10.1016/j.biopsycho.2013.11.010
[26]	CHEN C X, LI K, WU Q Y, et al. EEG-based detection and evaluation of fatigue caused by watching 3DTV[J]. Displays, 2013, 34(2): 81-88. doi: 10.1016/j.displa.2013.01.002
[27]	WIEHLER A, BRANZOLI F, ADANYEGUH I, et al. A neuro-metabolic account of why daylong cognitive work alters the control of economic decisions[J]. Current Biology, 2022, 32(16): 3564-3575. doi: 10.1016/j.cub.2022.07.010
[28]	伍金莉. 心率变异性分析方法的研究[D]. 北京: 北京交通大学, 2009. WU J L. Study on analysis method of heart rate variability[D]. Beijing: Beijing Jiaotong University, 2009 (in Chinese).
[29]	SZYPULSKA M, PIOTROWSKI Z. Prediction of fatigue and sleep onset using HRV analysis[C]// Proceedings of the 19th International Conference Mixed Design of Integrated Circuits and Systems- MIXDES 2012. Piscataway: IEEE Press, 2012: 543-546.
[30]	郭玮, 任杰. 脑力疲劳的实验室诱发模型和评价手段研究进展[J]. 中国运动医学杂志, 2013, 32(12): 1121-1128. GUO W, REN J. Research progress on laboratory-induced models and evaluation methods of mental fatigue[J]. Chinese Journal of Sports Medicine, 2013, 32(12): 1121-1128(in Chinese).
[31]	AHN S, NGUYEN T, JANG H, et al. Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data[J]. Frontiers in Human Neuroscience, 2016, 10: 219.
[32]	LIU Y S, LAN Z R, KHOO H H G, et al. EEG-based evaluation of mental fatigue using machine learning algorithms[C]//Proceedings of the International Conference on Cyberworlds. Piscataway: IEEE Press, 2018: 276-279.
[33]	张振祥, 张璟. 关于日本《疲劳症状自评量表》(2002)[J]. 人类工效学, 2003, 9(3): 60-62. doi: 10.3969/j.issn.1006-8309.2003.03.018 ZHANG Z X, ZHANG J. About Japan’s fatigue symptom checklist (2002)[J]. Chinese Ergonomics, 2003, 9(3): 60-62(in Chinese). doi: 10.3969/j.issn.1006-8309.2003.03.018