面部三维生理特征点在睡眠剥夺中的疲劳检测

李敬强; 房秋; 樊天辰; 马龙

doi:10.13700/j.bh.1001-5965.2022.0733

面部三维生理特征点在睡眠剥夺中的疲劳检测

doi: 10.13700/j.bh.1001-5965.2022.0733

李敬强^{1, 2, ,},
房秋^{1, 2},
樊天辰^{1, 2},
马龙³

1.
中国民航大学安全科学与工程学院，天津 300300
2.
中国民航大学民航安全科学研究所，天津 300300
3.
中国民航大学中欧航空工程师学院，天津 300300

基金项目: 国家自然科学基金(U1933122)；中央高校基本科研业务费专项资金(3122019186)

详细信息

通讯作者:
E-mail：jqli@cauc.edu.cn

中图分类号: X951
计量
- 文章访问数: 230
- HTML全文浏览量: 115
- PDF下载量: 19
- 被引次数: 0
出版历程
- 收稿日期: 2022-08-20
- 录用日期: 2023-02-03
- 网络出版日期: 2023-03-09
- 整期出版日期: 2024-09-27

Fatigue detection of facial 3D physiological feature points in sleep deprivation

LI Jingqiang^{1, 2
, ,},
FANG Qiu^{1, 2},
FAN Tianchen^{1, 2},
MA Long³

1.
College of Safety Science and Engineering，Civil Aviation University of China，Tianjin 300300，China
2.
Research Institute of Civil Aviation Safety，Civil Aviation University of China，Tianjin 300300，China
3.
Sino-European Institute of Aviation Engineering，Civil Aviation University of China，Tianjin 300300，China

Funds: National Natural Science Foundation of China (U1933122); The Fundamental Research Funds for the Central Universities (3122019186)

More Information

Corresponding author: E-mail：jqli@cauc.edu.cn

摘要

摘要:
睡眠不足、生理节律紊乱是导致交通运输岗位人员疲劳的重要因素，随着值勤“刷脸”技术的兴起，及时识别岗位人员疲劳状态是安全风险防范关口前移的必要举措。为深入探索面部疲劳的三维变化，通过30 h睡眠剥夺实验，基于主成分分析 (PCA) 法融合面部双侧曲率特征，提出皱纹严重程度指数(WSI)，结合主观疲劳评估、精神运动警觉度测试和客观心率(HR)监测方法，综合判断受试者的面部疲劳程度。结果表明：在睡眠剥夺条件下，WSI变化和疲劳的波动特征明显，整体呈逐渐升高趋势；30 h实验数据显示，WSI变化和主观疲劳指数、反应时的波动趋势有较强相关性(P<0.01)和HR的波动趋势有一定的关联性(P<0.05)。基于WSI指数将疲劳划分为4个等级，从三维生理特征的角度验证面部疲劳静态检测的可行性，研究结果为疲劳快速检测技术提供了新思路。
- 面部特征 /
- 疲劳检测 /
- 睡眠剥夺 /
- 三维建模 /
- 昼夜节律
Abstract:
Sleep deprivation and circadian rhythm disorder are important factors affecting the fatigue of transportation staff. In order to advance the safety risk prevention threshold, it is now vital to identify staff fatigue concurrently with the advent of "face brushing" technology while on duty. This study uses the 30 h sleep deprivation experiment to further explore the three-dimensional changes of facial fatigue. Based on the fusion of facial bilateral curvature characteristics and principal component analysis (PCA), it proposes the wrinkles severity index (WSI), which when paired with the objective heart rate (HR) monitoring method, psychomotor vigilance test, and subjective fatigue assessment, integrated subjects' facial fatigue judgment. The results showed that under the condition of sleep deprivation, the WSI changes and fatigue fluctuated significantly, and the overall trend was gradually increasing. The results of the 30 h trial revealed a high correlation between the change in WSI and the trends in subjective tiredness index, response time (P<0.01), and HR (P<0.05). Based on the WSI index, the fatigue was divided into four grades, and the feasibility of static detection of facial fatigue was verified from the perspective of three-dimensional physiological characteristics. The results provided a new idea for rapid fatigue detection technology.
- facial features /
- fatigue detection /
- sleep deprivation /
- three-dimensional modeling /
- circadian rhythm

HTML全文

图 1 面部轮廓识别

Figure 1. Facial contour recognition

下载: 全尺寸图片幻灯片

图 2 面部三维建模

Figure 2. 3D face modeling

下载: 全尺寸图片幻灯片

图 3 n=1时睡眠剥夺前的面部建模

Figure 3. Facial modeling before sleep deprivation with n=1

下载: 全尺寸图片幻灯片

图 4 n=30时睡眠剥夺后的面部建模

Figure 4. Facial modeling after sleep deprivation with n=30

下载: 全尺寸图片幻灯片

图 5 面部特征点定位

Figure 5. Facial feature point localization

下载: 全尺寸图片幻灯片

图 6 双侧面部曲率统计

Figure 6. Curvature in bilateral facial statistics

下载: 全尺寸图片幻灯片

图 7 4种类型疲劳指标随睡眠剥夺变化趋势

Figure 7. Variation trend of four types of fatigue indexes with sleep deprivation

下载: 全尺寸图片幻灯片

图 8 WSI的高斯多峰拟合

Figure 8. Gaussian multi-peak fitting of the WSI

下载: 全尺寸图片幻灯片

图 9 4阶段各疲劳指标的变化

Figure 9. Changes of fatigue indexes in four stages

下载: 全尺寸图片幻灯片

表 1 面部特征点坐标

Table 1. Facial feature point coordinates mm

序号	x	y	z	序号	x	y	z
1	−19.97	28.43	−338.2	6	−22.77	25.61	−338
2	−20.53	27.87	−338.2	7	−23.33	25.04	−338
3	−21.09	27.3	−338.2	8	−23.9	24.49	−338.1
4	−21.65	26.73	−338.1	9	−24.46	23.93	−338.1
5	−22.21	26.17	−338	10	−25.03	23.37	−338.2

下载: 导出CSV

表 2 PCA的总方差解释

Table 2. Total variance interpretation of PCA

新特征	成分	总计		方差百分比/%
新特征	成分	初始特征值	提取载荷平方和	初始特征值	提取载荷平方和
0.979	1	3.739	3.739	93.485	93.485
0.965	2	0.118		2.95
0.956	3	0.092		2.297
0.967	4	0.051		1.269

下载: 导出CSV

表 3 疲劳指标的统计分析

Table 3. Statistical analysis of fatigue index

指标	β	相关性
指标	β	WSI	KSS	RT	HR
WSI	0.971	1	0.904**	0.961**	0.407*
KSS	0.887	0.904**	1	0.875**	0.527**
RT	0.982	0.961**	0.875**	1	0.404*
HR	0.489	0.407*	0.527**	0.404*	1
注：*表示P<0.01，表示P<0.05。

下载: 导出CSV

表 4 基于WSI的疲劳等级划分

Table 4. Fatigue grade classification based on WSI

疲劳等级	WSI	等级描述	具体表现
1	0～0.15	精力充沛	注意程度高，精神饱满，面部皱纹肉眼不可见
2	0.15～0.25	轻度疲劳	注意程度一般，稍觉疲倦，面部皱纹较浅
3	0.25～0.28	中度疲劳	注意程度较低，非常疲倦，面部皱纹加深
4	0.28以上	重度疲劳	注意程度极低，非常疲倦想睡觉，面部皱纹很深

下载: 导出CSV

参考文献(37)

[1]	SKALA F, ZEMKOVÁ E. Effects of acute fatigue on cognitive performance in team sport players: Does it change the way they perform? A scoping review[J]. Applied Sciences, 2022, 12(3): 1736. doi: 10.3390/app12031736
[2]	VAN DEN BERG M J, WU L J, GANDER P H. Subjective measurements of In-flight sleep, circadian variation, and their relationship with fatigue[J]. Aerospace Medicine and Human Performance, 2016, 87(10): 869-875. doi: 10.3357/AMHP.4587.2016
[3]	SUNDELIN T, LEKANDER M, KECKLUND G, et al. Cues of fatigue: Effects of sleep deprivation on facial appearance[J]. Sleep, 2013, 36(9): 1355-1360. doi: 10.5665/sleep.2964
[4]	HOLDING B C, SUNDELIN T, CAIRNS P, et al. The effect of sleep deprivation on objective and subjective measures of facial appearance[J]. Journal of Sleep Research, 2019, 28(6): e12860. doi: 10.1111/jsr.12860
[5]	靳慧斌, 于桂花, 刘海波. 瞳孔直径检测管制疲劳的有效性分析[J]. 北京航空航天大学学报, 2018, 44(7): 1402-1407. JIN H B, YU G H, LIU H B. Effectiveness analysis of pupil diameter detection for air traffic controller’s fatigue[J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44(7): 1402-1407 (in Chinese).
[6]	LIM T J, LEONG H Y, JIA Y P, et al. Eye fatigue algorithm for driver drowsiness detection system[M]. Berlin: Springer, 2021: 638-652.
[7]	MUHIMMAH I, ABRIYANI F, KURNIAWARDHANI A. Automatic wrinkles detection on face image[J]. IOP Conference Series: Materials Science and Engineering, 2019, 482: 012026. doi: 10.1088/1757-899X/482/1/012026
[8]	ELBASHIR R M, HOON YAP M. Evaluation of automatic facial wrinkle detection algorithms[J]. Journal of Imaging, 2020, 6(4): 17. doi: 10.3390/jimaging6040017
[9]	SABINA U, WHANGBO T K. Edge-based effective active appearance model for real-time wrinkle detection[J]. Skin Research and Technology: Official Journal of International Society for Bioengineering and the Skin and International Society for Digital Imaging of Skin and International Society for Skin Imaging, 2021, 27(3): 444-452.
[10]	LIEW S, DE SILVA D. Tear trough and lid–cheek junction anatomy: Current understanding and nonsurgical treatment[J]. Facial Plastic Surgery, 2022, 38(2): 143-151. doi: 10.1055/s-0041-1741498
[11]	MAGNUSSON M R. Commentary on: Three-dimensional description of the angular artery in the nasolabial fold[J]. Aesthetic Surgery Journal, 2021, 41(6): 705-706. doi: 10.1093/asj/sjaa200
[12]	SEE M S, FOXTON M R, MIEDZIANOWSKI-SINCLAIR N A, et al. Stereophotogrammetric measurement of the nasolabial fold in repose: A study of age and posture-related changes[J]. European Journal of Plastic Surgery, 2007, 29(8): 387-393. doi: 10.1007/s00238-007-0123-0
[13]	SADICK N S, BOSNIAK S L, CANTISANO-ZILKHA M, et al. Definition of the tear trough and the tear trough rating scale[J]. Journal of Cosmetic Dermatology, 2007, 6(4): 218-222.
[14]	ARTHUR C A, HASSAN A S. Tear trough depth and quality across age groups: A cross-sectional study[J]. European Journal of Plastic Surgery, 2022, 45(1): 65-72.
[15]	李敬强, 王庆福, 张璐, 等. 不同睡眠类型选择性注意在睡眠剥夺中内稳态与昼夜节律研究[J]. 生物医学工程学杂志, 2022, 39(2): 248-256. doi: 10.7507/1001-5515.202110083 LI J Q, WANG Q F, ZHANG L, et al. Study on homeostasis and circadian rhythm of attention performance of different chronotypes in sleep deprivation[J]. Journal of Biomedical Engineering, 2022, 39(2): 248-256 (in Chinese). doi: 10.7507/1001-5515.202110083
[16]	SHAHID A, WILKINSON K, MARCU S, et al. Stanford sleepiness scale (SSS)[M]. New York: Springer New York, 2011: 369-370.
[17]	DINGES D F, POWELL J W. Microcomputer analyses of performance on a portable, simple visual RT task during sustained operations[J]. Behavior Research Methods, Instruments, & Computers, 1985, 17(6): 652-655.
[18]	RODHEIM K, JUNG C, WRIGHT K. 016 associations between circadian melatonin and temperature amplitudes during constant routine[J]. Sleep, 2021, 44(S2): A8.
[19]	汪锡, 王远, 许杨, 等. 非肥胖健康成人静息代谢率预测公式准确性研究[J]. 北京生物医学工程, 2017, 36(6): 607-613. doi: 10.3969/j.issn.1002-3208.2017.06.010 WANG X, WANG Y, XU Y, et al. Accuracy of predictive equations for resting metabolic rate in non-obese healthy Chinese adults[J]. Beijing Biomedical Engineering, 2017, 36(6): 607-613 (in Chinese). doi: 10.3969/j.issn.1002-3208.2017.06.010
[20]	PUTZ P, KOGLER B, BERSENKOWITSCH I. Reliability and validity of assessing energy and nutrient intake with the Vienna food record: A cross-over randomised study[J]. Nutrition Journal, 2019, 18(1): 1-10. doi: 10.1186/s12937-018-0428-9
[21]	马龙, 徐泓悦, 胡艳敏, 等. 基于投影条纹级次识别的发动机叶片三维测量方法[J]. 激光与光电子学进展, 2018, 55(4): 041201. MA L, XU H Y, HU Y M, et al. Three-dimensional measurement method of aero-engine blades based on projected fringe order identification[J]. Laser & Optoelectronics Progress, 2018, 55(4): 041201 (in Chinese).
[22]	CHARYTANOWICZ K, CIEŚLAK W, MOZGAWA W. A generalization of the total mean curvature[J]. Rendiconti Del Seminario Matematico Della Università Di Padova, 2021, 146: 223-231.
[23]	FAIRAG F, CHEN K, BRITO-LOEZA C, et al. A two-level method for image denoising and image deblurring models using mean curvature regularization[J]. International Journal of Computer Mathematics, 2022, 99(4): 693-713. doi: 10.1080/00207160.2021.1929939
[24]	ZHENG J W, RAKOVSKI C. On the application of principal component analysis to classification problems[J]. Data Science Journal, 2021, 20(1): 26.
[25]	ANTLER C A, YAMAZAKI E M, CASALE C E, et al. The 3-minute psychomotor vigilance test demonstrates inadequate convergent validity relative to the 10-minute psychomotor vigilance test across sleep loss and recovery[J]. Frontiers in Neuroscience, 2022, 16(2): 815697.
[26]	王海玮, 温惠英, 刘敏. 夜间环境驾驶员精神负荷的生理特性评估与实验[J]. 吉林大学学报(工学版), 2017, 47(2): 420-428. WANG H W, WEN H Y, LIU M. Experimental evaluation of nighttime driver’s physiological characteristics in driving simulator[J]. Journal of Jilin University (Engineering and Technology Edition), 2017, 47(2): 420-428 (in Chinese).
[27]	PAN L Y, MAI Z F, WANG J H, et al. Altered vigilant maintenance and reorganization of rich-clubs in functional brain networks after total sleep deprivation[J]. Cerebral Cortex, 2023, 33(4): 1140-1154. doi: 10.1093/cercor/bhac126
[28]	THOMAS S J. Sleep deprivation: Time to intervene[J]. Sleep Medicine, 2021, 86(10): 116-117.
[29]	ROHRICH R J, AVASHIA Y J, SAVETSKY I L. Prediction of facial aging using the facial fat compartments[J]. Plastic and Reconstructive Surgery, 2021, 147(1S-2): 38-42. doi: 10.1097/PRS.0000000000007624
[30]	WONG C H, HSIEH M K H, MENDELSON B. The tear trough ligament: Anatomical basis for the tear trough deformity[J]. Plastic and Reconstructive Surgery, 2012, 129(6): 1392-1402. doi: 10.1097/PRS.0b013e31824ecd77
[31]	HUR M S, O J, YANG H M, et al. Heights and spatial relationships of the facial muscles acting on the nasolabial fold by dissection and three-dimensional microcomputed tomography[J]. PLoS One, 2020, 15(8): e0237043. doi: 10.1371/journal.pone.0237043
[32]	BANI ISSA W, ABDUL RAHMAN H, ALBLUWI N, et al. Morning and evening salivary melatonin, sleepiness and chronotype: A comparative study of nurses on fixed day and rotating night shifts[J]. Journal of Advanced Nursing, 2020, 76(12): 3372-3384. doi: 10.1111/jan.14530
[33]	KUSZTOR A, RAUD L, JUEL B E, et al. Sleep deprivation differentially affects subcomponents of cognitive control[J]. Sleep, 2019, 42(4): zsz016.
[34]	VASEY C, MCBRIDE J, PENTA K. Circadian rhythm dysregulation and restoration: The role of melatonin[J]. Nutrients, 2021, 13(10): 3480. doi: 10.3390/nu13103480
[35]	MAIRE M, REICHERT C F, GABEL V, et al. Human brain patterns underlying vigilant attention: Impact of sleep debt, circadian phase and attentional engagement[J]. Scientific Reports, 2018, 8: 970. doi: 10.1038/s41598-017-17022-9
[36]	CRAIG A, TRAN Y, WIJESURIYA N, et al. Regional brain wave activity changes associated with fatigue[J]. Psychophysiology, 2012, 49(4): 574-582. doi: 10.1111/j.1469-8986.2011.01329.x
[37]	AHMED T, WILSON D. Exploiting circadian memory to hasten recovery from circadian misalignment[J]. Chaos, 2021, 31(7): 073130. doi: 10.1063/5.0053441