基于多任务学习的电力文本信息抽取

纪鑫; 武同心; 余婷; 董林啸; 陈屹婷; 米娜; 赵加奎

doi:10.13700/j.bh.1001-5965.2022.0683

基于多任务学习的电力文本信息抽取

doi: 10.13700/j.bh.1001-5965.2022.0683

纪鑫^{1, 2},
武同心^1, ,,
余婷¹,
董林啸¹,
陈屹婷¹,
米娜¹,
赵加奎¹

1.
国家电网有限公司大数据中心，北京 100031
2.
北京航空航天大学计算机学院，北京 100191

基金项目: 基于图神经网络和图深度学习的电力知识抽取技术研究(52999021N005)

详细信息

通讯作者:
E-mail：tongxin-wu@sgcc.com.cn

中图分类号: TP391.1
计量
- 文章访问数: 144
- HTML全文浏览量: 109
- PDF下载量: 1
- 被引次数: 0
出版历程
- 收稿日期: 2022-08-03
- 录用日期: 2022-09-23
- 网络出版日期: 2022-10-12
- 整期出版日期: 2024-08-28

Power text information extraction based on multi-task learning

JI Xin^{1, 2},
WU Tongxin^{1
, ,},
YU Ting¹,
DONG Linxiao¹,
CHEN Yiting¹,
MI Na¹,
ZHAO Jiakui¹

1.
Big Data Center of State Grid Corporation of China，Beijing 100031，China
2.
School of Computer Science and Engineering，Beihang University，Beijing 100191，China

Funds: Research on Power Knowledge Extraction Technology Based on Graph Neural Network and Graph Deep Learning (52999021N005)

More Information

Corresponding author: E-mail：tongxin-wu@sgcc.com.cn

摘要

摘要:
为提升电力系统故障文本在实际业务场景中的分析处理速度，提出基于预训练与多任务学习的电力故障文本信息自动抽取模型。利用预训练模型学习电力文本词语的上下文信息，挖掘词语的一阶和二阶融合特征，增强特征的表示能力，利用多任务学习框架结合命名实体识别和关系抽取2个任务的学习，实现实体识别和关系抽取的互相补充和互相促进，进而提高电力故障文本信息抽取的性能。通过对某电力网数据中心的日常业务数据进行模型验证，与其他模型相比，所提模型提高了电力故障文本实体识别和关系抽取的准确率和召回率。
- 电力故障 /
- 预训练 /
- 多任务学习 /
- 实体识别 /
- 关系抽取
Abstract:
In order to improve the analysis and processing speed of power system fault text in actual business scenarios, a power fault text information extraction model based on pre-training and multi-task learning was proposed. The pre-training model was used to learn the context information of power text words. The first-order and second-order fusion features of words were mined, which enhanced the representation ability of features. The multi-task learning framework was used to combine named entity recognition and relation extraction, which realized the mutual supplement and mutual promotion of entity recognition and relationship extraction, so as to improve the performance of power fault text information extraction. The model was verified by the daily business data of a power data center. Compared with other models, the proposed model’s accuracy and recall of power fault text entity recognition and relation extraction were improved.
- power fault /
- pre-training /
- multi-task learning /
- entity recognition /
- relation extraction

HTML全文

图 1 MTEIE模型框架

Figure 1. MTEIE model framework

下载: 全尺寸图片幻灯片

图 2 多头关系选择矩阵

Figure 2. Multi-head relation selection matrix

下载: 全尺寸图片幻灯片

图 3 实体关系分类过程

Figure 3. Entity relation classification procedure

下载: 全尺寸图片幻灯片

表 1 实体标签统计信息

Table 1. Entity statistics

实体类别	实体数量/个	实体占比/%
发电设备	2204	13.68
电动设备	2992	18.57
环境信息	3321	20.61
电力电缆	1046	6.49
显示设备	1543	9.575
控制器件	865	5.37
其他实体	4134	25.65

下载: 导出CSV

表 2 关系标签统计信息

Table 2. Relation statistics

关系类别	关系数量/个	关系占比/%
同类	2 023	21.95
原因	1757	19.06
故障	1245	13.50
从属	970	10.52
其他关系	3223	34.96

下载: 导出CSV

表 3 命名实体识别结果

Table 3. Results of named entity recognition %

模型	P	R	F₁
CRF^[27]	87.31	86.50	86.90
Bi-LSTM+CRF^[14]	89.03	88.12	88.57
LSTM-CNNS+CRF^[28]	89.34	88.56	88.95
MTEIE	91.16	90.23	90.69

下载: 导出CSV

表 4 多命名实体识别结果

Table 4. Results of multiple named entity recognition %

模型	P	R	F₁
CRF^[27]	85.21	84.51	84.86
Bi-LSTM+CRF^[14]	87.34	86.38	86.86
LSTM-CNNS+CRF^[28]	88.16	87.53	87.84
MTEIE	89.22	88.96	89.09

下载: 导出CSV

表 5 关系抽取结果

Table 5. Results of multiple relation extraction %

模型	P	R	F₁
基于CNN^[16]	81.31	80.12	80.71
基于Multi-head^[11]	82.67	81.23	81.94
基于BERT^[18]	83.94	82.77	83.35
MTEIE	85.19	84.67	84.93

下载: 导出CSV

表 6 多关系抽取结果

Table 6. Results of multiple relation extraction %

模型	P	R	F₁
基于CNN^[16]	78.61	77.33	77.96
基于Multi-head^[11]	80.15	79.45	79.80
基于BERT^[18]	81.76	81.23	81.49
MTEIE	83.01	82.71	82.86

下载: 导出CSV

表 7 消融实验结果

Table 7. Results of ablation tests %

模型	P		R
模型	实体	关系	实体	关系
MTEIE-BERT	89.56	83.35	88.35	82.12
MTEIE-Entity		84.34		83.54
MTEIE-Relation	90.05		89.78
MTEIE	91.16	85.19	90.23	84.67

下载: 导出CSV

表 8 F₁值混淆矩阵分析

Table 8. Confusion matrix analysis of F₁ %

方法	实体识别	关系抽取
仅关系抽取		83.94
仅命名实体识别	89.91
关系抽取+命名实体识别	90.69	84.93

下载: 导出CSV

表 9 案例分析

Table 9. Case analysis

模型	案例1	案例2
原文	1号主变汇控箱内液晶显示温度与变压器本体温度差值过大，怀疑显示不准确	2号主变有载分接开关机械限位松动，导致开关出现多次滑档
Bi-LSTM+CRF+BERT	{主变，汇控箱，从属关系}； {主变，温度差值，原因关系}	{主变，机械限位，从属关系}； {机械限位，开关，故障关系}
MTEIE	{主变，汇控箱，从属关系}； {汇控箱，温度差值，故障关系}； {温度差值，显示，原因关系}	{主变，机械限位，从属关系}； {机械限位，开关，原因关系}

下载: 导出CSV

参考文献(28)

[1]	周念成, 廖建权, 王强钢, 等. 深度学习在智能电网中的应用现状分析与展望[J]. 电力系统自动化, 2019, 43(4): 180-191. doi: 10.7500/AEPS20180323002 ZHOU N C, LIAO J Q, WANG Q G, et al. Analysis and prospect of deep learning application in smart grid[J]. Automation of Electric Power Systems, 2019, 43(4): 180-191 (in Chinese). doi: 10.7500/AEPS20180323002
[2]	李刚, 张博, 赵文清, 等. 电力设备状态评估中的数据科学问题: 挑战与展望[J]. 电力系统自动化, 2018, 42(21): 10-20. doi: 10.7500/AEPS20180331006 LI G, ZHANG B, ZHAO W Q, et al. Data science issues in state evaluation of power equipment: Challenges and prospects[J]. Automation of Electric Power Systems, 2018, 42(21): 10-20 (in Chinese). doi: 10.7500/AEPS20180331006
[3]	薛禹胜, 赖业宁. 大能源思维与大数据思维的融合(一)大数据与电力大数据[J]. 电力系统自动化, 2016, 40(1): 1-8. doi: 10.7500/AEPS20151208005 XUE Y S, LAI Y N. Integration of macro energy thinking and big data thinking part one big data and power big data[J]. Automation of Electric Power Systems, 2016, 40(1): 1-8 (in Chinese). doi: 10.7500/AEPS20151208005
[4]	邱剑, 王慧芳, 应高亮, 等. 文本信息挖掘技术及其在断路器全寿命状态评价中的应用[J]. 电力系统自动化, 2016, 40(6): 107-112. doi: 10.7500/AEPS20150812003 QIU J, WANG H F, YING G L, et al. Text mining technique and application of lifecycle condition assessment for circuit breaker[J]. Automation of Electric Power Systems, 2016, 40(6): 107-112(in Chinese). doi: 10.7500/AEPS20150812003
[5]	邵冠宇, 王慧芳, 吴向宏, 等. 基于依存句法分析的电力设备缺陷文本信息精确辨识方法[J]. 电力系统自动化, 2020, 44(12): 178-185. doi: 10.7500/AEPS20190401001 SHAO G Y, WANG H F, WU X H, et al. Precise information identification method of power equipment defect text based on dependency parsing[J]. Automation of Electric Power Systems, 2020, 44(12): 178-185 (in Chinese). doi: 10.7500/AEPS20190401001
[6]	郑伟彦, 杨勇, 卢家驹, 等. 面向配电网知识图谱的配电调度文本实体链接方法[J]. 电力系统保护与控制, 2021, 49(4): 111-117. ZHENG W Y, YANG Y, LU J J, et al. Entity linking method of distribution dispatching texts for a distribution network knowledge graph[J]. Power System Protection and Control, 2021, 49(4): 111-117 (in Chinese).
[7]	余建明, 王小海, 张越, 等. 面向智能调控领域的知识图谱构建与应用[J]. 电力系统保护与控制, 2020, 48(3): 29-35. YU J M, WANG X H, ZHANG Y, et al. Construction and application of knowledge graph for intelligent dispatching and control[J]. Power System Protection and Control, 2020, 48(3): 29-35 (in Chinese).
[8]	何炎祥, 罗楚威, 胡彬尧. 基于CRF和规则相结合的地理命名实体识别方法[J]. 计算机应用与软件, 2015, 32(1): 179-185. doi: 10.3969/j.issn.1000-386x.2015.01.046 HE Y X, LUO C W, HU B Y. Geographic entity recognition method based on CRF model and rules combination[J]. Computer Applications and Software, 2015, 32(1): 179-185 (in Chinese). doi: 10.3969/j.issn.1000-386x.2015.01.046
[9]	JACOBS P S, RAU L F. SCISOR: Extracting information from on-line news[J]. Communications of the ACM, 1990, 33(11): 88-97. doi: 10.1145/92755.92769
[10]	WILLIAMS R J, ZIPSER D. A learning algorithm for continually running fully recurrent neural networks[J]. Neural Computation, 1989, 1(2): 270-280. doi: 10.1162/neco.1989.1.2.270
[11]	BEKOULIS G, DELEU J, DEMEESTER T, et al. Joint entity recognition and relation extraction as a multi-head selection problem[J]. Expert Systems with Applications, 2018, 114: 34-45. doi: 10.1016/j.eswa.2018.07.032
[12]	BEKOULIS G, DELEU J, DEMEESTER T, et al. Adversarial training for multi-context joint entity and relation extraction [EB/OL]. (2018-08-21) [2019-01-14].https://arxiv.org/abs/ 1808.06876.
[13]	HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735
[14]	HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[EB/OL]. (2015-08-09) [2019-01-20]. http://arxiv.org/abs/1508.01991.
[15]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. doi: 10.1145/3065386
[16]	NGUYEN T H, GRISHMAN R. Relation extraction: Perspective from convolutional neural networks[C]//Proceedings of the Workshop on Vector Space Modeling for Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2015: 39-48.
[17]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the International Conference on Neural Information Processing Systems. New York: ACM, 2017: 6000-6010.
[18]	LI C, TIAN Y. Downstream model design of pre-trained language model for relation extraction task[EB/OL]. (2020-04-08) [2021-01-05]. http://arxiv.org/abs/2004.03786.
[19]	佟佳弘, 武志刚, 管霖, 等. 电力调度文本的自然语言理解与解析技术及应用[J]. 电网技术, 2020, 44(11): 4148-4156. TONG J H, WU Z G, GUAN L, et al. Power dispatching text analysis and application based on natural language understanding[J]. Power System Technology, 2020, 44(11): 4148-4156 (in Chinese).
[20]	蒋晨, 王渊, 胡俊华, 等. 基于深度学习的电力实体信息识别方法[J]. 电网技术, 2021, 45(6): 2141-2149. JIANG C, WANG Y, HU J H, et al. Power entity information recognition based on deep learning[J]. Power System Technology, 2021, 45(6): 2141-2149 (in Chinese).
[21]	LIU Y H, OTT M, GOYAL N, et al. RoBERTa: A robustly optimized BERT pretraining approach[EB/OL]. (2019-07-26) [2021-01-06]. http://arxiv.org/abs/1907.11692.
[22]	LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the Eighteenth International Conference on Machine Learning. New York: ACM, 2001: 282-289.
[23]	WEI M, XU Z P, HU J W. Entity relationship extraction based on Bi-LSTM and attention mechanism[C]//Proceedings of the International Conference on Artificial Intelligence and Information Systems. New York: ACM, 2021: 1-5.
[24]	杨秋勇 , 彭泽武, 苏华权, 等. 基于Bi-LSTM-CRF的中文电力实体识别[J]. 信息技术, 2021, 9: 45-50. YANG Q Y, PENG Z W, SU H Q, et al. Chinese electric power entity recognition based on Bi-LSTM-CRF[J]. Information Technology, 2021, 9: 45-50 (in Chinese).
[25]	DOZAT T, MANNING C D. Deep biaffine attention for neural dependency parsing [EB/OL]. (2016-11-06) [2021-02-10].https://arxiv.org/abs/1611.01734.
[26]	ZHOU W X, HUANG K, MA T Y, et al. Document-level relation extraction with adaptive thresholding and localized context pooling[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021, 35(16): 14612-14620.
[27]	RAM R V S, AKILANDESWARI A, DEVI S L. Linguistic features for named entity recognition using CRFs[C]//Proceedings of the International Conference on Asian Language Processing. Piscataway: IEEE Press, 2010: 158-161.
[28]	MA X Z, HOVY E. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2016: 781-792.