基于嵌套连接的多尺度红外与可见光图像融合

吴楠; 穆成坡; 贺杨; 刘太晔

doi:10.13700/j.bh.1001-5965.2023.0077

基于嵌套连接的多尺度红外与可见光图像融合

doi: 10.13700/j.bh.1001-5965.2023.0077

北京理工大学机电学院，北京 100080

详细信息

通讯作者:
E-mail：muchengpo@bit.edu.cn

中图分类号: V231.1；TP391.4
计量
- 文章访问数: 755
- HTML全文浏览量: 314
- PDF下载量: 49
- 被引次数: 0
出版历程
- 收稿日期: 2023-02-25
- 录用日期: 2023-03-31
- 网络出版日期: 2023-04-19
- 整期出版日期: 2025-02-28

Multi-scale infrared and visible image fusion based on nest connection

School of Mechatronical Engineering，Beijing Institute of Technology，Beijing 100080，China

More Information

Corresponding author: E-mail：muchengpo@bit.edu.cn

摘要

摘要:
为解决红外与可见光图像融合方法中依赖于人工设计的融合策略、运行时间过长以及无法提取多尺度深度特征的问题，提出了一种实时的基于嵌套连接的多尺度图像融合模型。特征提取器提取多尺度深层次特征，通过融合网络生成具有多尺度深度特征的特征图像，通过图像重构器得到融合图像。在公开数据集上与其他算法进行对比，在主观定性评价中，该算法在保持图像强度的同时能够包含锐利的图像边缘，在过度曝光、目标遮挡、细节模糊等复杂情况下具有更好的融合效果。客观指标对比中，在基于信息论、基于图像特征、基于图像结构相似性和人类感知度量4类9个指标上取得了5个第一2个第二的结果，另外2个指标的结果也处于较高水平，且融合时间大幅缩短。实验证明，该模型能够有效解决当前图像融合方法中存在的问题，具有较高的实用性。
- 图像融合 /
- 深度学习 /
- 嵌套连接 /
- 多尺度 /
- 实时
Abstract:
This work suggests a multi-scale real-time image fusion model based on nest connection, which aims to address the issues of lengthy running times, unnatural fusion strategies, and the incapacity to extract multi-scale deep features in existing infrared and visible image fusion approaches. Firstly, multi-scale deep features are extracted by a feature extractor. Then, feature maps with multi-scale deep features are generated from the fusion network. Finally, the fusion image is reconstructed by an image re-constructor. In comparison to other algorithms tested on common datasets, a subjective qualitative comparison shows that the algorithm in this paper can contain sharp image edges while maintaining image intensity and it has a better fusion effect under complex conditions such as over-exposure, target occlusion, and detail blurring. In objective quantitative metrics comparison, 5 best values and 2 second best values have been obtained on 9 evaluation metrics of 4 categories which are based on information theory, based on image feature, based on image structure similarity and human perception. Additionally, the other two metrics results have also showed good performance. Moreover, there has been a noticeable reduction in fusion time. The model presented in this paper has a high practicability and may successfully overcome the current issues with picture fusion techniques through experimental verification.
- image fusion /
- deep learning /
- nest connection /
- multi-scale /
- real-time

HTML全文

图 1 算法融合框架

Figure 1. The architecture of algorithm Fusion

下载: 全尺寸图片幻灯片

图 2 网络框架

Figure 2. Network framework

下载: 全尺寸图片幻灯片

图 3 基于嵌套连接的多尺度实时图像融合模型网络结构

Figure 3. The architecture of multi-scale real-time infrared and visible image fusion based on Nest Connection

下载: 全尺寸图片幻灯片

图 4 来自VIFB图像对的融合结果

Figure 4. Fusion results from VIFB image pairs

下载: 全尺寸图片幻灯片

图 5 来自RoadScene图像对的融合结果

Figure 5. Fusion results from RoadScene image pairs

下载: 全尺寸图片幻灯片

图 6 来自TNO图像对的融合结果

Figure 6. Fusion results from TNO image pairs

下载: 全尺寸图片幻灯片

图 7 融合质量评价指标

Figure 7. Indicators for evaluating the quality of fusion results

下载: 全尺寸图片幻灯片

表 1 网络结构参数

Table 1. Network structure parameters

模块	层	卷积核	输入	输出	步长	填充	激活
特征提取器	Conv1	3×3	2	16	1	1	leaky ReLU
	DPCB1		16	32
	DPCB2		32	48
	DPCB3		48	64
特征融合器	Conv2	3×3	32	16	1	1	leaky ReLU
	Conv3	3×3	64	32	1	1	leaky ReLU
	Conv4	3×3	96	48	1	1	leaky ReLU
	Conv5	3×3	48	32	1	1	leaky ReLU
	Conv6	3×3	96	32	1	1	leaky ReLU
图像重构器	Conv7	3×3	64	48	1	1	leaky ReLU
	Conv8	3×3	48	32	1	1	leaky ReLU
	Conv9	3×3	32	16	1	1	leaky ReLU
	Conv10	1×1	16	1	1	0	Tanh
DPCB	Max-pooling
	Conv	3×3	N_in	N_in	1	1	leaky ReLU
	Conv	3×3	N_in	N_out	1	1	leaky ReLU
上采样卷积	UpSamping
上采样卷积	Conv	1×1	N	N-16	1	0

下载: 导出CSV

表 2 实验评价结果

Table 2. Evaluation metric values of all methods

数据集	方法			评价指标
数据集	方法	CE	PSNR	Qabf	Qcb	Qcv	RMSE	SSIM	SD	VIF
RoadScene	DIDFuse(2,0,0)	1.070	59.275	0.511	0.511	289	0.077	1.344	69.910	0.735
	FusionGAN(0,0,0)	1.142	58.594	0.272	0.496	573	0.091	1.318	40.735	0.365
	GTF(0,3,0)	0.523	60.121	0.344	0.498	460	0.064	1.484	58.562	0.430
	U2Fusion(3,1,1)	0.822	60.194	0.526	0.573	464	0.063	1.534	39.507	0.437
	PIAFusion(1,1,0)	0.503	59.665	0.426	0.525	318	0.071	1.516	47.939	0.392
	NestFuse(0,0,3)	0.730	59.279	0.536	0.496	285	0.077	1.303	64.756	0.598
	STDFusion(0,1,0)	1.622	58.145	0.312	0.386	880	0.100	1.215	69.405	0.589
	SeAFusion(1,1,2)	0.673	59.389	0.540	0.545	248	0.075	1.486	57.127	0.573
	Proposed(2,2,3)	0.585	59.918	0.673	0.615	262	0.067	1.467	52.243	0.688
MSRS	DIDFuse(0,0,0)	2.274	59.374	0.241	0.375	717	0.077	0.631	34.740	0.419
	FusionGAN(0,1,0)	1.919	60.094	0.171	0.390	3892	0.064	1.266	20.460	0.122
	GTF(0,0,0)	0.891	59.994	0.271	0.432	4305	0.066	1.218	19.645	0.166
	U2Fusion(0,0,0)	2.694	59.184	0.380	0.393	1364	0.080	1.144	23.494	0.321
	PIAFusion(3,0,2)	0.638	60.003	0.696	0.583	89	0.066	1.434	52.559	0.795
	NestFuse(2,4,0)	0.697	60.188	0.723	0.567	68	0.062	1.428	50.046	0.715
	STDFusion(0,0,0)	4.365	59.361	0.497	0.350	301	0.076	1.183	37.551	0.414
	SeAFusion(1,3,3)	0.923	60.122	0.698	0.599	68	0.064	1.455	48.568	0.724
	Proposed(3,4,2)	0.742	60.152	0.737	0.635	64	0.064	1.448	48.830	0.748
TNO	DIDFuse(2,0,0)	1.069	59.714	0.351	0.507	315	0.071	1.282	45.935	0.643
	FusionGAN(0,0,0)	2.373	58.728	0.179	0.469	929	0.090	1.239	23.908	0.212
	GTF(2,1,1)	0.527	60.780	0.327	0.467	1159	0.055	1.379	29.650	0.223
	U2Fusion(3,1,2)	0.713	60.790	0.424	0.567	503	0.055	1.411	34.033	0.640
	PIAFusion(1,2,0)	0.786	59.957	0.559	0.596	247	0.067	1.399	39.709	0.362
	NestFuse(0,1,2)	0.686	60.064	0.420	0.521	220	0.065	1.286	39.968	0.343
	STDFusion(0,1,0)	1.222	59.398	0.339	0.464	382	0.076	1.217	40.790	0.271
	SeAFusion(1,0,0)	0.770	60.097	0.410	0.539	218	0.065	1.354	38.775	0.478
	Proposed(1,2,4)	0.600	60.238	0.486	0.611	222	0.063	1.320	38.724	0.479
VIFB	DIDFuse(2,0,0)	2.087	57.961	0.501	0.523	404	0.107	1.260	53.718	0.748
	FusionGAN(0,0,0)	1.947	58.633	0.223	0.357	1649	0.093	1.407	24.191	0.186
	GTF(0,2,0)	1.356	58.819	0.465	0.405	2178	0.090	1.433	27.859	0.256
	U2Fusion(3,0,2)	1.099	59.304	0.572	0.556	724	0.081	1.483	34.047	0.552
	PIAFusion(1,4,2)	1.051	58.633	0.657	0.575	250	0.093	1.457	46.840	0.621
	NestFuse(0,0,0)	1.535	58.536	0.494	0.435	397	0.095	1.451	43.936	0.386
	STDFusion(0,0,2)	1.669	58.692	0.503	0.411	511	0.095	1.419	45.281	0.456
	SeAFusion(0,1,2)	1.413	58.377	0.579	0.470	328	0.099	1.458	43.936	0.512
	Proposed(3,2,1)	1.010	58.658	0.693	0.581	278	0.092	1.450	45.171	0.657
数据集均值	DIDFuse(1,1,0)	1.625	59.081	0.401	0.479	431	0.083	1.129	51.076	0.636
	FusionGAN(0,0,0)	1.845	59.012	0.211	0.428	1761	0.084	1.308	27.323	0.221
	GTF(2,1,0)	0.824	59.929	0.352	0.450	2026	0.069	1.379	33.929	0.269
	U2Fusion(0,2,0)	1.332	59.868	0.476	0.522	764	0.070	1.393	32.770	0.487
	PIAFusion(1,3,2)	0.745	59.565	0.585	0.570	226	0.074	1.452	46.762	0.542
	NestFuse(0,1,0)	0.912	59.517	0.543	0.505	243	0.075	1.368	49.677	0.511
	STDFusion(0,0,1)	2.220	58.899	0.412	0.402	518	0.087	1.259	48.257	0.433
	SeAFusion(0,2,3)	0.945	59.496	0.557	0.538	216	0.076	1.438	47.102	0.572
	Proposed(5,0,2)	0.734	59.741	0.647	0.611	206	0.071	1.421	46.242	0.643
注：CE、Qcv和RMSE为负向指标，其余为正向指标，每个指标最佳3个值分别用红色、绿色和蓝色表示，每个方法后的3个数字为第1、第2、第3值的数量。

下载: 导出CSV

表 3 9种算法在4个数据集共40个图像对的平均运行时间

Table 3. Average running time of the 9 algorithms in 40 image pairs in 4 data sets

融合方法	时间均值/s
DIDFuse	0.161
FusionGAN	2.703
GTF	8.214
U2Fusion	3.495
PIAFusion	4.531
NestFuse	0.318
STDFusion	1.415
SeAFusion	0.089
本文	0.034
注：最佳3个值分别用红色、绿色和蓝色表示。

下载: 导出CSV

参考文献(21)

[1]	曹金亮, 薛伟民, 蔡雨含, 等. 一种红外与可见光图像信息交换融合网络[J]. 太原科技大学学报, 2022, 43(4): 277-282. CAO J L, XUE W M, CAI Y H, et al. Infrared and visible image information exchange-fusion algorithm[J]. Journal of Taiyuan University of Science and Technology, 2022, 43(4): 277-282 (in Chinese).
[2]	杨叶君, 刘刚, 肖刚, 等. 基于自适应特征增强和生成器路径交互的红外与可见光图像融合[J]. 激光与光电子学进展, 2022: 1-18. YANG Y J, LIU G, XIAO G, et al. Infrared and visible image fusion based on adaptive feature enhancement and generator path interaction[J]. Laser & Optoelectronics Progress, 2022: 1-18(in Chinese).
[3]	ZHANG X C, YE P, LEUNG H, et al. Object fusion tracking based on visible and infrared images: a comprehensive review[J]. Information Fusion, 2020, 63: 166-187.
[4]	王建中, 徐浩楠, 王洪枫, 等. 基于残差密集块和自编码网络的红外与可见光图像融合[J]. 北京理工大学学报, 2021, 41(10): 1077-1083. WANG J Z, XU H N, WANG H F, et al. Infrared and visible image fusion based on residual dense block and auto-encoder network[J]. Transactions of Beijing Institute of Technology, 2021, 41(10): 1077-1083 (in Chinese).
[5]	宋加文, 朱大明, 付志涛, 等. 基于对比度和结构提取的红外与可见光图像融合[J]. 激光与光电子学进展, 2023, 60(14): 1410005. SONG J W, ZHU D M, FU Z T, et al. Infrared and visible image fusion based on contrast and structure extraction[J]. Laser & Optoelectronics Progress, 2023, 60(14): 1410005(in Chinese).
[6]	张冬冬, 王春平, 付强. 深度学习框架下的红外与可见光图像融合算法综述[J]. 激光与红外, 2022, 52(9): 1288-1298. ZHANG D D, WANG C P, FU Q. Overview of infrared and visible image fusion algorithms based on deep learning framework[J]. Laser & Infrared, 2022, 52(9): 1288-1298 (in Chinese).
[7]	魏琦, 赵娟. 红外和可见光图像融合算法的研究进展[J]. 计算机科学, 2023, 50(2): 11. WEI Q, ZHAO J. Research progress of infrared and visible image fusion algorithms[J]. Computer Science, 2023, 50(2): 11(in Chinese).
[8]	LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation Measurement, 2020, 69(12): 9645-9656.
[9]	MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[10]	MA J Y, TANG L F, XU M L, et al. STDFusionNet: an infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5009513.
[11]	ZHAO Z X, XU S, ZHANG C X, et al. DIDFuse: deep image decomposition for infrared and visible image fusion[C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2020: 1-5.
[12]	TANG L F, YUAN J T, ZHANG H, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83-84: 79-92.
[13]	汤璐, 卢荣胜, 史艳琼, 等. 基于YCbCr空间融合的高动态范围成像方法[J]. 激光与光电子学进展, 2022, 59(14): 308-319. TANG L, LU R S, SHI Y Q, et al. High dynamic range imaging method based on YCbCr Color space fusion[J]. Laser & Optoelectronics Progress, 2022, 59(14): 308-319(in Chinese).
[14]	XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
[15]	ZHOU Z W, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet++: a nested U-net architecture for medical image segmentation[J]. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018. Granada, Spain: [s.n.], 2018, 11045: 3-11.
[16]	TANG L F, YUAN J T, MA J Y. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42.
[17]	闵莉, 曹思健, 赵怀慈, 等. 改进残差密集生成对抗网络的红外与可见光图像融合[J]. 控制与决策, 2022, 37(1): 1-8. MIN L, CAO S J, ZHAO H C, et al. Infrared and visible image fusion with improved residual dense generative adversarial network[J]. Control and Decision, 2022, 37(1): 1-8(in Chinese).
[18]	MA J Y, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109.
[19]	ZHANG X C, YE P, XIAO G. VIFB: a visible and infrared image fusion benchmark[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2020: 468-478.
[20]	LI H, WU X J, KITTLER J. RFN-Nest: an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86.
[21]	张勇, 马东玺, 闫鹏程. 灰度夜视融合图像质量结构相似度评价方法[J]. 光子学报, 2014, 43(S1): 242-248. ZHANG Y, MA D X, YAN P C. Structural similarity evaluation method to night vision fusion image quality[J]. Acta Photonica Sinica, 2014, 43(S1): 242-248(in Chinese).