| Citation: | ZHANG Y，SUN M L，LIU Z Y，et al. Infrared image super-resolution reconstruction based on visible image guidance and recursive fusion[J]. Journal of Beijing University of Aeronautics and Astronautics，2025，51（11）：3662-3673 （in Chinese） doi: 10.13700/j.bh.1001-5965.2023.0590
|
Due to hardware limitations, infrared images typically suffer from low resolution and blurred details when captured. Although visible light images can effectively guide the super-resolution reconstruction of infrared images, differences in image detail caused by their distinct imaging principles often result in issues such as blurring and ghosting during reconstruction. This paper proposes a super-resolution reconstruction network for infrared images based on visible image guidance and recursive fusion. In this network, a flowing Fourier residual module is designed to extract different frequency information from infrared and visible images using modules at different depths, enabling each module to focus on the appropriate frequency information. Simultaneously, a hybrid attention module is employed to capture detailed information in multimodal images from both channel and spatial perspectives, and to fuse it in a complementary manner. Based on this, a global recursive fusion branch is designed to model the correlations across multiple feature layers and adaptively fuse them, thus generating clearer high-resolution infrared images. Experimental results show that compared with comparison methods like Deep-ISTA and PAG-SR, the proposed method demonstrates better level in objective evaluation indicators. In terms of subjective visual comparison, the images reconstructed by this method exhibit clearer textures, fewer artifacts, and better object discrimination in complex environments.
| [1] |
GOLDBERG A C, FISCHER T, DERZKO Z I. Application of dual-band infrared focal plane arrays to tactical and strategic military problems[C]//Proceedings of the Infrared Technology and Applications XXVIII. Bellingham: SPIE, 2003.
|
| [2] |
熊光明, 罗震, 孙冬, 等. 基于红外相机和毫米波雷达融合的烟雾遮挡目标检测与跟踪技术[J]. 兵工学报, 2024, 45(3): 893-906.
XIONG G M, LUO Z, SUN D, et al. Detection and tracking technology of smoke occlusion target based on infrared camera and millimeter wave radar fusion[J]. Acta Armamentarii, 2024, 45(3): 893-906(in Chinese).
|
| [3] |
QI H, DIAKIDES N A. Thermal infrared imaging in early breast cancer detection-a survey of recent research[C]//Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway: IEEE Press, 2004: 1109-1112.
|
| [4] |
张倩, 杨颖, 刘刚, 等. 融合数据增强与改进ResNet34的奶牛热红外图像乳腺炎检测[J]. 光谱学与光谱分析, 2023, 43(1): 280-288. doi: 10.3964/j.issn.1000-0593(2023)01-0280-09
ZHANG Q, YANG Y, LIU G, et al. Detection of dairy cow mastitis from thermal images by data enhancement and improved ResNet34[J]. Spectroscopy and Spectral Analysis, 2023, 43(1): 280-288(in Chinese). doi: 10.3964/j.issn.1000-0593(2023)01-0280-09
|
| [5] |
侯义锋, 丁畅, 刘海, 等. 逆光海况下低质量红外目标的增强与识别[J]. 光学学报, 2023, 43(6): 0612003. doi: 10.3788/AOS221387
HOU Y F, DING C, LIU H, et al. Enhancement and recognition of infrared target with low quality under backlight maritime condition[J]. Acta Optica Sinica, 2023, 43(6): 0612003(in Chinese). doi: 10.3788/AOS221387
|
| [6] |
翁静, 袁盼, 王铭赫, 等. 基于支持向量机的泄漏气体云团热成像检测方法[J]. 光学学报, 2022, 42(9): 0911002. doi: 10.3788/AOS202242.0911002
WENG J, YUAN P, WANG M H, et al. Thermal imaging detection method of leak gas clouds based on support vector machine[J]. Acta Optica Sinica, 2022, 42(9): 0911002(in Chinese). doi: 10.3788/AOS202242.0911002
|
| [7] |
XIONG K N, JIANG J B, PAN Y Y, et al. Deep learning approach for detection of underground natural gas micro-leakage using infrared thermal images[J]. Sensors, 2022, 22(14): 5322. doi: 10.3390/s22145322
|
| [8] |
ZHANG L, WU X L. An edge-guided image interpolation algorithm via directional filtering and data fusion[J]. IEEE Transactions on Image Processing, 2006, 15(8): 2226-2238. doi: 10.1109/TIP.2006.877407
|
| [9] |
PAPYAN V, ELAD M. Multi-scale patch-based image restoration[J]. IEEE Transactions on Image Processing, 2016, 25(1): 249-261. doi: 10.1109/TIP.2015.2499698
|
| [10] |
ZHANG K, VAN GOOL L, TIMOFTE R. Deep unfolding network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 3214-3223.
|
| [11] |
SUN Y M, ZHANG Y, LIU S D, et al. Image super-resolution using supervised multi-scale feature extraction network[J]. Multimedia Tools and Applications, 2021, 80(2): 1995-2008. doi: 10.1007/s11042-020-09488-z
|
| [12] |
ZHANG X D, ZENG H, ZHANG L. Edge-oriented convolution block for real-time super resolution on mobile devices[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 4034-4043.
|
| [13] |
ALMASRI F, DEBEIR O. Multimodal sensor fusion in single thermal image super-resolution[C]//Proceedings of the 14th Asian Conference on Computer Vision. Berlin: Springer, 2019: 418-433.
|
| [14] |
ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 2472-2481.
|
| [15] |
ZHANG Y, LIU Z Y, LIU S D, et al. Frequency aggregation network for blind super-resolution based on degradation representation[J]. Digital Signal Processing, 2023, 133: 103837. doi: 10.1016/j.dsp.2022.103837
|
| [16] |
KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2016: 1637-1645.
|
| [17] |
LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE Press, 2017: 1132-1140.
|
| [18] |
ZHANG Y, XU F, SUN Y, et al. Spatial and frequency information fusion transformer for image super-resolution[J]. Neural Networks, 2025, 187: 107351.
|
| [19] |
QIU Y J, WANG R X, TAO D P, et al. Embedded block residual network: a recursive restoration model for single-image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 4179-4188.
|
| [20] |
CHOI Y, KIM N, HWANG S, et al. Thermal image enhancement using convolutional neural network[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2016: 223-230.
|
| [21] |
ALMASRI F, DEBEIR O. Multimodal sensor fusion in single thermal image super-resolution[C]//Proceedings of the Asian Conference on Computer Vision. Berlin: Springer, 2018: 418-433.
|
| [22] |
LEE K, LEE J, LEE J, et al. Brightness-based convolutional neural network for thermal image enhancement[J]. IEEE Access, 2017, 5: 26867-26879. doi: 10.1109/ACCESS.2017.2769687
|
| [23] |
HAN T Y, KIM Y J, SONG B C. Convolutional neural network-based infrared image super resolution under low light environment[C]//Proceedings of the 25th European Signal Processing Conference. Piscataway: IEEE Press, 2017: 803-807.
|
| [24] |
GUPTA H, MITRA K. Pyramidal edge-maps and attention based guided thermal super-resolution[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2020: 698-715.
|
| [25] |
MAO X, LIU Y, SHEN W, et al. Deep residual fourier transformation for single image deblurring[EB/OL]. (2022-11-29)[2023-09-01]. https://arxiv.org/abs/2111.11745.
|
| [26] |
BRIGHAM E O, MORROW R E. The fast Fourier transform[J]. IEEE Spectrum, 1967, 4(12): 63-70. doi: 10.1109/MSPEC.1967.5217220
|
| [27] |
CAO J M, LI Y Y, SUN M C, et al. DO-Conv: depthwise over-parameterized convolutional layer[J]. IEEE Transactions on Image Processing, 2022, 31: 3726-3736. doi: 10.1109/TIP.2022.3175432
|
| [28] |
JI S W, XU W, YANG M, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 221-231. doi: 10.1109/TPAMI.2012.59
|
| [29] |
NIU B, WEN W, REN W, et al. Single image super-resolution via a holistic attention network[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin: Springer, 2020: 191-207.
|
| [30] |
ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 586-595.
|
| [31] |
LEWIS J. FLIR releases machine learning thermal dataset for advanced driver assistance systems[J]. Vision Systems Design, 2018, 23(9): 1-10.
|
| [32] |
ZHANG K, ZUO W M, ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 3262-3271.
|
| [33] |
ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2018: 294-310.
|
| [34] |
DAI T, CAI J, ZHANG Y, et al. Second-order attention network for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2019: 11065-11074.
|
| [35] |
FERSTL D, REINBACHER C, RANFTL R, et al. Image guided depth upsampling using anisotropic total generalized variation[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2013: 993-1000.
|
| [36] |
BARRON J T, POOLE B. The fast bilateral solver[C]//Proceedings of the 14th European Conference on Computer Vision. Berlin: Springer, 2016: 617-632.
|
| [37] |
KOPF J, COHEN M F, LISCHINSKI D, et al. Joint bilateral upsampling[J]. ACM Transactions on Graphics, 2007, 26(99): 96. doi: 10.1145/1276377.1276497
|
| [38] |
HAM B, CHO M, PONCE J. Robust image filtering using joint static and dynamic guidance[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2015: 4823-4831.
|
| [39] |
HUI T W, LOY C C, TANG X O. Depth map super-resolution by deep multi-scale guidance[C]//Proceedings of the 14th European Conference on Computer Vision. Berlin: Springer, 2016: 353-369.
|
| [40] |
DE LUTIO R, D’ARONCO S, WEGNER J D, et al. Guided super-resolution as pixel-to-pixel transformation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 8828-8836.
|
| [41] |
DENG X, DRAGOTTI P L. Deep coupled ISTA network for multi-modal image super-resolution[J]. IEEE Transactions on Image Processing, 2019, 29: 1683-1698.
|
Figures(10) / Tables(3)
Copyright © Journal of Beijing University of Aeronautics and Astronautics
Address: Editorial Department of Journal of Beijing University of Aeronautics and Astronautics, 37 Xueyuan Road, Haidian District, Beijing Post Code: 100191 Email: jbuaa@buaa.edu.cn
Supported by:
Beijing Renhe Information Technology Co., Ltd.
DownLoad:
