Super-resolution reconstruction algorithm based on deep learning mechanism and wavelet fusion
-
摘要:
深度学习技术在超分辨率重建领域中发展迅速。为了进一步提升重建图像的质量和视觉效果，针对基于生成对抗网络（GAN）的超分辨率重建算法重建图像的纹理放大后不自然的问题，提出了一种结合小波变换和生成对抗网络的超分辨率重建算法。所提算法在生成对抗网络中将小波分解的每个分量在各自独立的子网中进行训练，实现网络对小波系数的预测，有效地重建出具有丰富的全局信息和局部纹理细节信息的高分辨率图像。实验结果表明，对比基于生成对抗网络的算法，所提算法重建图像的客观评价指标峰值信噪比（PSNR）和结构相似性分别能提高至少0.99 dB和0.031。
-
关键词:
- 小波变换 /
- 生成对抗网络(GAN) /
- 超分辨率重建 /
- 深度学习 /
- 多分辨分析
Abstract:Deep learning technology has developed rapidly in the field of super-resolution reconstruction. In order to further improve the quality and visual effect of reconstructed images, this paper proposes a super-resolution reconstruction based on wavelet transform and generative adversarial networks (GAN) for the unnatural problem of texture reconstruction based on the super-resolution reconstruction algorithm of GAN. In this paper, each component of the wavelet decomposition in the GAN is trained in separate subnets to realize the prediction of wavelet coefficients by the network. Effectively reconstruct high-resolution images with rich global information and local texture details. The experimental results show that the peak signal-to-noise ratio (PSNR) and structural similarity of the objective evaluation index of the reconstructed image can be improved by at least 0.99 dB and 0.031, respectively, based on the algorithm of GAN.
-
图 4 Set14数据集中图像“barbara”测试结果
Figure 4. Test results of "barbara" image in Set14 dataset
图 5 Set14数据集中图像“monarch”测试结果
Figure 5. Test results of "monarch" image in Set14 dataset
表 1 PSNR和SSIM的测试结果
Table 1. Test results of PSNR and SSIM
数据集 PSNR/dB, SSIM SRGAN 本文算法 Set5 29.50，0.841 30.63，0.897 Set14 25.95，0.740 26.94，0.771 BSD100 25.16，0.669 26.91，0.728 
下载: 导出CSV
表 2 FSIM和UIQ的测试结果
Table 2. Test results of FSIM and UIQ
数据集 FSIM，UIQ SRGAN 本文算法 Set5 0.907，0.925 0.929，0.935 Set14 0.891，0.973 0.900，0.978 BSD100 0.828，0.981 0.832，0.984
下载: 导出CSV
表 3 重建图像LBP特征图与原始图像LBP特征图之间方差
Table 3. Variance between LBP feature map of reconstructed image and LBP feature map of original image
图像 方差 SRGAN 本文算法 bird 0.124 0.061 barbara 0.261 0.216 monarch 0.187 0.121
下载: 导出CSV
-
[1] 曹杨.基于学习的超分辨率复原技术研究[D].北京: 北京工业大学, 2009: 3. http://www.wanfangdata.com.cn/details/detail.do?_type=degree&id=Y1569683CAO Y.Research on learning-based super-resolution restoration technology[D].Beijing: Beijing University of Technology, 2009: 3(in Chinese). http://www.wanfangdata.com.cn/details/detail.do?_type=degree&id=Y1569683 [2] TRINH D H, LUONG M, DIBOS F, et al.Novel example-based method for super-resolution and denoising of medical images[J].IEEE Transactions on Image Processing, 2014, 23(4):1882-1895. doi: 10.1109/TIP.2014.2308422 [3] LEI Z, YANG J, XUE B, et al.Super-resolution reconstruction of Chang'e-1 satellite CCD stereo camera images[J].Infrared and Laser Engineering, 2012, 41(2):404-408. [4] SHEN H, NG M K, LI P, et al.Super-resolution reconstruction algorithm to MODIS remote sensing images[J].Computer Journal, 2009, 52(1):90-100. http://cn.bing.com/academic/profile?id=499974c692e27f810c774c0356d3502f&encoded=0&v=paper_preview&mkt=zh-cn [5] 石晓勃, 高树辉.基于警用图像处理系统对超分辨率图像重建分析[J].中国人民公安大学学报, 2016, 22(4):14-17. http://d.old.wanfangdata.com.cn/Periodical/gadxxb-zrkxb201604004SHI X B, GAO B.Analysis of super-resolution image reconstruction based on police image processing system[J].Journal of Chinese People's Public Security University, 2016, 22(4):14-17(in Chinese). http://d.old.wanfangdata.com.cn/Periodical/gadxxb-zrkxb201604004 [6] 杨宇翔.图像超分辨率算法研究[D].北京: 中国科学技术大学, 2013: 14-28.YANG Y X.Image super resolution algorithm research[D].Beijing: University of Science and Technology of China, 2013: 14-28(in Chinese). [7] PARKER J A, KENYON R V, TROXEL D E.Comparison of interpolating methods for image resampling[J].IEEE Transactions on Medical Imaging, 1983, 2(1):31-39. doi: 10.1109/TMI.1983.4307610 [8] HOU H S, ANDREWS H.Cubic splines for image interpolation and digital filtering[J].IEEE Transactions on Acoustics Speech & Signal Processing, 1978, 26(6):508-517. doi: 10.1109-TASSP.1978.1163154/ [9] LI X, ORCHARD M T.New edge-directed interpolation[J].IEEE Transactions on Image Processing, 2001, 10(10):1521-1527. doi: 10.1109/83.951537 [10] OZKAN M K, TEKALP A M, SEZAN M I.POCS-based restoration of space-varying blurred images[J].IEEE Transactions on Image Processing, 1994, 3(4):450-454. doi: 10.1109/83.298398 [11] IRANI M, PELEG S.Improving resolution by image registration[J].CVGIP Graphical Models & Image Processing, 1991, 53(3):231-239. doi: 10.1017-S1041610209990342/ [12] SCHULTZ R R, STEVENSON R L.A Bayesian approach to image expansion for improved definition[J].IEEE Transactions on Image Processing, 1994, 3(3):233-242. http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=c4dbb80e6666be9055d6cde3254f2682 [13] YANG J, WANG Z, LIN Z, et al.Coupled dictionary training for image super-resolution[J].IEEE Transactions on Image Processing, 2012, 21(8):3467-3478. doi: 10.1109/TIP.2012.2192127 [14] DONG C, LO C C, HE K, et al.Learning a deep convolutional network for image super-resolution[C]//European Conference on Computer Vision.Berlin: Springer, 2014: 184-199. doi: 10.1007/978-3-319-10593-2_13 [15] KIM J, LEE J K, LEE K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2016: 1637-1645. [16] SHI W, CABALLERO J, HUSZAR F, et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2016: 1874-1883. https://www.researchgate.net/publication/307925703_Real-Time_Single_Image_and_Video_Super-Resolution_Using_an_Efficient_Sub-Pixel_Convolutional_Neural_Network [17] TONG T, LI G, LIU X, et al.Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision.Piscataway, NJ: IEEE Press, 2017: 4809-4817. https://www.researchgate.net/publication/322057608_Image_Super-Resolution_Using_Dense_Skip_Connections [18] LEDIG C, WANG Z, SHI W, et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2017: 4681-4690. https://www.researchgate.net/publication/308152499_Photo-Realistic_Single_Image_Super-Resolution_Using_a_Generative_Adversarial_Network [19] LAI W S, HUANG J B, AHUJA N, et al.Deep Laplacian pyramid networks for fast and accurate super-resolution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 1:5835-5843. http://cn.bing.com/academic/profile?id=87defcb6cbf436c9e84391087df5f972&encoded=0&v=paper_preview&mkt=zh-cn [20] GUO L, HE Z M.A projection on convex sets super-resolution algorithm using wavelet transform[C]//2008 9th International Conference on Signal Processing.Piscataway, NJ: IEEE Press, 2008: 1039-1041. https://www.researchgate.net/publication/224355735_A_projection_on_Convex_Sets_super-resolution_algorithm_using_wavelet_transform [21] AKBARZADEH S, HASSAN G, FAEZE V.An efficient single image super resolution algorithm based on wavelet transforms[C]//2015 9th Iranian Conference on Machine Vision and Image Processing (MVIP).Piscataway, NJ: IEEE Press, 2015: 111-114. https://www.researchgate.net/publication/284174229_An_efficient_single_image_super_resolution_algorithm_based_on_wavelet_transforms [22] KUMAR N, VERMA R, SETHI A.Convolutional neural networks for wavelet domain super resolution[J].Pattern Recognition Letters, 2017, 90:65-71. doi: 10.1016/j.patrec.2017.03.014 [23] HUANG H, HE R, SUN Z, et al.Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution[C]//Proceedings of the IEEE International Conference on Computer Vision.Piscataway, NJ: IEEE Press, 2017: 1689-1697. https://www.researchgate.net/publication/322058760_Wavelet-SRNet_A_Wavelet-Based_CNN_for_Multi-scale_Face_Super_Resolution [24] HE K, ZHANG X, REN S, et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2016: 770-778. https://www.researchgate.net/publication/307964703_Deep_Residual_Learning [25] LIM B, SON S, KIM H, et al.Enhanced deep residual networks for single image super-resolution[C]//2011 IEEE Computer Vision and Pattern Recognition Workshops.Piscataway, NJ: IEEE Press, 2017: 1132-1140. https://www.researchgate.net/publication/318337451_Enhanced_Deep_Residual_Networks_for_Single_Image_Super-Resolution [26] TIMOFTE R, DE SMET V, VAN GOOL L.Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway, NJ: IEEE Press, 2013: 1920-1927. https://www.researchgate.net/publication/262323537_Anchored_Neighborhood_Regression_for_Fast_Example-Based_Super-Resolution [27] RUSSAKOVSKY O, DENG J, SU H, et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision, 2014, 115(3):211-252. http://d.old.wanfangdata.com.cn/NSTLHY/NSTL_HYCC0214533907/ [28] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al.Neighbor embedding based single-image super-resolution using semi-nonnegative matrix factorization[C]//2012 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP).Piscataway, NJ: IEEE Press, 2012: 1289-1292. https://www.researchgate.net/publication/261466481_Neighbor_embedding_based_single-image_super-resolution_using_Semi-Nonnegative_Matrix_Factorization [29] ZEYDE R, ELAD M, PROTTER M.On single image scale-up using sparse-representations[C]//International Conference on Curves and Surfaces.Berlin: Springer, 2010: 711-730. [30] MARTIN D, FOWLKES C, TAL D, et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//IEEE Conference on Computer Vision.Piscataway, NJ: IEEE Press, 2001: 416-423. https://www.researchgate.net/publication/3906161_A_Database_of_Human_Segmented_Natural_Images_and_its_Application_to_Evaluating_Segmentation_Algorithms_and_Measuring_Ecological_Statistics [31] WANG Z, BOVIK A C, SHEIKH H R, et al.Image quality assessment:From error visibility to structural similarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612. doi: 10.1109/TIP.2003.819861 [32] ZHANG L, ZHANG L, MOU X, et al.FSIM:A feature similarity index for image quality assessment[J].IEEE Transactions on Image Processing, 2011, 20(8):2378-2386. doi: 10.1109/TIP.2011.2109730 [33] WANG Z, BOVIK A C.A universal image quality index[J].IEEE Signal Processing Letters, 2002, 9(3):81-84. http://d.old.wanfangdata.com.cn/OAPaper/oai_doaj-articles_9471bc783cfb47b797e41099fa1c8cf0 [34] OJALA T, PIETILAINEN M, HARWOOD D.Performance evaluation of texture measures with classification based on Kullback discrimination of distributions[C]//Proceedings of 12th International Conference on Pattern Recognition.Piscataway, NJ: IEEE Press, 1994, 1: 582-585. https://www.researchgate.net/publication/3684008_Performance_evaluation_of_texture_measures_with_classification_based_on_Kullback_discrimination_of_distributions -
下载:
点击查看大图
图(6) / 表(3)
计量
- 文章访问数: 1330
- HTML全文浏览量: 219
- PDF下载量: 321
- 被引次数: 0
