留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于残缺图像样本的生成对抗网络图像修复方法

李海燕 黄和富 郭磊 李海江 陈建华 李红松

李海燕, 黄和富, 郭磊, 等 . 基于残缺图像样本的生成对抗网络图像修复方法[J]. 北京航空航天大学学报, 2021, 47(10): 1949-1958. doi: 10.13700/j.bh.1001-5965.2020.0374
引用本文: 李海燕, 黄和富, 郭磊, 等 . 基于残缺图像样本的生成对抗网络图像修复方法[J]. 北京航空航天大学学报, 2021, 47(10): 1949-1958. doi: 10.13700/j.bh.1001-5965.2020.0374
LI Haiyan, HUANG Hefu, GUO Lei, et al. Image inpainting method based on incomplete image samples in generative adversarial network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(10): 1949-1958. doi: 10.13700/j.bh.1001-5965.2020.0374(in Chinese)
Citation: LI Haiyan, HUANG Hefu, GUO Lei, et al. Image inpainting method based on incomplete image samples in generative adversarial network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(10): 1949-1958. doi: 10.13700/j.bh.1001-5965.2020.0374(in Chinese)

基于残缺图像样本的生成对抗网络图像修复方法

doi: 10.13700/j.bh.1001-5965.2020.0374
基金项目: 

云南省万人计划"教学名师" 

云南省基础研究计划重点项目 202101AS070031

国家自然科学基金 61861045

详细信息
    通讯作者:

    郭磊, E-mail: lei_guo@ynu.edu.cn

  • 中图分类号: TP391.4

Image inpainting method based on incomplete image samples in generative adversarial network

Funds: 

"Famous Teacher" of Yunnan 10000 Talents Program 

Basic Research Key Project of Yunnan Province 202101AS070031

National Natural Science Foundation of China 61861045

More Information
  • 摘要:

    针对大面积图像修复缺失严重时,需要完整且高质量训练样本的问题,提出了一种将残缺或含噪图像样本作为训练集的双生成器深度卷积生成对抗网络(DGDCGAN)模型。构建两个生成器和一个鉴别器以解决单一生成器收敛慢的问题,用残缺图像样本作为训练集,通过交叉计算、搜索损失区域类似的图像信息作为训练生成模型的样本,收敛速度更快。鉴别器损失函数改进为输出的Wasserstein距离,使用自适应估计算法优化生成器损失函数和鉴别器损失函数的模型参数,最小化两两图像之间的总距离差,使用鉴别模型和修复图像总距离变化均方差最小化两个指标优化修复结果。在4个公开数据集上进行主客观实验,结果表明:所提方法能使用残缺图像样本作为训练集,有效实现大面积失真图像的修复,且收敛速度和修复效果优于现有图像修复方法。

     

  • 图 1  GAN模型训练

    Figure 1.  GAN model training

    图 2  DGDCGAN的网络结构

    Figure 2.  Network structure of DGDCGAN

    图 3  图像残缺分布

    Figure 3.  Image fragmentation distribution

    图 4  兔子训练样本流程

    Figure 4.  Rabbit training sample flow

    图 5  MNIST数字的残缺图像

    Figure 5.  Incomplete image of MNIST number

    图 6  熊的训练图像样本

    Figure 6.  Training samples of bears

    图 7  随迭代次数增加损失函数和修复效果

    Figure 7.  Results of loss function and inpainting as the number of iterations increases

    图 8  使用单个生成器时模型损失函数变化

    Figure 8.  Change of model loss values when using single generator

    图 9  CelebA数据集修复效果对比

    Figure 9.  Comparison of inpainting result of CelebA dataset

    图 10  Cifar10数据集修复效果对比

    Figure 10.  Comparison of inpainting result of Cifar10 dataset

    图 11  SVHN数据集中门牌号修复效果对比

    Figure 11.  Comparison of inpainting results of door number in SVHN dataset

    图 12  修复方法效果对比

    Figure 12.  Results comparison of inpainting methods

    表  1  不同方法的PSNR和SSIM对比

    Table  1.   PSNR/SSIM comparison of different methods

    方法 PSNR SSIM
    Car Black man Horse Car Black man Horse
    FMM[1] 24.49 22.25 23.03 0.79 0.73 0.75
    PM[4] 23.39 20.40 23.98 0.72 0.70 0.76
    SI[20] 31.12 40.39 38.35 0.97 0.96 0.91
    GAN_1[26] 24.54 25.57 22.79 0.91 0.91 0.83
    GAN_2[27] 30.38 36.22 37.98 0.94 0.91 0.95
    本文方法 32.19 41.06 38.59 0.99 0.99 0.97
    下载: 导出CSV
  • [1] BERTALMÍO M, BERTOZZI A L, SAPIRO G. Navier-Stokes, fluid dynamics, and image and video inpainting[C]//2001 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2001: 7176886.
    [2] LIU D, SUN X, WU F, et al. Image compression with edge-based inpainting[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(10): 1273-1287. doi: 10.1109/TCSVT.2007.903663
    [3] BUYSSENS P, DAISY M, TSCHUMPERLE D, et al. Exemplar-based inpainting: Technical review and new heuristics for better geometric reconstructions[J]. IEEE Transactions on Image Processing, 2015, 24(6): 1809-1824.
    [4] DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: Combining inconsistent images using patch-based synthesis[J]. ACM Transactions on Graphics, 2012, 31(4): 82.
    [5] HUANG J B, KANG S B, AHUJA N, et al. Image completion using planar structure guidance[J]. ACM Transactions on Graphics, 2014, 33(4): 1-10.
    [6] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24.
    [7] WEXLER Y, SHECHTMAN E, IRANI M. Space-time completion of video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3): 463-476. doi: 10.1109/TPAMI.2007.60
    [8] HAYS J, EFROS A A. Scene completion using millions of photographs[J]. ACM Transactions on Graphics, 2007, 26(3): 4. doi: 10.1145/1276377.1276382
    [9] DEMIR U, UNAL G. Patch-based image inpainting with generative adversarial networks[EB/OL]. (2018-03-20)[2020-07-25]. https://arxiv.org/abs/1803.07422v1.
    [10] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning[EB/OL]. (2019-01-11)[2020-07-25]. https://arxiv.org/abs/1901.00212?context=cs.
    [11] XIE J, XU L, CHEN E. Image denoising and inpainting with deep neural networks[C]//International Conference on Neural Information Processing Systems, 2012, 1: 341-349.
    [12] YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 17355590.
    [13] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
    [14] LI Y, LIU S, YANG J, et al. Generative face completion[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 17355119.
    [15] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: Feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2016: 16526723.
    [16] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680.
    [17] BROCK A, DONAHUE J, SIMONYAN K. Large scale GAN training for high fidelity natural image synthesis[EB/OL]. (2018-09-28)[2020-07-25]. https://arxiv.org/abs/1809.11096v1.
    [18] AHMADI M, NEST T, ABDELNAIM M, et al. Reproducing AmbientGAN: Generative models from lossy measurements[EB/OL]. (2018-10-23)[2020-07-25]. https://arxiv.org/abs/1810.10108.
    [19] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs[C]//International Conference on Neural Information Processing Systems, 2016: 2234-2242.
    [20] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2017: 17453078.
    [21] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL]. (2017-01-26)[2020-07-25]. https://arxiv.org/abs/1701.07875.
    [22] ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[EB/OL]. (2017-01-17)[2020-07-25]. https://arxiv.org/abs/1701.04862.
    [23] CHEN Y, HU H. An improved method for semantic image inpainting with GANs: Progressive inpainting[J]. Neural Processing Letters, 2019, 49(3): 1355-1367. doi: 10.1007/s11063-018-9877-6
    [24] LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]//2015 IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2015: 3730-3738.
    [25] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
    [26] 李雪瑾, 李昕, 徐艳杰. 基于生成对抗网络的数字图像修复技术[J]. 电子测量与仪器学报, 2019, 33(1): 40-46. https://www.cnki.com.cn/Article/CJFDTOTAL-DZIY201901006.htm

    LI X J, LI X, XU Y J. Digital image restoration technology based on generative adversarial networks[J]. Journal of Electronic Measurement and Instrumentation, 2019, 33(1): 40-46(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-DZIY201901006.htm
    [27] 李天成, 何嘉. 一种基于生成对抗网络的图像修复算法[J]. 计算机应用与软件, 2019, 36(12): 195-200. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201912032.htm

    LI T C, HE J. An image inpainting algorithm based on generative adversarial networks[J]. Computer Applications and Software, 2019, 36(12): 195-200(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201912032.htm
  • 加载中
图(12) / 表(1)
计量
  • 文章访问数:  463
  • HTML全文浏览量:  89
  • PDF下载量:  172
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-07-30
  • 录用日期:  2020-10-25
  • 网络出版日期:  2021-10-20

目录

    /

    返回文章
    返回
    常见问答