留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

结构引导的渐进式生成对抗壁画修复

陈永 陈锦 陶美风

陈永,陈锦,陶美风. 结构引导的渐进式生成对抗壁画修复[J]. 北京航空航天大学学报,2023,49(6):1247-1259 doi: 10.13700/j.bh.1001-5965.2021.0440
引用本文: 陈永,陈锦,陶美风. 结构引导的渐进式生成对抗壁画修复[J]. 北京航空航天大学学报,2023,49(6):1247-1259 doi: 10.13700/j.bh.1001-5965.2021.0440
CHEN Y,CHEN J,TAO M F. Mural inpainting progressive generative adversarial networks based on structure guided[J]. Journal of Beijing University of Aeronautics and Astronautics,2023,49(6):1247-1259 (in Chinese) doi: 10.13700/j.bh.1001-5965.2021.0440
Citation: CHEN Y,CHEN J,TAO M F. Mural inpainting progressive generative adversarial networks based on structure guided[J]. Journal of Beijing University of Aeronautics and Astronautics,2023,49(6):1247-1259 (in Chinese) doi: 10.13700/j.bh.1001-5965.2021.0440

结构引导的渐进式生成对抗壁画修复

doi: 10.13700/j.bh.1001-5965.2021.0440
基金项目: 国家自然科学基金(61963023);教育部人文社会科学研究青年基金(19YJC760012);兰州交通大学基础研究拔尖人才项目(2022JC36);兰州交通大学天佑创新团体(TY202003)
详细信息
    通讯作者:

    E-mail:edukeylab@126.com

  • 中图分类号: TN911.73

Mural inpainting progressive generative adversarial networks based on structure guided

Funds: National Natural Science Foundation of China (61963023); Ministry of Education in China Project of Humanities and Social Sciences Youth Foundation (19YJC760012); Basic Top-Notch Personnel Project of Lanzhou Jiaotong University (2022JC36); Tianyou Innovation Team of Lanzhou Jiaotong University (TY202003)
More Information
  • 摘要:

    针对破损壁画图像修复过程中存在的结构修复不当及修复后壁画细节重构丢失等问题,提出了一种基于结构引导的渐进式生成对抗壁画修复深度学习模型。设计结构生成器对壁画缺失结构内容进行修复,得到修复的壁画结构图像。通过壁画生成器生成对抗学习,结合改进的双池化SKNet多尺度特征提取模块,利用修复后的结构图像引导破损壁画实现渐进式修复,以提高壁画的细节特征学习能力。通过局部判别器和全局判别器,完成对结构图像和壁画图像的重构判别,增强壁画修复效果的全局一致性。通过对真实敦煌壁画数字化修复的实验表明:所提方法能够有效修复破损的敦煌壁画,修复后的壁画具有更好的结构及细节信息,在主客观评价指标上均优于比较方法。

     

  • 图 1  本文总体模型框架

    Figure 1.  Model framework of the proposed method

    图 2  结构生成器示意图

    Figure 2.  Schematic diagram of structure generator

    图 3  编码器和解码器结构

    Figure 3.  Structure of auto encoder and decoder

    图 4  壁画生成器结构

    Figure 4.  Structure of mural generative network

    图 5  特征提取效果示意图

    Figure 5.  Schematic diagram of feature extraction effect

    图 6  平均池化

    Figure 6.  Average pooling

    图 7  最大池化

    Figure 7.  Max pooling

    图 8  双池化示意图

    Figure 8.  Schematic diagram of double pooling

    图 9  双池化SKNet多尺度特征提取模块结构

    Figure 9.  Framework of double pooling SKNet multi-scale feature extraction module

    图 10  全局判别器和局部判别器

    Figure 10.  Global discriminator and local discriminator

    图 11  中心掩膜修复实验对比

    Figure 11.  Comparison of center mask inpainting experiments

    图 12  随机掩膜修复实验对比

    Figure 12.  Comparison of random mask inpainting experiments

    图 13  真实破损壁画修复实验对比

    Figure 13.  Comparison of inpainting experiments of real broken murals

    图 14  大幅复杂结构破损壁画修复实验对比

    Figure 14.  Comparison of inpainting experiments of large-scale and complex broken murals

    图 15  不同方法训练样本比例与PLCC关系

    Figure 15.  Relationship between proportion of different algorithm training samples and PLCC

    表  1  中心掩膜修复结果PSNR和SSIM对比

    Table  1.   Comparison of PSNR and SSIM of center mask inpainting results

    图像PSNR/dBSSIM
    文献[11] 方法文献[13] 方法本文方法文献[11] 方法文献[13] 方法本文方法
    128.9729.5531.120.84880.86700.9381
    224.2127.3633.820.81520.84920.9565
    325.7526.5728.950.82810.86510.9035
    421.5728.1933.420.83680.86900.9558
    527.0128.3931.720.88660.88500.9156
    下载: 导出CSV

    表  2  随机掩膜修复结果PSNR和SSIM对比

    Table  2.   Comparison of PSNR and SSIM of random mask inpainting results

    图像PSNR/dBSSIM
    文献[11] 方法文献[13] 方法本文方法文献[11] 方法文献[13] 方法本文方法
    117.3123.2123.450.61350.86140.8947
    221.3124.0024.100.58460.65470.6771
    321.2927.6329.460.64310.79350.8495
    423.9823.7323.890.75500.78490.8034
    523.9827.8927.950.47910.57990.5745
    下载: 导出CSV

    表  3  不同方法模型平均准确率比较

    Table  3.   Comparison of average accuracy of different algorithm models

    方法均方误差
    文献[11] 377.774
    文献[13]147.384
    本文136.939
    下载: 导出CSV

    表  4  不同方法修复平均时间对比

    Table  4.   Comparison of average repair time of different algorithms

    方法修复平均时间/s
    文献[11]8.21
    文献[13]10.62
    本文7.58
    下载: 导出CSV
  • [1] WANG H, LI Q Q, JIA S. A global and local feature weighted method for ancient murals inpainting[J]. International Journal of Machine Learning and Cybernetics, 2020, 11(6): 1197-1216. doi: 10.1007/s13042-019-01032-2
    [2] SHAO H, WANG Y X. Generative image inpainting with salient prior and relative total variation[J]. Journal of Visual Communication and Image Representation, 2021, 79: 103231. doi: 10.1016/j.jvcir.2021.103231
    [3] BRKIC A L, MITROVIC D, NOVAK A. On the image inpainting problem from the viewpoint of a nonlocal Cahn-Hilliard type equation[J]. Journal of Advanced Research, 2020, 25: 67-76. doi: 10.1016/j.jare.2020.04.015
    [4] YANG X H, GUO B L, XIAO Z L, et al. Improved structure tensor for fine-grained texture inpainting[J]. Signal Processing:Image Communication, 2019, 73: 84-95. doi: 10.1016/j.image.2018.02.006
    [5] FAN Y. Damaged region filling by improved criminisi image inpainting algorithm for thangka[J]. Cluster Computing, 2019, 22(6): 13683-13691.
    [6] LI P, CHEN W G, NG M K. Compressive total variation for image reconstruction and restoration[J]. Computers and Mathematics with Applications, 2020, 80(5): 874-893. doi: 10.1016/j.camwa.2020.05.006
    [7] BINI A A. Image restoration via DOST and total variation regularisation[J]. IET Image Processing, 2019, 13(3): 458-468. doi: 10.1049/iet-ipr.2018.5504
    [8] WAN W, HUANG H Y, LIU J. Local block operators and TV regularization based image inpainting[J]. Inverse Problems & Imaging, 2018, 12(6): 1389-1410.
    [9] 陈永, 艾亚鹏, 郭红光. 改进曲率驱动模型的敦煌壁画修复算法[J]. 计算机辅助设计与图形学学报, 2020, 32(5): 787-796.

    CHEN Y, AI Y P, GUO H G. Improved curvature-driven model of Dunhuang mural restoration algorithm[J]. Journal of Computer Aided Design and Graphics, 2020, 32(5): 787-796(in Chinese).
    [10] QIN J, BAI H H, ZHAO Y. Multi-scale attention network for image inpainting[J]. Computer Vision and Image Understanding, 2021, 204: 103155. doi: 10.1016/j.cviu.2020.103155
    [11] XIE C H, LIU S H, LI C, et al. Image inpainting with learnable bidirectional attention maps[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 8858-8867.
    [12] DU W C, CHEN H, YANG H. Learning invariant representation for unsupervised image restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 11224-11233.
    [13] DEMIR U, UNAL G. Patch-based image inpainting with generative adversarial networks[EB/OL]. (2018-03-20)[2021-08-01].https://arxiv.org/abs/1803.07422.
    [14] FANG Y C, LI Y F, TU X K, et al. Face completion with hybrid dilated convolution[J]. Signal Processing:Image Communication, 2020, 80: 115664. doi: 10.1016/j.image.2019.115664
    [15] HE X, YIN Y. Non-local and multi-scale mechanisms for image inpainting[J]. Sensors, 2021, 21(9): 3281. doi: 10.3390/s21093281
    [16] SHI Y, FAN Y, ZHANG N. A generative image inpainting network based on the attention transfer network across layer mechanism[J]. Optik, 2021, 242: 167101. doi: 10.1016/j.ijleo.2021.167101
    [17] 胡凯, 赵健, 刘昱, 等. 结构引导的图像修复[J]. 北京航空航天大学学报, 2022, 48(7): 1269-1277. doi: 10.13700/j.bh.1001-5965.2021.0004

    HU K, ZHAO J, LIU Y, et al. Structure-guided image restoration[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(7): 1269-1277(in Chinese). doi: 10.13700/j.bh.1001-5965.2021.0004
    [18] 李清泉, 王欢, 邹勤. 一种基于稀疏表示模型的壁画修复算法[J]. 武汉大学学报(信息科学版), 2018, 43(12): 1847-1853.

    LI Q Q, WANG H, ZOU Q. A mural restoration algorithm based on sparse representation model[J]. Journal of Wuhan University (Information Science Edition), 2018, 43(12): 1847-1853(in Chinese).
    [19] REN Y R, YU X M, ZHANG R N, et al. StructureFlow: Image inpainting via structure-aware appearance flow[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 181-190.
    [20] LIU H Y, JIANG B, SONG Y B, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C]//Computer Vision-ECCV 2020. Berlin: Springer, 2020, 12347: 725-741.
    [21] LI X, WANG W H, HU X L, et al . Selective kernel networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2019: 510-519.
    [22] HOU Q B, ZHANG L, CHENG M M, et al. Strip pooling: Rethinking spatial pooling for scene parsing[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 4002- 4011.
  • 加载中
图(15) / 表(4)
计量
  • 文章访问数:  428
  • HTML全文浏览量:  113
  • PDF下载量:  83
  • 被引次数: 0
出版历程
  • 收稿日期:  2021-08-04
  • 录用日期:  2021-11-05
  • 网络出版日期:  2021-11-16
  • 整期出版日期:  2023-06-30

目录

    /

    返回文章
    返回
    常见问答