留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

多尺度特征和注意力融合的生成对抗壁画修复

陈永 陈锦 陶美风

陈永,陈锦,陶美风. 多尺度特征和注意力融合的生成对抗壁画修复[J]. 北京航空航天大学学报,2023,49(2):254-264 doi: 10.13700/j.bh.1001-5965.2021.0242
引用本文: 陈永,陈锦,陶美风. 多尺度特征和注意力融合的生成对抗壁画修复[J]. 北京航空航天大学学报,2023,49(2):254-264 doi: 10.13700/j.bh.1001-5965.2021.0242
CHEN Y,CHEN J,TAO M F. Mural inpainting with generative adversarial networks based on multi-scale feature and attention fusion[J]. Journal of Beijing University of Aeronautics and Astronautics,2023,49(2):254-264 (in Chinese) doi: 10.13700/j.bh.1001-5965.2021.0242
Citation: CHEN Y,CHEN J,TAO M F. Mural inpainting with generative adversarial networks based on multi-scale feature and attention fusion[J]. Journal of Beijing University of Aeronautics and Astronautics,2023,49(2):254-264 (in Chinese) doi: 10.13700/j.bh.1001-5965.2021.0242

多尺度特征和注意力融合的生成对抗壁画修复

doi: 10.13700/j.bh.1001-5965.2021.0242
基金项目: 国家自然科学基金(61963023);教育部人文社会科学研究青年基金(19YJC760012);兰州交通大学天佑创新团队(TY202003)
详细信息
    通讯作者:

    E-mail:edukeylab@126.com

  • 中图分类号: TN911.73

Mural inpainting with generative adversarial networks based on multi-scale feature and attention fusion

Funds: National Natural Science Foundation of China (61963023); Ministry of Education in China Project of Humanities and Social Sciences (19YJC760012); Tianyou Innovation Team of Lanzhou Jiaotong University (TY202003)
More Information
  • 摘要:

    针对现有深度学习图像修复算法修复壁画时,存在特征提取不足及细节重构丢失等问题,提出了一种多尺度特征和注意力融合的生成对抗壁画修复深度学习模型。设计多尺度特征金字塔网络提取壁画中不同尺度的特征信息,增强特征关联性;采用自注意力机制及特征融合模块构建多尺度特征生成器,以获取丰富的上下文信息,提升网络的修复能力;引入最小化对抗损失与均方误差促进判别器的残差反馈,从而结合不同尺度的特征信息完成壁画修复。通过对真实敦煌壁画数字化修复的实验结果表明,所提算法能够有效保护壁画图像的边缘和纹理等重要特征信息,并且主观视觉效果及客观评价指标均优于比较算法。

     

  • 图 1  生成对抗网络基本结构框架[18]

    Figure 1.  Basic structural framework of GAN[18]

    图 2  本文总体模型框架

    Figure 2.  Overall framework of the proposed model

    图 3  多尺度特征融合的生成器结构

    Figure 3.  Structure of generator based on multi-scale feature fusion

    图 4  特征融合过程示意图

    Figure 4.  Schematic diagram of feature fusion process

    图 5  融合结构原理

    Figure 5.  Schematic of fusion structure

    图 6  编码器与解码器示意图

    Figure 6.  Diagram of encoder and decoder

    图 7  自注意力模型结构

    Figure 7.  Structure of self-attention model

    图 8  不同算法对人为添加中心掩膜破损壁画的修复结果对比

    Figure 8.  Comparison of different algorithms in inpainting of murals with artificially added central damage

    图 9  不同算法对人为添加随机掩膜破损壁画的修复结果对比

    Figure 9.  Comparison of different algorithms in inpainting of murals with artificially added random damage

    图 10  不同算法人为添加破损修复结果PSNR和SSIM对比

    Figure 10.  Comparison of PSNR and SSIM repair results of different algorithms with artificially added random damage

    图 11  不同算法对真实破损壁画的修复效果对比

    Figure 11.  Comparison of different algorithms in inpainting of murals with real damage

    表  1  不同算法修复结果PSNR和SSIM对比

    Table  1.   Comparison of PSNR and SSIM repair results of different algorithms

    原始壁画
    图像
    Criminisi算法[5]文献[14]算法文献[17]算法 本文算法
    PSNR/dBSSIMPSNR/dBSSIMPSNR/dBSSIMPSNR/dBSSIM
    121.460.811326.740.888827.330.8229 34.170.9153
    222.590.803531.160.889928.550.836633.960.9048
    318.940.780628.410.868520.510.783429.420.8737
    424.380.844831.160.893527.950.872132.780.9212
    516.410.682621.190.816318.490.758124.780.8593
    620.570.758923.400.789422.060.782327.190.9199
    719.290.740819.520.771820.940.758726.930.9155
    下载: 导出CSV
  • [1] WANG H, LI Q Q, JIA S. A global and local feature weighted method for ancient murals inpainting[J]. International Journal of Machine Learning and Cybernetics, 2020, 11(6): 1197-1216. doi: 10.1007/s13042-019-01032-2
    [2] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2000: 417-424.
    [3] CHAN T F, SHEN J H. Nontexture inpainting by curvature-driven diffusions[J]. Journal of Visual Communication and Image Representation, 2001, 12(4): 436-449. doi: 10.1006/jvci.2001.0487
    [4] SHEN J H, CHAN T F. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019-1043. doi: 10.1137/S0036139900368844
    [5] CRIMINISI A, PEREZ P, TOYAMA K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212. doi: 10.1109/TIP.2004.833105
    [6] LI P, CHEN W G, MICHAEL K N. Compressive total variation for image reconstruction and restoration[J]. Computers and Mathematics with Applications, 2020, 80(5): 874-893. doi: 10.1016/j.camwa.2020.05.006
    [7] FAN Y. Damaged region filling by improved criminisi image inpainting algorithm for thangka[J]. Cluster Computing, 2019, 22(6): 13683-13691.
    [8] 陈永, 艾亚鹏, 郭红光. 改进曲率驱动模型的敦煌壁画修复算法[J]. 计算机辅助设计与图形学学报, 2020, 32(5): 787-796.

    CHEN Y, AI Y P, GUO H G. Improved curvature-driven model of Dunhuang mural restoration algorithm[J]. Journal of Computer-Aided Design and Graphics, 2020, 32(5): 787-796(in Chinese).
    [9] YANG X H, GUO B L, XIAO Z L, et al. Improved structure tensor for fine-grained texture inpainting[J]. Signal Processing:Image Communication, 2019, 73: 84-95. doi: 10.1016/j.image.2018.02.006
    [10] XU S X, LIU D, XIONG Z W. E2I: Generative inpainting from edge to image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(4): 1308-1322. doi: 10.1109/TCSVT.2020.3001267
    [11] QIN J, BAI H H, ZHAO Y. Multi-scale attention network for image inpainting[J]. Computer Vision and Image Understanding, 2021, 204: 103155. doi: 10.1016/j.cviu.2020.103155
    [12] ZENG Y H, FU J L, CHAO H Y, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2019: 1486-1494.
    [13] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transaction on Graphics, 2017, 36(4): 1-14.
    [14] YAN Z Y, LI X M, LI M, et al. Shift-Net: Image inpainting via deep feature rearrangement[EB/OL]. (2018-04-13) [2021-05-01]. https://arxiv.org/abs/1801.09392v2.
    [15] ZENG Y, GONG Y, ZENG X. Controllable digital restoration of ancient paintings using convolutional neural network and nearest neighbor[J]. Pattern Recognition Letters, 2020, 133: 158-164. doi: 10.1016/j.patrec.2020.02.033
    [16] 曹建芳, 李艳飞, 崔红艳, 等. 自适应样本块局部搜索算法对古代寺观壁画的修复[J]. 计算机辅助设计与图形学学报, 2019, 31(11): 2030-2037.

    CAO J F, LI Y F, CUI H Y, et al. Restoration of ancient temple and temple mural paintings by adaptive local search algorithm of Sample blocks[J]. Journal of Computer-Aided Design and Graphics, 2019, 31(11): 2030-2037(in Chinese).
    [17] LIU H Y, JIANG B, SONG Y B, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C]// Proceedings of the 16th European Conference on Computer Vision. Berlin: Springer, 2020, 12347: 725-741.
    [18] 胡凯, 赵健, 刘昱, 等. 结构引导的图像修复[J]. 北京航空航天大学学报, 2022, 48(7): 1269-1277.

    HU K, ZHAO J, LIU Y, et al. Structure-guided image restoration[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(7): 1269-1277(in Chinese).
    [19] GADIPUDI A, DEIVALAKSHMI S, SEOK-BUM K. Deep dilated and densely connected parallel convolutional groups for compression artifacts reduction[J]. Digital Signal Processing, 2020, 106: 102804. doi: 10.1016/j.dsp.2020.102804
    [20] GUO X P, MENG L Y, MEI L Y, et al. Multi-focus image fusion with Siamese self-attention network[J]. IET Image Processing, 2020, 14(7): 1339-1346. doi: 10.1049/iet-ipr.2019.0883
  • 加载中
图(11) / 表(1)
计量
  • 文章访问数:  597
  • HTML全文浏览量:  145
  • PDF下载量:  105
  • 被引次数: 0
出版历程
  • 收稿日期:  2021-05-08
  • 录用日期:  2021-06-25
  • 网络出版日期:  2021-07-14
  • 整期出版日期:  2023-02-28

目录

    /

    返回文章
    返回
    常见问答