基于美学评判的文本生成图像优化

doi:10.13700/j.bh.1001-5965.2019.0366

北京航空航天大学学报 ›› 2019, Vol. 45 ›› Issue (12): 2438-2448.doi: 10.13700/j.bh.1001-5965.2019.0366

基于美学评判的文本生成图像优化

徐天宇, 王智

清华大学计算机科学与技术系, 深圳 518055

收稿日期:2019-07-09 出版日期:2019-12-20 发布日期:2019-12-31
通讯作者: 王智 E-mail:wangzhi@sz.tsinghua.edu.cn
作者简介:徐天宇男,硕士研究生。主要研究方向:数据挖掘与深度学习;王智男,博士,副教授。主要研究方向:多媒体网络。
基金资助:
国家自然科学基金（61872215，61531006）

Text-to-image synthesis optimization based on aesthetic assessment

XU Tianyu, WANG Zhi

Department of Computer Science and Technology, Tsinghua University, Shenzhen 518055, China

Received:2019-07-09 Online:2019-12-20 Published:2019-12-31
Supported by:
National Natural Science Foundation of China (61872215,61531006)

摘要/Abstract

摘要： 在对抗生成网络（GAN）这一概念的诞生及发展推动下，文本生成图像的研究取得进展和突破，但大部分的研究内容集中于提高生成图片稳定性和解析度的问题，提高生成结果美观度的研究则很少。而计算机视觉中另一项经典的课题——图像美观度评判的研究也在深度神经网络的推动下提出了一些成果可信度较高的美观度评判模型。本文借助美观度评判模型，对实现文本生成图像目标的GAN模型进行了改造，以期提高其生成图片的美观度指标。首先针对StackGAN++模型，通过选定的美观度评判模型从美学角度评估其生成结果；然后通过借助评判模型构造美学损失的方式对其进行优化。结果使得其生成图像的总体美学分数比原模型提高了3.17%，同时Inception Score提高了2.68%，证明所提方法具有一定效果，但仍存在一定缺陷和提升空间。

关键词: 文本生成图像, 对抗生成网络(GAN), 美观度评判, StackGAN++, 美学损失

Abstract: Due to the development of generative adversarial network (GAN), much progress has been achieved in the research of text-to-image synthesis. However, most of the researches focus on improving the stability and resolution of generated images rather than aesthetic quality. On the other hand, image aesthetic assessment research is also a classic task in computer vision field, and currently there exists several state-of-the-art models on image aesthetic assessment. In this work, we propose to improve the aesthetic quality of images generated by text-to-image GAN by incorporating an image aesthetic assessment model into a conditional GAN. We choose StackGAN++, a state-of-the-art text-to-image synthesis model, assess the aesthetic quality of images generated by it with a chosen aesthetic assessment model, then define a new loss function:aesthetic loss, and use it to improve StackGAN++. Compared with the original model, the total aesthetic score of generated images is improved by 3.17% and the inception score is improved by 2.68%, indicating that the proposed optimization is effective but still has several weaknesses that can be improved in future work.

Key words: text-to-image synthesis, generative adversarial networks (GAN), aesthetic assessment, StackGAN++, aesthetic loss

中图分类号:

徐天宇, 王智. 基于美学评判的文本生成图像优化[J]. 北京航空航天大学学报, 2019, 45(12): 2438-2448.

XU Tianyu, WANG Zhi. Text-to-image synthesis optimization based on aesthetic assessment[J]. JOURNAL OF BEIJING UNIVERSITY OF AERONAUTICS AND A, 2019, 45(12): 2438-2448.

参考文献

[1] BODNAR C.Text to image synthesis using generative adversarial networks[EB/OL].(2018-05-02)[2019-07-08].https://arxiv.org/abs/1805.00676.
[2] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press,2014:2672-2680.
[3] ZHANG H,XU T,LI H,et al.Stackgan++:Realistic image synthesis with stacked generative adversarial networks[EB/OL].(2018-06-28)[2019-07-08].https://arxiv.org/abs/1710.10916.
[4] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training gans[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press,2016:2234-2242.
[5] LI Z,TANG J,MEI T.Deep collaborative embedding for social image understanding[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,41(9):2070-2083.
[6] DENG Y,LOY C C,TANG X.Image aesthetic assessment:An experimental survey[J].IEEE Signal Processing Magazine,2017,34(4):80-106.
[7] DATTA R,JOSHI D,LI J,et al.Studying aesthetics in photographic images using a computational approach[C]//European Conference on Computer Vision.Berlin:Springer,2006:288-301.
[8] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press, 2012:1097-1105.
[9] KONG S,SHEN X,LIN Z,et al.Photo aesthetics ranking network with attributes and content adaptation[C]//European Conference on Computer Vision.Berlin:Springer,2016:662-679.
[10] CHOPRA S,HADSELL R,LECUN Y.Learning a similarity metric discriminatively,with application to face verification[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2005:539-546.
[11] RADFORD A,METZ L,CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].(2016-01-07)[2019-07-08]. https://arxiv.org/abs/1511.06434.
[12] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training gans[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press,2016:2234-2242.
[13] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein gan[EB/OL].(2017-12-06)[2019-07-08].https://arxiv.org/abs/1701.07875.
[14] MIRZA M,OSINDERO S.Conditional generative adversarial nets[EB/OL].(2014-11-06)[2019-07-08].https://arxiv.org/abs/1411.1784.
[15] REED S,AKATA Z,YAN X,et al.Generative adversarial text to image synthesis[EB/OL].(2016-06-05)[2019-07-08].https://arxiv.org/abs/1605.05396.
[16] ZHANG H,XU T,LI H,et al.Stackgan:Text to photo-realistic image synthesis with stacked generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision.Piscataway,NJ:IEEE Press,2017:5907-5915.
[17] XU T,ZHANG P,HUANG Q,et al.Attngan:Fine-grained text to image generation with attentional generative adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2018:1316-1324.
[18] CHA M,GWON Y,KUNG H T.Adversarial nets with perceptual losses for text-to-image synthesis[C]//2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP).Piscataway,NJ:IEEE Press,2017:1-6.
[19] JOHNSON J,ALAHI A,LI F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Berlin:Springer,2016:694-711.
[20] SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2016:2818-2826.

基于美学评判的文本生成图像优化

Text-to-image synthesis optimization based on aesthetic assessment

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

编辑推荐

Metrics

本文评价

[1]	杨勇, 卢航远, 黄淑英, 涂伟, 李露奕. 基于自适应注入模型的遥感图像融合方法[J]. 北京航空航天大学学报, 2019, 45(12): 2351-2363.
[2]	湛颖, 高妍, 谢凌云. 中国国画艺术美感特征分析与分类[J]. 北京航空航天大学学报, 2019, 45(12): 2514-2522.
[3]	谌炳汉, 左峥嵘, 夏鲁瑞. 单平台成像目标运动特性反演方法[J]. 北京航空航天大学学报, 2018, 44(12): 2528-2535.
[4]	汪溁鹤, 陈裕雄, 马世龙, 吕江花. 实时鲁棒的频域空间目标跟踪方法[J]. 北京航空航天大学学报, 2017, 43(12): 2457-2465.
[5]	郭立新, 王小超, 郝爱民. 基于经验模态分解的点云数据平滑与增强[J]. 北京航空航天大学学报, 2017, 43(5): 1045-1052.
[6]	郭立新王小超郝爱民. 基于经验模态分解的点云数据平滑与增强[J]. 北京航空航天大学学报, 0, (): 0-0.
[7]	高明, 秦世引. 基于非线性直方图变换的对比度畸变图像校正[J]. 北京航空航天大学学报, 2016, 42(3): 514-521.
[8]	魏巍, 老松杨, 康来, 白亮. 基于三维光场的静态场景前景分割[J]. 北京航空航天大学学报, 2015, 41(7): 1330-1336.
[9]	陈锋, 刘越, 王涌天. 基于多通道投影系统的纹理表面实时绘制技术[J]. 北京航空航天大学学报, 2015, 41(3): 466-471.
[10]	张浩鹏, 姜志国. 基于姿态加权核回归的航天器姿态估计[J]. 北京航空航天大学学报, 2014, 40(4): 494-499.