[1]刘万军,佟畅,曲海成.空洞卷积与注意力融合的对抗式图像阴影去除算法[J].智能系统学报,2021,16(6):1081-1089.[doi:10.11992/tis.202011022]
LIU Wanjun,TONG Chang,QU Haicheng.An antagonistic image shadow removal algorithm based on dilated convolution and attention mechanism[J].CAAI Transactions on Intelligent Systems,2021,16(6):1081-1089.[doi:10.11992/tis.202011022]
点击复制
LIU Wanjun,TONG Chang,QU Haicheng.An antagonistic image shadow removal algorithm based on dilated convolution and attention mechanism[J].CAAI Transactions on Intelligent Systems,2021,16(6):1081-1089.[doi:10.11992/tis.202011022]
空洞卷积与注意力融合的对抗式图像阴影去除算法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
16
期数:
2021年第6期
页码:
1081-1089
栏目:
学术论文—智能系统
出版日期:
2021-11-05
- Title:
- An antagonistic image shadow removal algorithm based on dilated convolution and attention mechanism
- Keywords:
- generative adversarial networks; hole convolution; multiple attention; residual network; multi-scale; autocoder; long short-term memory; shadow removal
- 分类号:
- TP391.41
- 摘要:
- 为了解决暗区域、纹理复杂或半影区域的阴影去除效果不明显的问题，提出了空洞卷积与注意力机制融合的对抗式图像阴影去除算法。该算法基于生成对抗网络的总体思想，将空洞卷积引入残差网络中，用自定义的空洞残差块进行特征提取，扩大了特征提取的感受野。在注意力编码阶段，加入4层相同结构的空洞卷积，确保最小计算量的情况下为解码阶段提供更抽象、更本质的全局的语义特征。运用多重注意力机制，引导判别网络对无阴影图像的鉴别，提高判别网络能力。该算法分别在ISTD(image shadow triplets dataset)与SRD(shadow removal dataset)公开数据集上进行检验，SSIM(structural similarity)值达到97.77%。该算法图像特征信息保存完整，画面清晰，暗区域及地物复杂的区域阴影去除效果较好，对半影区域，也有具有良好的表现。
- Abstract:
- To solve the problem of the unobvious shadow removal effect in dark areas or complex textured and penumbra areas, an antagonistic image shadow removal algorithm is proposed based on a dilated convolution and attention mechanism. The algorithm is based on the general idea of generative adversarial networks. First, the dilated convolution is introduced into the residual network, and the user-defined hole residual block is used for feature extraction, expanding the receptive field of feature extraction. Second, in the attention coding stage, four layers of dilated convolution with the same structure are added to provide more abstract and essential global semantic features for the decoding phase with a minimum calculation amount. Finally, the multiple attention mechanism is used to guide the discrimination network to identify the unshadowed image; thus, improving the discrimination network’s ability. The proposed algorithm is tested on image shadow triplets dataset and shadow removal public datasets and achieves the structural similarity of 97.77%. The image feature information of the algorithm is well preserved, the picture is clear, the shadow removal effect is good in the dark area and complex area, and the algorithm has good performance for the penumbra area.
- 参考文献/References:
-
[1] HUANG Xiang, HUA Gang, TUMBLIN J, et al. What characterizes a shadow boundary under the sun and sky?[C]//Proceedings of the 2011 International Conference on Computer Vision. Barcelona, Spain, 2011: 898-905.
[2] FINLAYSON G D, HORDLEY S D, LU Cheng, et al. On the removal of shadows from images[J]. IEEE transactions on pattern analysis and machine intelligence, 2006, 28(1): 59-68.
[3] YU Xiaoming, LI Ge, YING Zhenqiang, et al. A new shadow removal method using color-lines[C]//International Conference on Computer Analysis of Images and Patterns. Ystad, Sweden, 2017: 307-319.
[4] 林雨准, 张保明, 郭海涛, 等. 结合多尺度分割和形态学运算的高分辨率遥感影像阴影检测[J]. 中国图象图形学报, 2018, 23(8): 1263-1272
LIN Yuzhun, ZHANG Baoming, GUO Haitao, et al. Shadow detection from high resolution remote sensing imagery based on multi-scale segmentation and morphology operation[J]. Journal of image and graphics, 2018, 23(8): 1263-1272
[5] GUO Ruiqi, DAI Qieyun, HOIEM D. Single-image shadow detection and removal using paired regions[C]//CVPR 2011. Colorado Springs, USA, 2011: 2033-2040.
[6] FAN Xinyun, WU Wenjun, ZHANG Ling, et al. Shading-aware shadow detection and removal from a single image[J]. The visual computer, 2020, 36(10/11/12): 2175-2188.
[7] 焦俊男, 石静, 田庆久, 等. 多光谱影像的NDVI阴影影响去除模型[J]. 遥感学报, 2020, 24(1): 53-66
JIAO Junnan, SHI Jing, TIAN Qingjiu, et al. Researchon multispectral-image-based NDVI shadow-effect-eliminating model[J]. Journal of remote sensing, 2020, 24(1): 53-66
[8] ZHANG Yindan, CHEN Gang, Vukomanovi J, et al. Recurrent Shadow Attention Model (RSAM) for shadow removal in high-resolution urban land-cover mapping[J]. Remote sensing of environment, 2020, 247: 111945.
[9] 吴文, 万毅. 基于低尺度细节恢复的单幅图像阴影去除方法[J]. 电子学报, 2020, 48(7): 1293-1302
WU Wen, WAN Yi. Single image shadow removal using low-scale detail recovering[J]. Acta electronica sinica, 2020, 48(7): 1293-1302
[10] 张德干, 陈晨, 董悦, 等. 一种基于机器学习的运动目标阴影检测新方法[J]. 光电子·激光, 2018, 29(12): 1317-1324
ZHANG Degan, CHEN Chen, DONG Yue, et al. A new method of moving object shadow detection based on machine learning[J]. Journal of optoelectronics·laser, 2018, 29(12): 1317-1324
[11] HOSSEINZADEH S, SHAKERI M, ZHANG H. Fast shadow detection from a single image using a patched convolutional neural network[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain, 2018: 1-5.
[12] HU Xiaowei, JIANG Yitong, FU C W, et al. Mask-ShadowGAN: learning to remove shadows from unpaired data[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South), 2019: 2472-2481.
[13] FAN Hui, HAN Meng, LI Jinjiang. Image shadow removal using end-to-end deep convolutional neural networks[J]. Applied sciences, 2019, 9(5): 1009-1026.
[14] HU Xiaowei, FU C W, ZHU Lei, et al. Direction-aware spatial context features for shadow detection and removal[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42(11): 2795-2808.
[15] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada, 2014: 2672-2680.
[16] 张瑞倩, 邵振峰, PORTNOV A, 等. 多尺度空洞卷积的无人机影像目标检测方法[J]. 武汉大学学报(信息科学版), 2020, 45(6): 895-903
ZHANG Ruiqian, SHAO Zhenfeng, PORTNOV A, et al. Multi-scale dilated convolutional neural network for object detection in UAV images[J]. Journal of Wuhan University (information science edition), 2020, 45(6): 895-903
[17] 颜铭靖, 苏喜友. 基于三维空洞卷积残差神经网络的高光谱影像分类方法[J]. 光学学报, 2020, 40(16): 1628002
YAN Mingjing, SU Xiyou. Hyperspectral image classification based on three-dimensional dilated convolutional residual neural network[J]. Acta optica sinica, 2020, 40(16): 1628002
[18] 贾宇峰, 马力. 条件约束下的自我注意生成对抗网络[J]. 西安电子科技大学学报, 2019, 46(6): 163-170
JIA Yufeng, MA Li. Self-attention generative adversarial network with the conditional constraint[J]. Journal of Xidian University, 2019, 46(6): 163-170
[19] QIAN Rui, TAN R T, YANG Wenhan, et al. Attentive generative adversarial network for raindrop removal from a single image[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 2482-2491.
[20] 刘烨, 黄金筱, 马于涛. 基于混合神经网络和注意力机制的软件缺陷自动分派方法[J]. 计算机研究与发展, 2020, 57(3): 459-473
LIU Ye, HUANG Jinxiao, MA Yutao. An automatic method using hybrid neural networks and attention mechanism for software bug triaging[J]. Journal of Computer research and development, 2020, 57(3): 459-473
[21] WANG Jifeng, LI Xiang, YANG Jian. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 1788-1797.
[22] QU Liangqiong, TIAN Jiandong, HE Shengfeng, et al. DeshadowNet: a multi-context embedding deep network for shadow removal[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA, 2017: 2308-2316.
[23] DING Bin, LONG Chengjiang, ZHANG Ling, et al. ARGAN: attentive recurrent generative adversarial network for shadow detection and removal[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South), 2019: 10212-10221.
- 相似文献/References:
-
[1]单义,杨金福,武随烁,等.基于跳跃连接金字塔模型的小目标检测[J].智能系统学报,2019,14(6):1144.[doi:10.11992/tis.201905041]
SHAN Yi,YANG Jinfu,WU Suishuo,et al.Skip feature pyramid network with a global receptive field for small object detection[J].CAAI Transactions on Intelligent Systems,2019,14():1144.[doi:10.11992/tis.201905041]
[2]朱文霖,刘华平,王博文,等.基于视-触跨模态感知的智能导盲系统[J].智能系统学报,2020,15(1):33.[doi:10.11992/tis.201908015]
ZHU Wenlin,LIU Huaping,WANG Bowen,et al.An intelligent blind guidance system based on visual-touch cross-modal perception[J].CAAI Transactions on Intelligent Systems,2020,15():33.[doi:10.11992/tis.201908015]
[3]毕晓君,潘梦迪.基于生成对抗网络的机载遥感图像超分辨率重建[J].智能系统学报,2020,15(1):74.[doi:10.11992/tis.202002002]
BI Xiaojun,PAN Mengdi.Super-resolution reconstruction of airborne remote sensing images based on the generative adversarial networks[J].CAAI Transactions on Intelligent Systems,2020,15():74.[doi:10.11992/tis.202002002]
[4]王昌安,田金文.生成对抗网络辅助学习的舰船目标精细识别[J].智能系统学报,2020,15(2):296.[doi:10.11992/tis.201901004]
WANG Changan,TIAN Jinwen.Fine-grained inshore ship recognition assisted by deep-learning generative adversarial networks[J].CAAI Transactions on Intelligent Systems,2020,15():296.[doi:10.11992/tis.201901004]
[5]张智,毕晓君.基于风格转换的无监督聚类行人重识别[J].智能系统学报,2021,16(1):48.[doi:10.11992/tis.202012014]
ZHANG Zhi,BI Xiaojun.Clustering approach based on style transfer for unsupervised person re-identification[J].CAAI Transactions on Intelligent Systems,2021,16():48.[doi:10.11992/tis.202012014]
[6]姜义,吕荣镇,刘明珠,等.基于生成对抗网络的人脸口罩图像合成[J].智能系统学报,2021,16(6):1073.[doi:10.11992/tis.202012010]
JIANG Yi,LYU Rongzhen,LIU Mingzhu,et al.Masked face image synthesis based on a generative adversarial network[J].CAAI Transactions on Intelligent Systems,2021,16():1073.[doi:10.11992/tis.202012010]
[7]吴晓东,熊伟丽.采用编码输入的生成对抗网络故障检测方法及应用[J].智能系统学报,2022,17(3):496.[doi:10.11992/tis.202102003]
WU Xiaodong,XIONG Weili.Fault detection method and its application using GAN with an encoded input[J].CAAI Transactions on Intelligent Systems,2022,17():496.[doi:10.11992/tis.202102003]
[8]张冀,曹艺,王亚茹,等.融合VAE和StackGAN的零样本图像分类方法[J].智能系统学报,2022,17(3):593.[doi:10.11992/tis.202107012]
ZHANG Ji,CAO Yi,WANG Yaru,et al.Zero-shot image classification method combining VAE and StackGAN[J].CAAI Transactions on Intelligent Systems,2022,17():593.[doi:10.11992/tis.202107012]
[9]严浙平,曲思瑜,邢文.水下图像增强方法研究综述[J].智能系统学报,2022,17(5):860.[doi:10.11992/tis.202108022]
YAN Zheping,QU Siyu,XING Wen.An overview of underwater image enhancement methods[J].CAAI Transactions on Intelligent Systems,2022,17():860.[doi:10.11992/tis.202108022]
[10]翟永杰,张智柏,王亚茹.基于改进TransGAN的零样本图像识别方法[J].智能系统学报,2023,18(2):352.[doi:10.11992/tis.202111002]
ZHAI Yongjie,ZHANG Zhibai,WANG Yaru.An image recognition method of zero -shot learning based on an improved TransGAN[J].CAAI Transactions on Intelligent Systems,2023,18():352.[doi:10.11992/tis.202111002]
备注/Memo
收稿日期:2020-11-20。
基金项目:国家自然科学基金项目(41701479)；辽宁省教育厅科学研究经费项目(LJ2019JL010)；辽宁省自然科学基金面上项目(20180550529)
作者简介:刘万军，教授，博士生导师，主要研究方向为图像与智能信息处理。获国家发明专利5项，出版专著教材6部，发表学术论文130余篇;佟畅，硕士研究生，主要研究方向为图像与智能信息处理;曲海成，副教授，博士，主要研究方向为图像与智能信息处理
通讯作者:佟畅.E-mail:1879031567@qq.com
更新日期/Last Update:
2021-12-25