[1]PAN Liwen,WANG Jun,SHI Jun,et al.Two-stage adverse weather image restoration based on a conditional diffusion model[J].CAAI Transactions on Intelligent Systems,2026,21(4):943-951.[doi:10.11992/tis.202509039]
Copy

Two-stage adverse weather image restoration based on a conditional diffusion model

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 21 Number of periods: 2026 4 Page number: 943-951 Column: 学术论文—机器感知与模式识别 Public date: 2026-07-05
Title:
Two-stage adverse weather image restoration based on a conditional diffusion model
Author(s):
PAN Liwen1;  WANG Jun1;  SHI Jun1;  LI Juncheng2
1. School of Communications and Information Engineering, Shanghai University, Shanghai 200444, China;
2. School of Computer Science and Technology, East China Normal University, Shanghai 200300, China
Keywords:
deep learning; adverse weather image restoration; image restoration; diffusion model; wavelet transform; feature fusion; multi-scale features; encoder-decoder structure
CLC:
TP391
DOI:
10.11992/tis.202509039
Abstract:
To address common challenges in image restoration under inclement weather, such as blurred details, texture loss, and high video memory usage, this study proposes a two-stage image restoration method based on a conditional diffusion model. The first stage constructs an encoder–decoder architecture utilizing a dual-branch feature fusion module. This architecture leverages the local perception capabilities of convolutional operations and the global dependency modeling advantages of the transformer architecture to achieve preliminary reconstruction of degraded images. This stage also incorporates a wavelet transform to effectively extract multiscale frequency features, preserving structural information integrity while considerably reducing computational complexity. The second stage embeds a conditional diffusion model between the encoder and decoder networks, using the output of the previous stage as a structural condition. Texture and detail are enhanced through an iterative denoising mechanism via a diffusion process. This module guides the degraded image to steadily approximate the true image distribution through probabilistic mapping in the latent space, ultimately achieving restoration results with considerably improved visual quality. Experimental results demonstrate that our method outperforms several mainstream image restoration methods in rain and snow removal tasks, improving the peak signal-to-noise ratio by 0.06~1.13 and 0.23~5.72 dB, respectively, across various datasets. The proposed method also achieves a good balance between model parameter count and graphics memory usage. In short, this research provides a novel, effective, and reliable solution for image restoration in inclement weather.
References:
[1] ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6230-6239.
[2] 谭睿俊, 赵志诚, 谢新林. 双层残差语义分割网络及交通场景应用[J]. 智能系统学报, 2022, 17(4): 780-787 TAN Ruijun, ZHAO Zhicheng, XIE Xinlin. Double-residual semantic segmentation network and traffic scenic application[J]. CAAI transactions on intelligent systems, 2022, 17(4): 780-787
[3] 陆军, 鲁林超, 翟晓阳, 等. 面向道路交通场景的高效3D目标检测[J]. 智能系统学报, 2025, 20(1): 91-100 LU Jun, LU Linchao, ZHAI Xiaoyang, et al. High-efficiency 3D object detection for road traffic scenes[J]. CAAI transactions on intelligent systems, 2025, 20(1): 91-100
[4] 贾亮, 戚丽瑾, 林铭文, 等. 改进的YOLOv7轻量化目标检测模型[J]. 计算机工程与设计, 2025, 46(8): 2350-2357 JIA Liang, QI Lijin, LIN Mingwen, et al. Improved YOLOv7 lightweight target detection model[J]. Computer engineering and design, 2025, 46(8): 2350-2357
[5] HU Yihan, YANG Jiazhi, CHEN Li, et al. Planning-oriented autonomous driving[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 17853-17862.
[6] 吕家威, 吕丹璇. 多模态交互技术在自动驾驶舱环境中的应用[J]. 汽车电器, 2025(9): 13-15 L? Jiawei, L? Danxuan. Application of multimodal interaction technology in automatic cockpit environment[J]. Auto electric parts, 2025(9): 13-15
[7] 王美华, 柯凡晖, 梁云, 等. 融合3D注意力和Transformer的图像去雨网络[J]. 中国图象图形学报, 2022, 27(5): 1509-1521 WANG Meihua, KE Fanhui, LIANG Yun, et al. 3D attention and Transformer based single image deraining network[J]. Journal of image and graphics, 2022, 27(5): 1509-1521
[8] JIANG Kui, WANG Zhongyuan, YI Peng, et al. Multi-scale progressive fusion network for single image deraining[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 8343-8352.
[9] LIANG Jingyun, CAO Jiezhang, SUN Guolei, et al. SwinIR: image restoration using swin transformer[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021: 1833-1844.
[10] ZAMIR S W, ARORA A, KHAN S, et al. Restormer: efficient transformer for high-resolution image restoration[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5718-5729.
[11] WANG Zhendong, CUN Xiaodong, BAO Jianmin, et al. Uformer: a general U-shaped transformer for image restoration[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17662-17672.
[12] HE Kaiming, SUN Jian, TANG Xiaoou. Single image haze removal using dark channel prior[J]. IEEE transactions on pattern analysis and machine intelligence, 2011, 33(12): 2341-2353
[13] 王平, 张云峰, 包芳勋, 等. 基于雾天图像降质模型的优化去雾方法[J]. 中国图象图形学报, 2018, 23(4): 605-616 WANG Ping, ZHANG Yunfeng, BAO Fangxun, et al. Multivariate optimized haze removal method based on hazy image degradation model[J]. Journal of image and graphics, 2018, 23(4): 605-616
[14] ZHU Lei, FU C W, LISCHINSKI D, et al. Joint bi-layer optimization for single-image rain streak removal[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2545-2553.
[15] YOU Jing, JIA Shaocheng, PEI Xin, et al. DMRVisNet: deep multihead regression network for pixel-wise visibility estimation under foggy weather[J]. IEEE transactions on intelligent transportation systems, 2022, 23(11): 22354-22366
[16] DONG Chao, LOY C C, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]//Computer Vision-ECCV 2014. Cham: Springer, 2014: 184-199.
[17] ZHANG Kai, ZUO Wangmeng, CHEN Yunjin, et al. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising[J]. IEEE transactions on image processing, 2017, 26(7): 3142-3155
[18] VALANARASU J M J, YASARLA R, PATEL V M. Transweather: Transformer based restoration of images degraded by adverse weather conditions[C]//2022 IEEE Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 2353-2363.
[19] LIU Pengju, ZHANG Hongzhi, ZHANG Kai, et al. Multi-level wavelet-CNN for image restoration[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Boston: IEEE, 2018: 886-88609.
[20] 王维, 王习博, 廖万正, 等. 基于多尺度小波变换的雨天列车外观图像恢复[J/OL]. 北京: 自动化技术与应用, 2025. (2025-08-21)[2025-09-21]. https://link.cnki.net/urlid/23.1474.tp.20250821.1548.024. WANG Wei, WANG Xibo, LIAO Wanzheng, et al. R-ainy train appearance image restoration based on mul-tiscale wavelet transform[J/OL]. Automation technology and application, 2025. (2025-08-21)[2025-09-21]. https://link.cnki.net/urlid/23.1474.tp.20250821.1548.024.
[21] ?ZDENIZCI O, LEGENSTEIN R. Restoring vision in adverse weather conditions with patch-based denoising diffusion models[J]. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(8): 10346-10357
[22] HUANG Yi, HUANG Jiancheng, LIU Jianzhuang, et al. WaveDM: wavelet-based diffusion models for image restoration[J]. IEEE transactions on multimedia, 2024, 26: 7058-7073
[23] 管凤旭, 张涵宇, 路斯棋, 等. 扩散模型在计算机视觉领域的研究现状[J]. 智能系统学报, 2025, 20(2): 265-282 GUAN Fengxu, ZHANG Hanyu, LU Siqi, et al. Research status of diffusion models in computer vision[J]. CAAI transactions on intelligent systems, 2025, 20(2): 265-282
[24] QIAN Rui, TAN R T, YANG Wenhan, et al. Attentive generative adversarial network for raindrop removal from a single image[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2482-2491.
[25] LIU Yunfu, JAW D W, HUANG S C, et al. DesnowNet: context-aware deep network for snow removal[J]. IEEE transactions on image processing, 2018, 27(6): 3064-3073
[26] HOR? A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. Istanbul: IEEE, 2010: 2366-2369.
[27] LIU Xing, SUGANUMA M, SUN Zhun, et al. Dual residual networks leveraging the potential of paired operations for image restoration[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 7000-7009.
[28] QUAN Ruijie, YU Xin, LIANG Yuanzhi, et al. Removing raindrops and rain streaks in one go[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 9143-9152.
[29] QUAN Yuhui, DENG Shijie, CHEN Yixin, et al. Deep learning for seeing through window with raindrops[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2020: 2463-2471.
[30] XIAO Jie, FU Xueyang, LIU Aiping, et al. Image de-raining transformer[J]. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(11): 12978-12995
[31] GUO Hang, LI Jinmin, DAI Tao, et al. MambaIR: a simple baseline forImage restoration withState-space model[C]//Computer Vision-ECCV 2024. Cham: Springer, 2025: 222-241.
[32] CHEN Weiting, FANG Haoyu, DING J J, et al. JSTASR: joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal[C]//Computer Vision-ECCV 2020. Cham: Springer, 2020: 754-770.
[33] ZHANG Kaihao, LI Rongqing, YU Yanjiang, et al. Deep dense multi-scale network for snow removal using semantic and depth priors[J]. IEEE transactions on image processing, 2021, 30: 7419-7431
[34] CHU Xiaojie, CHEN Liangyu, YU Wenqing. NAFSSR: stereo image super-resolution using NAFNet[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New Orleans: IEEE, 2022: 1238-1247.
[35] LI Ruoteng, TAN R T, CHEONG L F. All in one bad weather removal using architectural search[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3172-3182.
[36] CHEN Weiting, HUANG Zhikai, TSAI C C, et al. Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: toward a unified model[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17632-17641.
[37] ZHU Yurui, WANG Tianyu, FU Xueyang, et al. Learning weather-general and weather-specific features for image restoration under multiple adverse weather conditions[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 21747-21758.
[38] Wu Y, Kirillov A, Massa F, e tal. Detectron2[J/OL]. Facebook AI research, 2019. (2019-10-28)[2025-09-25]. https://github.com/facebookresearch/detectron2.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems