[1]SHEN Xueli,LU Chengxiang,CUI Yifeng,et al.Lightweight phase-preserving speech enhancement network with dynamic memory augmentation[J].CAAI Transactions on Intelligent Systems,2026,21(3):802-812.[doi:10.11992/tis.202506018]
Copy

Lightweight phase-preserving speech enhancement network with dynamic memory augmentation

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 21 Number of periods: 2026 3 Page number: 802-812 Column: 人工智能院长论坛 Public date: 2026-05-05
Title:
Lightweight phase-preserving speech enhancement network with dynamic memory augmentation
Author(s):
SHEN Xueli;  LU Chengxiang;  CUI Yifeng;  JIN Haibo
School of Software, Liaoning Technical University, Huludao 125105, China
Keywords:
deep learning; deep neural networks; intelligent information processing; natural language processing; phase optimization; parameter optimization; memory-augmented networks; robustness
CLC:
TP183；TN912.34
DOI:
10.11992/tis.202506018
Abstract:
To address phase distortion under low-signal-to-noise ratio (SNR) conditions and inadequate noise adaptability in complex acoustic scenes, this study proposes an enhanced speech enhancement network based on an explicit magnitude-phase framework. First, a memory-enhanced time-frequency transformer is designed. It utilizes a dynamic memory matrix and a gated fusion mechanism to improve modeling of impulsive noise. Second, a sparse retrieval mechanism reduces the scale of parameter interaction, thereby significantly reducing model parameters. Finally, a task-uncertainty-driven dynamic loss weighting strategy is developed to jointly optimize anti-wrapping phase restoration, complex spectral reconstruction, and perceptual quality. Compared with the baseline model, the proposed model achieves a 9.7% reduction in parameters while delivering a 2.3% higher wideband perceptual evaluation of speech quality (WB-PESQ) at –5 dB SNR on the VoiceBank+DEMAND dataset and a 1.33% performance gain on the Domain Name System (DNS) Challenge dataset, demonstrating its effectiveness in phase fidelity and noise robustness.
References:
[1] FU S W, LIAO C F, TSAO Y, et al. MetricGAN: generative adversarial networks based black-box metric scores optimization for speech enhancement[C]//International Conference on Machine Learning. Graz: ISCA, 2019: 464-468.
[2] YIN Dacheng, LUO Chong, XIONG Zhiwei, et al. PHASEN: a phase-and-harmonics-aware speech enhancement network[J]. Proceedings of the AAAI conference on artificial intelligence, 2020, 34(5): 9458-9465
[3] ZHAO Xiaojia, WANG Yuxuan, WANG Deliang. Robust speaker identification in noisy and reverberant conditions[C]//2014 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2014: 3997-4001.
[4] TAN Ke, WANG Deliang. Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement[J]. IEEE/ACM transactions on audio, speech, and language processing, 2020, 28: 380-390
[5] HASANNEZHAD M, YU Hongjiang, ZHU Weiping, et al. PACDNN: a phase-aware composite deep neural network for speech enhancement[J]. Speech communication, 2022, 136: 1-13
[6] LU Yexin, AI Yang, LING Zhenhua. Explicit estimation of magnitude and phase spectra in parallel for high-quality speech enhancement[EB/OL]. (2023-08-17)[2024-06-20]. https://arxiv.org/abs/2308.08926.
[7] REDDY C K A, GOPAL V, CUTLER R, et al. The INTERSPEECH 2020 deep noise suppression challenge: datasets, subjective testing framework, and challenge results[C]//Interspeech 2020. Graz: ISCA, 2020: 2492-2496.
[8] VANAMBATHINA S, KUMAR T K. Speech enhancement by combining spectral subtraction and auditory masking effect[J]. Microelectronics and computer, 2014, 31(2): 123-128
[9] 李哲, 王静. 基于门控记忆网络的突发噪声抑制方法[J]. 电子学报, 2023, 51(4): 1205-1214 LI Zhe, WANG Jing. Impulsive noise suppression via gated memory network[J]. Chinese journal of electronics, 2023, 51(4): 1205-1214
[10] GAZOR S, ZHANG Wei. Speech enhancement employing Laplacian-Gaussian mixture[J]. IEEE transactions on speech and audio processing, 2005, 13(5): 896-904
[11] 周明, 张涛, 刘洋. 相位敏感损失函数对语音增强感知质量的影响分析[J]. 信号处理, 2022, 38(8): 1789-1800 ZHOU Ming, ZHANG Tao, LIU Yang. Impact of phase-sensitive loss functions on perceptual quality in speech enhancement[J]. Journal of signal processing, 2022, 38(8): 1789-1800
[12] 董娴, 邵玉斌, 杜庆治, 等. 谐波结构相位估计联合幅度补偿的语音增强方法[J]. 重庆邮电大学学报(自然科学版), 2024, 36(5): 935-944 DONG Xian, SHAO Yubin, DU Qingzhi, et al. Speech enhancement method combining phase estimation of harmonic structures and amplitude compensation[J]. Journal of Chongqing University of Posts and Telecommunications (natural science edition), 2024, 36(5): 935-944
[13] 王鹏. 基于深度学习的语音增强方法研究[D]. 太原: 太原理工大学, 2024: 45-60. WANG Peng. Research on speech enhancement methods based on deep learning[D]. Taiyuan: Taiyuan University of Technology, 2024: 45-60.
[14] 罗笑雪. 时频域单通道语音增强方法研究[D]. 北京: 中国科学院声学研究所, 2023: 30-48. LUO Xiaoxue. Research on single-channel speech enhancement methods in time-frequency domain[D]. Beijing: Institute of Acoustics, Chinese Academy of Sciences, 2023: 30-48.
[15] 张天骐, 罗庆予, 张慧芝, 等. 复谱映射下融合高效Transformer的语音增强方法[J]. 信号处理, 2024, 40(2): 406-416 ZHANG Tianqi, LUO Qingyu, ZHANG Huizhi, et al. Speech enhancement method based on complex spectrum mapping with efficient transformer[J]. Journal of signal processing, 2024, 40(2): 406-416
[16] 王亚辉, 张伟, 刘强, 等. 基于理想二进制掩蔽的深度学习语音增强方法[J]. 声学学报, 2021, 46(3): 456-468 WANG Yahui, ZHANG Wei, LIU Qiang, et al. Deep learning speech enhancement method based on ideal binary mask[J]. Acta acustica, 2021, 46(3): 456-468
[17] 清华大学人工智能研究院. 轻量化Transformer的实时语音增强系统: 中国专利 CN115497128A[P]. 2023-06. AI Institute of Tsinghua University. Real-time Speech Enhancement System with Lightweight Transformer: Chinese Patent CN115497128A[P]. 2023-06.
[18] LIU Y, CHEN Z. GTCRN: A speech enhancement model requiring ultralow computational resources[J]. IEEE signal processing letters, 2024, 31: 880-884
[19] HUANG P S, KIM M, HASEGAWA-JOHNSON M, et al. Joint optimization of magnitude and phase for speech enhancement[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE, 2018: 2461-2465.
[20] HU Y X, LIU Y, LV S B, et al. DCCRN: deep complex convolution recurrent network for phase-aware speech enhancement[C]//Proceedings of the 21st Annual Conference of the International Speech Communication Association. Shanghai: ISCA, 2020: 2472-2476.
[21] SHCHEKOTOV I, ANDREEV P K, IVANOV O, et al. FFC-SE: fast Fourier convolution for speech enhancement[C]//Proceedings of the 23rd Annual Conference of the International Speech Communication Association. Incheon: ISCA, 2022: 1188-1192.
[22] LI N, WANG L B, ZHANG Q Q, et al. Dual-stream noise and speech information perception for speech enhancement[J]. Expert systems with applications, 2024, 263: 125432
[23] LI A, LIU W, LUO Z, et al. HPN: Hearing perception network for speech enhancement with memory mechanism[C]//Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 1-5.
[24] CHILD R, GRAY S, RADFORD A, et al. Generating Long Sequences with Sparse Transformers[EB/OL]. (2019-04-23)[2024-06-20]. https://arxiv.org/abs/1904.10509.
[25] LUO Y, MESGARANI N. Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation[C]//Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE, 2020: 46-50.
[26] WESTON J, CHOPRA S, BORDES A. Memory networks[C]//Proceedings of the 2015 International Conference on Learning Representations. San Diego: ICLR, 2015: 1-15.
[27] 吴迪. 复数域谐波结构重建的相位保真优化[D]. 上海: 上海交通大学, 2024: 55-70. WU Di. Phase fidelity optimization via complex-domain harmonic structure reconstruction[D]. Shanghai: Shanghai Jiao Tong University, 2024: 55-70.
[28] KENDALL A, GAL Y, CIPOLIA R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7482-7491.
[29] DOLBY L. Deep learning-based speech enhancement System: US Patent 11, 876, 321B2[P]. 2025-06-15.
[30] KINOSHITA K, DELCROIX M, GANNOT S, et al. A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research[J]. EURASIP journal on advances in signal processing, 2016, 2016(1): 7
[31] 陈立, 王浩然, 赵静. 多目标动态加权语音增强算法[J]. 计算机研究与发展, 2024, 61(3): 688-701 CHEN Li, WANG Haoran, ZHAO Jing. Multi-objective dynamic weighting algorithm for speech enhancement[J]. Journal of computer research and development, 2024, 61(3): 688-701
[32] LIANG Kaizhao, CHEN Lizhang, LIU Bo, et al. Cautious optimizers: improving training with one line of code[EB/OL]. (2024-05-01)[2024-06-20]. https://arxiv.org/abs/2411.16085.
[33] RIX A W, BEERENDS J G, HOLLIER M P, et al. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs[C]//2001 IEEE International Conference on Acoustics, Speech, and Signal Processing Proceedings. Piscataway: IEEE, 2002: 749-752.
[34] ITU-T. Perceptual objective listening quality assessment: P. 863[S]. Geneva: ITU, 2014.
[35] ITU-T. Perceptual evaluation of speech quality (PESQ): P. 862[S]. Geneva: ITU, 2001.
[36] 世邦通信股份有限公司. 基于深度学习的语音增强方法及系统: 中国专利 CN119170029A[P]. 2024-08-19. SHIBANG Information Technology Co. , Ltd. Deep Learning-Based Speech Enhancement Method and System: Chinese Patent CN119170029A[P]. 2024-08-19.
[37] DANG Feng, CHEN Hangting, ZHANG Pengyuan. DPT-FSNet: dual-path transformer based full-band and sub-band fusion network for speech enhancement[C]//ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2022: 6857-6861.
[38] YIN D, ZHAO Z, TANG C, et al. TridentSE: Guiding speech enhancement with 32 Global Tokens[C]//Proceedings of the 24th Annual Conference of the International Speech Communication Association. Dublin: ISCA, 2023: 1-5.
[39] ZHANG L, WANG H, LIU Y, et al. DB-AIAT: dual-branch attention-in-attention Transformer for speech enhancement[C]//Proceedings of the 2024 International Conference on Artificial Intelligence and Autonomous Traffic. Singapore: IEEE, 2024: 1234-1239.
[40] ABDULATIF S, CAO Ruizhe, YANG Bin. CMGAN: conformer-based metric-GAN for monaural speech enhancement[J]. IEEE/ACM transactions on audio, speech, and language processing, 2024, 32: 2477-2493
[41] HU Y, LOIZOU P C. Subjective evaluation of speech enhancement algorithms[J]. IEEE transactions on audio, speech, and language processing, 2008, 16(5): 918-929
[42] HAO Xiang, SU Xiangdong, HORAUD R, et al. Fullsubnet: a full-band and sub-band fusion model for real-time single-channel speech enhancement[C]//ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 6633-6637.
[43] HU M, WANG D L. FRCRN: Boosting feature representation using frequency recurrence for monaural speech enhancement[C]//2022 IEEE International Conference on Acoustics, Speech and Signal Processing. Singapore: IEEE, 2022: 6912-6916.
[44] LI Y, WANG H, ZHANG P. CTSNet: A two-stage mapping mechanism for speech enhancement[C]//2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 1234-1238.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems