[1]殷泽众,郭茂祖,田乐.基于傅里叶频域截断的神经辐射场优化[J].智能系统学报,2024,19(5):1319-1330.[doi:10.11992/tis.202401036]
YIN Zezhong,GUO Maozu,TIAN Le.Neural radiance field optimization based on Fourier frequency domain truncation[J].CAAI Transactions on Intelligent Systems,2024,19(5):1319-1330.[doi:10.11992/tis.202401036]
点击复制
YIN Zezhong,GUO Maozu,TIAN Le.Neural radiance field optimization based on Fourier frequency domain truncation[J].CAAI Transactions on Intelligent Systems,2024,19(5):1319-1330.[doi:10.11992/tis.202401036]
基于傅里叶频域截断的神经辐射场优化
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第5期
页码:
1319-1330
栏目:
吴文俊人工智能科学技术奖论坛
出版日期:
2024-09-05
- Title:
- Neural radiance field optimization based on Fourier frequency domain truncation
- Keywords:
- neural radiance field; frequency domain truncation; Fourier transform; 3D reconstruction; sparse view; fine rendering; positional encoding; scene representation
- 分类号:
- TP391
- 文献标志码:
- 2024-08-28
- 摘要:
- 神经辐射场(neural radiance fields, NeRF)作为一种通用的场景表达方法，可以更好地理解三维世界的同时创造出更加逼真的感官体验。然而在实际应用中，输入图像较少导致重建效果不佳是一个常见的问题。为此，本文提出了基于傅里叶频域截断的神经辐射场(sparse views neural radiance fields, Sv-NeRF)，通过在频域空间对输入频率进行截断并应用正则化策略来控制高频信号的输入来优化NeRF的位置编码机制，有效地降低了高频噪声，保留了关键的细节信息以提升渲染的质量和稳定性。该方法提升了模型对场景的理解能力，相较于现有方法在渲染质量、细节保留能力上均有显著提升，尤其适用于稀疏输入视角的场景重建工作。
- Abstract:
- Neural radiance field (NeRF), as a general method of scene expression, can better understand the three-dimensional (3D) world and create a more realistic sensory experience. However, in practical application, it is a common problem that less input images lead to poor reconstruction effect. Thus, a sparse-view neural radiance field (Sv-NeRF) is proposed in this study based on Fourier frequency domain truncation. The position coding mechanism of NeRF is optimized by truncating the input frequency in the frequency domain space and applying a regularization strategy to control the input of high-frequency signals. This method effectively reduces high-frequency noise and retains key details to improve the quality and stability of rendering. Compared with other methods, the proposed method improves the ability of the model to understand the scene significantly, and improves the rendering quality and detail preservation ability. It is especially suitable for scene reconstruction from the sparse input perspective.
- 参考文献/References:
-
[1] 李静, 杨宜民, 蔡述庭. 多视图的三维景物中平表面重建[J]. 智能系统学报, 2014, 9(4): 454-460.
LI Jing, YANG Yimin, CAI Shuting. 3-D scene plane reconstruction based on multiple views[J]. CAAI transactions on intelligent systems, 2014, 9(4): 454-460.
[2] MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.
[3] BARRON J T, MILDENHALL B, TANCIK M, et al. Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 5835-5844.
[4] MüLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM transactions on graphics, 2022, 41(4): 1-15.
[5] 陈国龙. 郑州双槐树遗址景观演化复原与三维建模[D]. 北京: 中国科学院大学(中国科学院空天信息创新研究院), 2022.
CHEN Guolong. Landscape evolution restoration and 3D modeling of Shuanghuaishu site in Zhengzhou[D]. Beijing: Aerospace Information Research Institute, Chinese Academy of Sciences, 2022.
[6] 罗畅. 基于GIS与BIM技术的历史建筑数字孪生管理系统研究[J]. 房地产世界, 2023(21): 82-84.
LUO Chang. Research on digital twin management system of historical buildings based on GIS and BIM technology[J]. Real estate world, 2023(21): 82-84.
[7] YU A, YE V, TANCIK M, et al. PixelNeRF: neural radiance fields from one or few images[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 4576-4585.
[8] CHEN Anpei, XU Zexiang, ZHAO Fuqiang, et al. MVSNeRF: fast generalizable radiance field reconstruction from multi-view stereo[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 14104-14113.
[9] TANCHENKO A. Visual-PSNR measure of image quality[J]. Journal of visual communication and image representation, 2014, 25(5): 874-878.
[10] HORé A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. Istanbul: IEEE, 2010: 2366-2369.
[11] KETTUNEN M, H?RK?NEN E, LEHTINEN J. E-LPIPS: robust perceptual image similarity via random transformation ensembles[EB/OL]. (2019-06-10)[2024-01-29]. https://arxiv.org/abs/1906.03973.
[12] DENG Kangle, LIU A, ZHU Junyan, et al. Depth-supervised NeRF: fewer views and faster training for free[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 12872-12881.
[13] 范腾, 杨浩, 尹稳, 等. 基于神经辐射场的多尺度视图合成研究[J]. 图学学报, 2023, 44(6): 1140-1148.
FAN Teng, YANG Hao, YIN Wen, et al. Multi-scale view synthesis based on neural radiance field[J]. Journal of graphics, 2023, 44(6): 1140-1148.
[14] 付前程. 基于神经隐式学习的多视图三维重建算法研究[D]. 武汉: 华中科技大学, 2023.
FU Qiancheng. Research on multi-view 3D reconstruction algorithm based on neural implicit learning[D]. Wuhan: Huazhong University of Science and Technology, 2023.
[15] NIEMEYER M, BARRON J T, MILDENHALL B, et al. RegNeRF: regularizing neural radiance fields for view synthesis from sparse inputs[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5470-5480.
[16] ZHANG J, YANG Gengshan, TULSIANI S, et al. NeRS: neural reflectance surfaces for sparse-view 3D reconstruction in the wild[J]. Advances in neural information processing systems., 2021(34): 29835-29847.
[17] ROESSLE B, BARRON J T, MILDENHALL B, et al. Dense depth priors for neural radiance fields from sparse input views[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 12882-12891.
[18] GOODWIN M M, AVENDANO C. Frequency-domain algorithms for audio signal enhancement based on transient modification[J]. AES: journal of the audio engineering society, 2006, 54(9): 827-840.
[19] 王玉文, 胡顺波. 数字图像处理形态学的空域与频域实现[J]. 电脑知识与技术, 2022, 18(18): 74-76.
WANG Yuwen, HU Shunbo. Spatial and frequency domain realization of digital image processing morphology[J]. Computer knowledge and technology, 2022, 18(18): 74-76.
[20] 陈树丽, 张学帅, 张鹏远, 等. 静音掩蔽和频域分段的音频指纹检索算法[J]. 声学学报, 2022, 47(4): 531-540.
CHEN Shuli, ZHANG Xueshuai, ZHANG Pengyuan, et al. Audio fingerprint retrieval method using anti-fingerprint and frequency domain segmentation[J]. Acta acustica, 2022, 47(4): 531-540.
[21] SHEN Weichao, JIA Yunde, WU Yuwei. 3D shape reconstruction from images in the frequency domain[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 4466-4474.
[22] ?IRI? D, PERI? Z, NIKOLI? J, et al. Audio signal mapping into spectrogram-based images for deep learning applications[C]//2021 20th International Symposium INFOTEH-JAHORINA. East Sarajevo: IEEE, 2021: 1-6.
[23] SHARAN R V, MOIR T J. Subband time-frequency image texture features for robust audio surveillance[J]. IEEE transactions on information forensics and security, 2015, 10(12): 2605-2615.
[24] ZHANG Ruiqi, SONG Peng, LIU Baohua, et al. Low-frequency swell noise suppression based on U-Net[J]. Applied geophysics, 2020, 17(3): 419-431.
[25] DUCHêNE S, RIANT C, CHAURASIA G, et al. Multiview intrinsic images of outdoors scenes with an application to relighting[J]. ACM transactions on graphics, 2015, 34(5): 1-16.
[26] LIANG Zexiao, TAN Guoliang, SUN Chen, et al. An effective clustering algorithm for the low-quality image of integrated circuits via high-frequency texture components extraction[J]. Electronics, 2022, 11(4): 572.
[27] SONG Liangchen, LI Zhong, GONG Xuan, et al. Harnessing low-frequency neural fields for few-shot view synthesis[EB/OL]. (2023-03-15)[2024-01-29]. https://arxiv.org/abs/2303.08370.
[28] TANCIK M, SRINIVASAN P P, MILDENHALL B, et al. Fourier features let networks learn high frequency functions in low dimensional domains[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020, 33: 7537-7547.
[29] SHENG Zehua, LIU Xiongwei, CAO Siyuan, et al. Frequency-domain deep guided image denoising[J]. IEEE transactions on multimedia, 2022, 25: 6767-6781.
[30] PREWITT J M S. Object enhancement and extraction[J]. Picture processing and psychopictorics, 1970, 10(1): 15-19.
[31] WANG Guangcong, CHEN Zhaoxi, LOY C C, et al. SparseNeRF: distilling depth ranking for few-shot novel view synthesis[C]//2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 9031-9042.
备注/Memo
收稿日期:2024-1-29。
基金项目:国家自然科学基金面上项目(62271036)；北京市自然科学基金面上项目(4232021).
作者简介:殷泽众，硕士研究生，主要研究方向为计算机视觉、智慧城市。E-mail：13717744389@163.com;郭茂祖，教授，博士生导师，博士，中国计算机学会杰出会员。主要研究方向为机器学习与人工智能、智能建造与智慧城市、生物信息学。发表学术论文 200余篇。E-mail：guomaozu@bucea.edu.cn;田乐，副教授，博士，主要研究方向为计算机网络、无线通信、大数据处理。获得2022年中国发明协会一等奖，授权发明专利3项，软件著作权2项，出版专著1部。E-mail：tianle@bucea.edu.cn。
通讯作者:郭茂祖. E-mail：guomaozu@bucea.edu.cn
更新日期/Last Update:
2024-09-05