[1]YIN Zezhong,GUO Maozu,TIAN Le.Neural radiance field optimization based on Fourier frequency domain truncation[J].CAAI Transactions on Intelligent Systems,2024,19(5):1319-1330.[doi:10.11992/tis.202401036]
Copy

Neural radiance field optimization based on Fourier frequency domain truncation

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 5 Page number: 1319-1330 Column: 吴文俊人工智能科学技术奖论坛 Public date: 2024-09-05
Title:
Neural radiance field optimization based on Fourier frequency domain truncation
Author(s):
YIN Zezhong1; 2;  GUO Maozu1; 2;  TIAN Le1; 2
1. School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing 100044, China;
2. Research on Intelligent Processing Method of Building Big Data Beijing Key Laboratory, Beijing University of Civil Engineering and Architecture, Beijing 100044, China
Keywords:
neural radiance field; frequency domain truncation; Fourier transform; 3D reconstruction; sparse view; fine rendering; positional encoding; scene representation
CLC:
TP391
DOI:
10.11992/tis.202401036
Abstract:
Neural radiance field (NeRF), as a general method of scene expression, can better understand the three-dimensional (3D) world and create a more realistic sensory experience. However, in practical application, it is a common problem that less input images lead to poor reconstruction effect. Thus, a sparse-view neural radiance field (Sv-NeRF) is proposed in this study based on Fourier frequency domain truncation. The position coding mechanism of NeRF is optimized by truncating the input frequency in the frequency domain space and applying a regularization strategy to control the input of high-frequency signals. This method effectively reduces high-frequency noise and retains key details to improve the quality and stability of rendering. Compared with other methods, the proposed method improves the ability of the model to understand the scene significantly, and improves the rendering quality and detail preservation ability. It is especially suitable for scene reconstruction from the sparse input perspective.
References:
[1] 李静, 杨宜民, 蔡述庭. 多视图的三维景物中平表面重建[J]. 智能系统学报, 2014, 9(4): 454-460.
LI Jing, YANG Yimin, CAI Shuting. 3-D scene plane reconstruction based on multiple views[J]. CAAI transactions on intelligent systems, 2014, 9(4): 454-460.
[2] MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.
[3] BARRON J T, MILDENHALL B, TANCIK M, et al. Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 5835-5844.
[4] MüLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM transactions on graphics, 2022, 41(4): 1-15.
[5] 陈国龙. 郑州双槐树遗址景观演化复原与三维建模[D]. 北京: 中国科学院大学(中国科学院空天信息创新研究院), 2022.
CHEN Guolong. Landscape evolution restoration and 3D modeling of Shuanghuaishu site in Zhengzhou[D]. Beijing: Aerospace Information Research Institute, Chinese Academy of Sciences, 2022.
[6] 罗畅. 基于GIS与BIM技术的历史建筑数字孪生管理系统研究[J]. 房地产世界, 2023(21): 82-84.
LUO Chang. Research on digital twin management system of historical buildings based on GIS and BIM technology[J]. Real estate world, 2023(21): 82-84.
[7] YU A, YE V, TANCIK M, et al. PixelNeRF: neural radiance fields from one or few images[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 4576-4585.
[8] CHEN Anpei, XU Zexiang, ZHAO Fuqiang, et al. MVSNeRF: fast generalizable radiance field reconstruction from multi-view stereo[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 14104-14113.
[9] TANCHENKO A. Visual-PSNR measure of image quality[J]. Journal of visual communication and image representation, 2014, 25(5): 874-878.
[10] HORé A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. Istanbul: IEEE, 2010: 2366-2369.
[11] KETTUNEN M, H?RK?NEN E, LEHTINEN J. E-LPIPS: robust perceptual image similarity via random transformation ensembles[EB/OL]. (2019-06-10)[2024-01-29]. https://arxiv.org/abs/1906.03973.
[12] DENG Kangle, LIU A, ZHU Junyan, et al. Depth-supervised NeRF: fewer views and faster training for free[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 12872-12881.
[13] 范腾, 杨浩, 尹稳, 等. 基于神经辐射场的多尺度视图合成研究[J]. 图学学报, 2023, 44(6): 1140-1148.
FAN Teng, YANG Hao, YIN Wen, et al. Multi-scale view synthesis based on neural radiance field[J]. Journal of graphics, 2023, 44(6): 1140-1148.
[14] 付前程. 基于神经隐式学习的多视图三维重建算法研究[D]. 武汉: 华中科技大学, 2023.
FU Qiancheng. Research on multi-view 3D reconstruction algorithm based on neural implicit learning[D]. Wuhan: Huazhong University of Science and Technology, 2023.
[15] NIEMEYER M, BARRON J T, MILDENHALL B, et al. RegNeRF: regularizing neural radiance fields for view synthesis from sparse inputs[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5470-5480.
[16] ZHANG J, YANG Gengshan, TULSIANI S, et al. NeRS: neural reflectance surfaces for sparse-view 3D reconstruction in the wild[J]. Advances in neural information processing systems., 2021(34): 29835-29847.
[17] ROESSLE B, BARRON J T, MILDENHALL B, et al. Dense depth priors for neural radiance fields from sparse input views[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 12882-12891.
[18] GOODWIN M M, AVENDANO C. Frequency-domain algorithms for audio signal enhancement based on transient modification[J]. AES: journal of the audio engineering society, 2006, 54(9): 827-840.
[19] 王玉文, 胡顺波. 数字图像处理形态学的空域与频域实现[J]. 电脑知识与技术, 2022, 18(18): 74-76.
WANG Yuwen, HU Shunbo. Spatial and frequency domain realization of digital image processing morphology[J]. Computer knowledge and technology, 2022, 18(18): 74-76.
[20] 陈树丽, 张学帅, 张鹏远, 等. 静音掩蔽和频域分段的音频指纹检索算法[J]. 声学学报, 2022, 47(4): 531-540.
CHEN Shuli, ZHANG Xueshuai, ZHANG Pengyuan, et al. Audio fingerprint retrieval method using anti-fingerprint and frequency domain segmentation[J]. Acta acustica, 2022, 47(4): 531-540.
[21] SHEN Weichao, JIA Yunde, WU Yuwei. 3D shape reconstruction from images in the frequency domain[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 4466-4474.
[22] ?IRI? D, PERI? Z, NIKOLI? J, et al. Audio signal mapping into spectrogram-based images for deep learning applications[C]//2021 20th International Symposium INFOTEH-JAHORINA. East Sarajevo: IEEE, 2021: 1-6.
[23] SHARAN R V, MOIR T J. Subband time-frequency image texture features for robust audio surveillance[J]. IEEE transactions on information forensics and security, 2015, 10(12): 2605-2615.
[24] ZHANG Ruiqi, SONG Peng, LIU Baohua, et al. Low-frequency swell noise suppression based on U-Net[J]. Applied geophysics, 2020, 17(3): 419-431.
[25] DUCHêNE S, RIANT C, CHAURASIA G, et al. Multiview intrinsic images of outdoors scenes with an application to relighting[J]. ACM transactions on graphics, 2015, 34(5): 1-16.
[26] LIANG Zexiao, TAN Guoliang, SUN Chen, et al. An effective clustering algorithm for the low-quality image of integrated circuits via high-frequency texture components extraction[J]. Electronics, 2022, 11(4): 572.
[27] SONG Liangchen, LI Zhong, GONG Xuan, et al. Harnessing low-frequency neural fields for few-shot view synthesis[EB/OL]. (2023-03-15)[2024-01-29]. https://arxiv.org/abs/2303.08370.
[28] TANCIK M, SRINIVASAN P P, MILDENHALL B, et al. Fourier features let networks learn high frequency functions in low dimensional domains[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020, 33: 7537-7547.
[29] SHENG Zehua, LIU Xiongwei, CAO Siyuan, et al. Frequency-domain deep guided image denoising[J]. IEEE transactions on multimedia, 2022, 25: 6767-6781.
[30] PREWITT J M S. Object enhancement and extraction[J]. Picture processing and psychopictorics, 1970, 10(1): 15-19.
[31] WANG Guangcong, CHEN Zhaoxi, LOY C C, et al. SparseNeRF: distilling depth ranking for few-shot novel view synthesis[C]//2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 9031-9042.
Similar References:

Memo

-

Last Update: 2024-09-05

Copyright © CAAI Transactions on Intelligent Systems