[1]WEI Junyi,DONG Hongbin,YU Zikang.Hypernetwork design for feature selection of high-dimensional small samples[J].CAAI Transactions on Intelligent Systems,2025,20(2):465-474.[doi:10.11992/tis.202402018]
Copy

Hypernetwork design for feature selection of high-dimensional small samples

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 20 Number of periods: 2025 2 Page number: 465-474 Column: 学术论文—人工智能基础 Public date: 2025-03-05
Title:
Hypernetwork design for feature selection of high-dimensional small samples
Author(s):
WEI Junyi;  DONG Hongbin;  YU Zikang
College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, China
Keywords:
feature selection; regularization network; overfitting; end-to-end; sparse reconstruction; singular value; auxiliary network; hypernetwork; high-dimensional small sample
CLC:
TP311
DOI:
10.11992/tis.202402018
Abstract:
Feature selection is a widely recognized challenge across various industries. They typically target high-dimensional datasets with fewer samples, such as those in biology and medicine field. Many regularization networks outperform complex network structures on such datasets. However, numerous underlying feature relationships can still be overfitted, particularly with limited data. This study proposes an end-to-end sparse reconstruction network to address this issue. First, the model enhances features through sparsity and singular value embedding. Then, it trains the embedding matrix through a parallel auxiliary network to reconstruct prediction weights, which implements a parameter-reducing super-network learning approach. This approach reduces the impact of overfitting on networks with fewer parameters, which effectively mitigates the influence of ineffective parameters on the network. Experiments conducted on 12 high-dimensional small-sample datasets in biology and medicine field reveal an average improvement of 3.26 percentage point in classification accuracy after dimensionality reduction in eight feature selection networks. Furthermore, the roles of the disintegration layer, reconstruction, and correlation layer are separately validated through ablation experiments, followed by weight result analysis, which further elucidates the extended applications of the model.
References:
[1] 余紫康, 董红斌. 具有混合策略的樽海鞘群特征选择算法[J]. 智能系统学报, 2024, 19(3): 757-765.
YU Zikang, DONG Hongbin. Salp swarm feature selection algorithm with a hybrid strategy[J]. CAAI transactions on intelligent systems, 2024, 19(3): 757-765.
[2] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of machine learning research, 2014, 15: 1929-1958.
[3] 金红, 胡智群. 基于非负矩阵分解的稀疏网络社区发现算法[J]. 电子学报, 2023, 51(10): 2950-2959.
JIN Hong, HU Zhiqun. The non-negative matrix factorization based algorithm for community detection in sparse networks[J]. Acta electronica sinica, 2023, 51(10): 2950-2959.
[4] 谢承旺, 郭华, 韦伟, 等. MaOEA/d2: 一种基于双距离构造的高维多目标进化算法[J]. 软件学报, 2023, 34(4): 1523-1542.
XIE Chengwang, GUO Hua, WEI Wei, et al. MaOEA/d2: many-objective evolutionary algorithm based on double distances[J]. Journal of software, 2023, 34(4): 1523-1542.
[5] 胡艳梅, 杨波, 多滨. 基于网络结构的正则化逻辑回归[J]. 计算机科学, 2021, 48(7): 281-291.
HU Yanmei, YANG Bo, DUO Bin. Logistic regression with regularization based on network structure[J]. Computer science, 2021, 48(7): 281-291.
[6] LIU Bo, WEI Ying, ZHANG Yu, et al. Deep neural networks for high dimension, low sample size data[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Melbourne: IJCAI, 2017: 2287-2293.
[7] KADRA A, LINDAUER M, HUTTER F, et al. Well-tuned simple nets excel on tabular datasets[EB/OL]. (2021-06-21)[2024-02-20]. https://arxiv.org/abs/2106.11189.
[8] CHEN Tianqi, GUESTRIN C. XGBoost: a scalable tree boosting system[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco: ACM, 2016: 785-794.
[9] VON OSWALD J, HENNING C, GREWE B F, et al. Continual learning with hypernetworks[EB/OL]. (2019-06-03)[2024-02-20]. https://arxiv.org/abs/1906.00695.
[10] ZHANG Lin, LI Xin, HE Dongliang, et al. RRSR: reciprocal reference-based image super-resolution with progressive feature alignment and selection[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 648-664.
[11] ROMERO A, CARRIER P L, ERRAQABI A, et al. Diet networks: thin parameters for fat genomics[C]//International Conference on Learning Representations. Toulon: OpenReview.net, 2017: 1-11.
[12] BALIN M F, ABID A, ZOU J Y. Concrete autoencoders: differentiable feature selection and reconstruction[C]//International Conference on Machine Learning. Los Angeles: PMLR, 2019: 444-453.
[13] SINGH D, CLIMENTE-GONZALEZ H, PETROVICH M, et al. FsNet: feature selection network on high-dimensional biological data[C]//2023 International Joint Conference on Neural Networks. Gold Coast: IEEE, 2023: 1-9.
[14] MARGELOIU A, SIMIDJIEVSKI N, LIò P, et al. Weight predictor network with feature selection for small sample tabular biomedical data[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Washington: AAAI, 2023: 9081-9089.
[15] 黄路路, 唐舒宇, 张伟, 等. 基于Lp范数的非负矩阵分解并行优化算法[J]. 计算机科学, 2024, 51(2): 100-106.
HUANG Lulu, TANG Shuyu, ZHANG Wei, et al. Non-negative matrix factorization parallel optimization algorithm based on Lp-norm[J]. Computer science, 2024, 51(2): 100-106.
[16] LEMHADRI I, RUAN Feng, TIBSHIRANI R. LassoNet: neural networks with feature sparsity[J]. The journal of machine learning research, 2021, 22(1): 5633-5661.
[17] OBOZINSKI G, TASKAR B, JORDAN M. Multi-task feature selection[J]. Statistics department, UC Berkeley, Tech. Rep, 2006, 2(2.2): 2.
[18] HUANG Xiao, SONG Qingquan, YANG Fan, et al. Large-scale heterogeneous feature embedding[C]// Proceedings of the AAAI conference on artificial intelligence. Honolulu: AAAI, 2019: 3878-3885.
[19] 刘宇宸, 宗成庆. 跨模态信息融合的端到端语音翻译[J]. 软件学报, 2023, 34(4): 1837-1849.
LIU Yuchen, ZONG Chengqing. End-to-end speech translation by integrating cross-modal information[J]. Journal of software, 2023, 34(4): 1837-1849.
[20] CHEN Haochen, SULTAN S F, TIAN Yingtao, et al. Fast and accurate network embeddings via very sparse random projection[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Beijing: ACM, 2019: 399-408.
[21] KAUR G, SINGH S, RANI R, et al. A comprehensive study of reversible data hiding (RDH) schemes based on pixel value ordering (PVO)[J]. Archives of computational methods in engineering, 2021, 28(5): 3517-3568.
[22] KHAN Z, ILTAF N, AFZAL H, et al. Enriching non-negative matrix factorization with contextual embeddings for recommender systems[J]. Neurocomputing, 2020, 380: 246-258.
[23] HUANG Tianlin, ZHAO Rujie, BI Lvqing, et al. Neural embedding singular value decomposition for collaborative filtering[J]. IEEE transactions on neural networks and learning systems, 2022, 33(10): 6021-6029.
[24] LIU Haoyue, ZHOU Mengchu, LIU Qing. An embedded feature selection method for imbalanced data classification[J]. IEEE/CAA journal of automatica sinica, 2019, 6(3): 703-715.
[25] YAO Zhewei, GHOLAMI A, SHEN Sheng, et al. ADAHESSIAN: an adaptive second order optimizer for machine learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2021: 10665-10673.
[26] YANG Junchen, LINDENBAUM O, KLUGER Y. Locally sparse neural networks for tabular biomedical data[C]//International Conference on Machine Learning. Baltimore: PMLR, 2022: 25123-25153.
Similar References:

Memo

-

Last Update: 2025-03-05

Copyright © CAAI Transactions on Intelligent Systems