[1]魏俊伊,董红斌,余紫康.用于高维小样本特征选择的超网络设计[J].智能系统学报,2025,20(2):465-474.[doi:10.11992/tis.202402018]
WEI Junyi,DONG Hongbin,YU Zikang.Hypernetwork design for feature selection of high-dimensional small samples[J].CAAI Transactions on Intelligent Systems,2025,20(2):465-474.[doi:10.11992/tis.202402018]
点击复制
WEI Junyi,DONG Hongbin,YU Zikang.Hypernetwork design for feature selection of high-dimensional small samples[J].CAAI Transactions on Intelligent Systems,2025,20(2):465-474.[doi:10.11992/tis.202402018]
用于高维小样本特征选择的超网络设计
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第2期
页码:
465-474
栏目:
学术论文—人工智能基础
出版日期:
2025-03-05
- Title:
- Hypernetwork design for feature selection of high-dimensional small samples
- Keywords:
- feature selection; regularization network; overfitting; end-to-end; sparse reconstruction; singular value; auxiliary network; hypernetwork; high-dimensional small sample
- 分类号:
- TP311
- 摘要:
- 特征选择是受各行业广泛关注的问题。特征选择针对的数据集通常是高维的，且样本数较少，例如生物、医学领域的数据集。虽然很多的正则化网络在这种数据集上的表现能够优于复杂的网络，但是在小数据量上许多潜在的特征关系仍然会被过度挖掘，从而出现过拟合的情况。为了解决此类问题，提出了端到端的稀疏重构网络，模型先对特征进行稀有增强和奇异值嵌入，之后通过并行辅助网络对嵌入矩阵进行训练，重构预测权重，实现了削减参数的超网络学习方式。参数较少的网络受过拟合的影响也会随之减少，有效降低了无效参数对网络的影响。对生物、医学领域的12种高维小样本数据集进行了实验，并通过对比实验发现在8种特征选择网络中降维后，本网络的分类准确率平均提升了3.26百分点。另外，通过消融实验分别证明了分解层、重构层、关联层的作用，最后分析权重结果，进一步阐述了模型的扩展应用。
- Abstract:
- Feature selection is a widely recognized challenge across various industries. They typically target high-dimensional datasets with fewer samples, such as those in biology and medicine field. Many regularization networks outperform complex network structures on such datasets. However, numerous underlying feature relationships can still be overfitted, particularly with limited data. This study proposes an end-to-end sparse reconstruction network to address this issue. First, the model enhances features through sparsity and singular value embedding. Then, it trains the embedding matrix through a parallel auxiliary network to reconstruct prediction weights, which implements a parameter-reducing super-network learning approach. This approach reduces the impact of overfitting on networks with fewer parameters, which effectively mitigates the influence of ineffective parameters on the network. Experiments conducted on 12 high-dimensional small-sample datasets in biology and medicine field reveal an average improvement of 3.26 percentage point in classification accuracy after dimensionality reduction in eight feature selection networks. Furthermore, the roles of the disintegration layer, reconstruction, and correlation layer are separately validated through ablation experiments, followed by weight result analysis, which further elucidates the extended applications of the model.
- 参考文献/References:
-
[1] 余紫康, 董红斌. 具有混合策略的樽海鞘群特征选择算法[J]. 智能系统学报, 2024, 19(3): 757-765.
YU Zikang, DONG Hongbin. Salp swarm feature selection algorithm with a hybrid strategy[J]. CAAI transactions on intelligent systems, 2024, 19(3): 757-765.
[2] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of machine learning research, 2014, 15: 1929-1958.
[3] 金红, 胡智群. 基于非负矩阵分解的稀疏网络社区发现算法[J]. 电子学报, 2023, 51(10): 2950-2959.
JIN Hong, HU Zhiqun. The non-negative matrix factorization based algorithm for community detection in sparse networks[J]. Acta electronica sinica, 2023, 51(10): 2950-2959.
[4] 谢承旺, 郭华, 韦伟, 等. MaOEA/d2: 一种基于双距离构造的高维多目标进化算法[J]. 软件学报, 2023, 34(4): 1523-1542.
XIE Chengwang, GUO Hua, WEI Wei, et al. MaOEA/d2: many-objective evolutionary algorithm based on double distances[J]. Journal of software, 2023, 34(4): 1523-1542.
[5] 胡艳梅, 杨波, 多滨. 基于网络结构的正则化逻辑回归[J]. 计算机科学, 2021, 48(7): 281-291.
HU Yanmei, YANG Bo, DUO Bin. Logistic regression with regularization based on network structure[J]. Computer science, 2021, 48(7): 281-291.
[6] LIU Bo, WEI Ying, ZHANG Yu, et al. Deep neural networks for high dimension, low sample size data[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Melbourne: IJCAI, 2017: 2287-2293.
[7] KADRA A, LINDAUER M, HUTTER F, et al. Well-tuned simple nets excel on tabular datasets[EB/OL]. (2021-06-21)[2024-02-20]. https://arxiv.org/abs/2106.11189.
[8] CHEN Tianqi, GUESTRIN C. XGBoost: a scalable tree boosting system[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco: ACM, 2016: 785-794.
[9] VON OSWALD J, HENNING C, GREWE B F, et al. Continual learning with hypernetworks[EB/OL]. (2019-06-03)[2024-02-20]. https://arxiv.org/abs/1906.00695.
[10] ZHANG Lin, LI Xin, HE Dongliang, et al. RRSR: reciprocal reference-based image super-resolution with progressive feature alignment and selection[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 648-664.
[11] ROMERO A, CARRIER P L, ERRAQABI A, et al. Diet networks: thin parameters for fat genomics[C]//International Conference on Learning Representations. Toulon: OpenReview.net, 2017: 1-11.
[12] BALIN M F, ABID A, ZOU J Y. Concrete autoencoders: differentiable feature selection and reconstruction[C]//International Conference on Machine Learning. Los Angeles: PMLR, 2019: 444-453.
[13] SINGH D, CLIMENTE-GONZALEZ H, PETROVICH M, et al. FsNet: feature selection network on high-dimensional biological data[C]//2023 International Joint Conference on Neural Networks. Gold Coast: IEEE, 2023: 1-9.
[14] MARGELOIU A, SIMIDJIEVSKI N, LIò P, et al. Weight predictor network with feature selection for small sample tabular biomedical data[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Washington: AAAI, 2023: 9081-9089.
[15] 黄路路, 唐舒宇, 张伟, 等. 基于Lp范数的非负矩阵分解并行优化算法[J]. 计算机科学, 2024, 51(2): 100-106.
HUANG Lulu, TANG Shuyu, ZHANG Wei, et al. Non-negative matrix factorization parallel optimization algorithm based on Lp-norm[J]. Computer science, 2024, 51(2): 100-106.
[16] LEMHADRI I, RUAN Feng, TIBSHIRANI R. LassoNet: neural networks with feature sparsity[J]. The journal of machine learning research, 2021, 22(1): 5633-5661.
[17] OBOZINSKI G, TASKAR B, JORDAN M. Multi-task feature selection[J]. Statistics department, UC Berkeley, Tech. Rep, 2006, 2(2.2): 2.
[18] HUANG Xiao, SONG Qingquan, YANG Fan, et al. Large-scale heterogeneous feature embedding[C]// Proceedings of the AAAI conference on artificial intelligence. Honolulu: AAAI, 2019: 3878-3885.
[19] 刘宇宸, 宗成庆. 跨模态信息融合的端到端语音翻译[J]. 软件学报, 2023, 34(4): 1837-1849.
LIU Yuchen, ZONG Chengqing. End-to-end speech translation by integrating cross-modal information[J]. Journal of software, 2023, 34(4): 1837-1849.
[20] CHEN Haochen, SULTAN S F, TIAN Yingtao, et al. Fast and accurate network embeddings via very sparse random projection[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Beijing: ACM, 2019: 399-408.
[21] KAUR G, SINGH S, RANI R, et al. A comprehensive study of reversible data hiding (RDH) schemes based on pixel value ordering (PVO)[J]. Archives of computational methods in engineering, 2021, 28(5): 3517-3568.
[22] KHAN Z, ILTAF N, AFZAL H, et al. Enriching non-negative matrix factorization with contextual embeddings for recommender systems[J]. Neurocomputing, 2020, 380: 246-258.
[23] HUANG Tianlin, ZHAO Rujie, BI Lvqing, et al. Neural embedding singular value decomposition for collaborative filtering[J]. IEEE transactions on neural networks and learning systems, 2022, 33(10): 6021-6029.
[24] LIU Haoyue, ZHOU Mengchu, LIU Qing. An embedded feature selection method for imbalanced data classification[J]. IEEE/CAA journal of automatica sinica, 2019, 6(3): 703-715.
[25] YAO Zhewei, GHOLAMI A, SHEN Sheng, et al. ADAHESSIAN: an adaptive second order optimizer for machine learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2021: 10665-10673.
[26] YANG Junchen, LINDENBAUM O, KLUGER Y. Locally sparse neural networks for tabular biomedical data[C]//International Conference on Machine Learning. Baltimore: PMLR, 2022: 25123-25153.
- 相似文献/References:
-
[1]孙正兴,张尧烨,李? 彬.基于线性规划分类器的相关反馈技术[J].智能系统学报,2007,2(3):34.
SUN Zheng-xing,ZHANG Yao-ye,LI Bin.Applying relevance feedback with a linear programming classifier[J].CAAI Transactions on Intelligent Systems,2007,2():34.
[2]张志飞,苗夺谦.基于粗糙集的文本分类特征选择算法[J].智能系统学报,2009,4(5):453.[doi:10.3969/j.issn.1673-4785.2009.05.011]
ZHANG Zhi-fei,MIAO Duo-qian.Feature selection for text categorization based on rough set[J].CAAI Transactions on Intelligent Systems,2009,4():453.[doi:10.3969/j.issn.1673-4785.2009.05.011]
[3]顾成杰,张顺颐,杜安源.结合粗糙集和禁忌搜索的网络流量特征选择[J].智能系统学报,2011,6(3):254.
GU Chengjie,ZHANG Shunyi,DU Anyuan.Feature selection of network traffic using a rough set and tabu search[J].CAAI Transactions on Intelligent Systems,2011,6():254.
[4]孙倩茹,王文敏,刘宏.视频序列的人体运动描述方法综述[J].智能系统学报,2013,8(3):189.
SUN Qianru,WANG Wenmin,LIU Hong.Study of human action representation in video sequences[J].CAAI Transactions on Intelligent Systems,2013,8():189.
[5]曹晋,张莉,李凡长.一种基于支持向量数据描述的特征选择算法[J].智能系统学报,2015,10(2):215.[doi:10.3969/j.issn.1673-4785.201405063]
CAO Jin,ZHANG Li,LI Fanzhang.A noval support vector data description-based feature selection method[J].CAAI Transactions on Intelligent Systems,2015,10():215.[doi:10.3969/j.issn.1673-4785.201405063]
[6]张佳骕,蒋亦樟,王士同.基于特征选择聚类方法的稀疏TSK模糊系统[J].智能系统学报,2015,10(4):583.[doi:10.3969/j.issn.1673-4785.201412001]
ZHANG Jiasu,JIANG Yizhang,WANG Shitong.Sparse TSK fuzzy system based on feature selection clustering method[J].CAAI Transactions on Intelligent Systems,2015,10():583.[doi:10.3969/j.issn.1673-4785.201412001]
[7]陈玉明,吴克寿,李向军.基因表达数据在邻域关系中的特征选择[J].智能系统学报,2014,9(2):210.[doi:10.3969/j.issn.1673-4785.201307014]
CHEN Yuming,WU Keshou,LI Xiangjun.Gene expression data feature selection with neighborhood relation[J].CAAI Transactions on Intelligent Systems,2014,9():210.[doi:10.3969/j.issn.1673-4785.201307014]
[8]郭雨萌,李国正.一种多标记数据的过滤式特征选择框架[J].智能系统学报,2014,9(3):292.[doi:10.3969/j.issn.1673-4785.201403064]
GUO Yumeng,LI Guozheng.A filtering framework for the multi-label feature selection[J].CAAI Transactions on Intelligent Systems,2014,9():292.[doi:10.3969/j.issn.1673-4785.201403064]
[9]滕旭阳,董红斌,孙静.面向特征选择问题的协同演化方法[J].智能系统学报,2017,12(1):24.[doi:10.11992/tis.201611029]
TENG Xuyang,DONG Hongbin,SUN Jing.Co-evolutionary algorithm for feature selection[J].CAAI Transactions on Intelligent Systems,2017,12():24.[doi:10.11992/tis.201611029]
[10]路子祥,屠黎阳,祖辰,等.基于脑连接网络的阿尔茨海默病临床变量值预测[J].智能系统学报,2017,12(3):355.[doi:10.11992/tis.201607020]
LU Zixiang,TU Liyang,ZU Chen,et al.Prediction of clinical variables in Alzheimer’s disease using brain connective networks[J].CAAI Transactions on Intelligent Systems,2017,12():355.[doi:10.11992/tis.201607020]
备注/Memo
收稿日期:2024-2-20。
基金项目:黑龙江自然科学基金项目(LH2020F023).
作者简介:魏俊伊，硕士研究生，主要研究方向为群智能算法和深度学习。E-mail：weijunyi@hrbeu.edu.cn;董红斌，教授，博士生导师，中国计算机学会高级会员，主要研究方向为多智能体系统、机器学习。主持和完成国家自然科学基金项目、工信部基础研究项目、黑龙江省自然科学基金项目，荣获黑龙江省高校科学技术奖和黑龙江省优秀高等教育科学成果奖。发表学术论文90余篇，出版教材2部。E-mail： donghongbin@hrbeu.edu.cn;余紫康，硕士研究生，主要研究方向为群智能算法、数据挖掘。E-mail：yuzk6@mail2.sysu.edu.cn。
通讯作者:董红斌. E-mail：donghongbin@hrbeu.edu.cn
更新日期/Last Update:
2025-03-05