[1]亢洁,刘威.特征融合的装修案例跨模态检索方法[J].智能系统学报,2024,19(2):429-437.[doi:10.11992/tis.202207030]
KANG Jie,LIU Wei.A cross-modal retrieval algorithm of decoration cases on feature fusion[J].CAAI Transactions on Intelligent Systems,2024,19(2):429-437.[doi:10.11992/tis.202207030]
点击复制
KANG Jie,LIU Wei.A cross-modal retrieval algorithm of decoration cases on feature fusion[J].CAAI Transactions on Intelligent Systems,2024,19(2):429-437.[doi:10.11992/tis.202207030]
特征融合的装修案例跨模态检索方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第2期
页码:
429-437
栏目:
学术论文—人工智能基础
出版日期:
2024-03-05
- Title:
- A cross-modal retrieval algorithm of decoration cases on feature fusion
- Keywords:
- home decoration customer service system; decoration case retrieval; cross-modal retrieval; style aggregation; multimodal; feature fusion; channel attention mechanism; semantic information
- 分类号:
- TP391
- 文献标志码:
- 2023-11-16
- 摘要:
- 目前家装客服系统中主要依靠人工方式进行装修案例检索，导致该系统不能满足用户对咨询服务快捷、及时的需求而且人力成本高，故提出一种基于特征融合的装修案例跨模态检索算法。针对多模态数据的语义信息挖掘不充分，模型检索精度低等问题，对现有的风格聚合模块进行改进，在原始模块中引入通道注意力机制，以此来为每组装修案例中不同图片的特征向量添加合适的权重，从而增强包含更多有用信息的重要特征并削弱其他不重要的特征。同时，为充分利用多模态信息，设计一种适用于检索场景下的多模态特征融合模块，该模块能够自适应地控制2种不同模态的特征向量进行一系列的融合操作，以实现跨模态数据间的知识流动与共享，从而生成语义更丰富、表达能力更强的特征向量，进一步提升模型的检索性能。在自建的装修案例多模态数据集上将该方法与其他方法进行比较，试验结果表明本文方法在装修案例检索上具有更优越的性能。
- Abstract:
- Currently, home decoration customer service systems chiefly depend on manual decoration case retrieval, which leads to the system not meeting user demand for fast and timely consulting service and high labor costs. Thus, we propose a feature fusion-based cross-modal retrieval algorithm for decoration cases. Aiming at the problems of insufficient semantic information mining of multimodal data and low accuracy of model retrieval, the existing style aggregation module is improved. The channel attention mechanism is introduced into the original module to add suitable weights to the feature vectors of different pictures in each group of decoration cases, thereby improving the important features that include more helpful information and weakening other unimportant features. Conversely, a multimodal feature fusion module is developed for retrieval scenarios to make full use of multimodal information. The module can adaptively control a series of fusion operations of feature vectors from two different modalities to achieve knowledge flow and sharing between cross-modal data, thereby producing feature vectors with richer semantics and stronger expressive ability to improve the retrieval performance of the model further. Our developed algorithm is compared with other methods on a self-built multimodal dataset of decoration cases, and results show that the algorithm performs better in decoration case retrieval.
- 参考文献/References:
-
[1] 刘颖,郭莹莹,房杰,等. 深度学习跨模态图文检索研究综述[J]. 计算机科学与探索, 2022, 16(3): 489–511
LIU Ying, GUO Yingying, FANG Jie, et al. Survey of research on deep learning cross-modal image-text retrieval[J]. Journal of frontiers of computer science and technology, 2022, 16(3): 489–511
[2] 徐文婉, 周小平, 王佳. 跨模态检索技术研究综述[J]. 计算机工程与应用, 2022, 58(23): 12–23
XU Wenwan, ZHOU Xiaoping, WANG Jia. Overview of cross-modal retrieval technology[J]. Computer engineering and applications, 2022, 58(23): 12–23
[3] 宫大汉, 陈辉, 陈仕江, 等. 一致性协议匹配的跨模态图像文本检索方法[J]. 智能系统学报, 2021, 16(6): 1143–1150
GONG Dahan, CHEN Hui, CHEN Shijiang, et al. Matching with agreement for cross-modal image-text retrieval[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1143–1150
[4] 刘卓锟, 刘华平, 黄文美, 等. 视听觉跨模态表面材质检索[J]. 智能系统学报, 2019, 14(3): 423–429
LIU Zhuokun, LIU Huaping, HUANG Wenmei, et al. Audiovisual cross-modal retrieval for surface material[J]. CAAI transactions on intelligent systems, 2019, 14(3): 423–429
[5] ZHANG Qi, LEI Zhen, ZHANG Zhaoxiang, et al. Context-aware attention network for image-text retrieval[C]// Computer Vision and Pattern Recognition.Seattle: IEEE, 2020: 3536-3545.
[6] DONG Jianfeng, LI Xirong, XU Chaoxi, et al. Dual encoding for video retrieval by text[EB/OL]. (2020-10-10)[2022-01-01]. http://arxiv.org/abs/2009.05381.
[7] RAMACHANDRAM D, TAYLOR G W. Deep multimodal learning: a survey on recent advances and trends[J]. IEEE signal processing magazine, 2017, 34(6): 96–108.
[8] 彭良康, 卢向明, 徐清波. 基于深度学习的跨模态哈希检索研究进展[J]. 数据通信, 2022(3): 32–38
PENG Liangkang, LU Xiangming, XU Qingbo. Research progress of cross-modal hash retrieval based on deep learning[J]. Data communications, 2022(3): 32–38
[9] 许炫淦, 房小兆, 孙为军, 等. 语义嵌入重构的跨模态哈希检索[J]. 计算机应用研究, 2022, 39(6): 1645–1650,1672
XU Xugeng, FANG Xiaozhao, SUN Weijun, et al. Semantics embedding and reconstructing for cross-modal hashing retrieval[J]. Application research of computers, 2022, 39(6): 1645–1650,1672
[10] 冯霞, 胡志毅, 刘才华. 跨模态检索研究进展综述[J]. 计算机科学, 2021, 48(8): 13–23
FENG Xia, HU Zhiyi, LIU Caihua. Survey of research progress on cross-modal retrieval[J]. Computer science, 2021, 48(8): 13–23
[11] 尹奇跃, 黄岩, 张俊格, 等. 基于深度学习的跨模态检索综述[J]. 中国图象图形学报, 2021, 26(6): 1368–1388
YIN Qiyue, HUANG Yan, ZHANG Junge, et al. Survey on deep learning based cross-modal retrieval[J]. Journal of image and graphics, 2021, 26(6): 1368–1388
[12] RASIWASIA N, PEREIRA J C, COVIELLO E, et al. A new approach to cross-modal multimedia retrieval[C]//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM, 2010: 251-260.
[13] ZHANG Hong, LIU Yun, MA Zhigang. Fusing inherent and external knowledge with nonlinear learning for cross-media retrieval[J]. Neurocomputing, 2013, 119: 10–16.
[14] KAN Meina, SHAN Shiguang, ZHANG Haihong, et al. Multi-view discriminant analysis[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 38(1): 188-194.
[15] YANG Xiangfei , LI Chunna, SHAO Yuanhai. Robust multi-view discriminant analysis with view-consistency[J]. Information sciences, 2022, 596: 153–168.
[16] PENG Yuxin, QI Jinwei, HUANG Xin, et al. CCL: cross-modal correlation learning with multigrained fusion by hierarchical network[J]. IEEE transactions on multimedia, 2018, 20(2): 405–420.
[17] ZHANG Ying, LU Huchuan. Deep cross-modal projection learning for image-text matching[C]//European Conference on Computer Vision. Cham: Springer, 2018: 707-723.
[18] 陈曦, 彭姣, 张鹏飞, 等. 基于预训练模型和编码器的图文跨模态检索算法[J]. 北京邮电大学学报, 2023, 46(5): 112–117
CHEN Xi, PENG Jiao, ZHANG Pengfei, et al. Cross-modal retrieval algorithm for image and text based on pretrained models and encoders[J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(5): 112–117
[19] ZHEN Liangli, HU Peng, WANG Xu, et al. Deep supervised cross-modal retrieval[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach: IEEE, 2020: 10386-10395.
[20] ZHANG Yifan, ZHOU Wengang, WANG Min, et al. Deep relation embedding for cross-modal retrieval[J]. IEEE transactions on image processing, 2021, 30: 617–627.
[21] WANG Cheng, YANG Haojin, MEINEL C. Deep semantic mapping for cross-modal retrieval[C]//2015 IEEE 27th International Conference on Tools with Artificial Intelligence. Vietri sul Mare: IEEE, 2016: 234-241.
[22] XU Xing, HE Li, LU Huimin, et al. Deep adversarial metric learning for cross-modal retrieval[J]. World wide web, 2019, 22(2): 657–672.
[23] ZHANG Qi, LEI Zhen, ZHANG Zhaoxiang, et al. Context-aware attention network for image-text retrieval[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3533-3542.
[24] 亢洁, 刘威. 面向装修案例智能匹配的跨模态检索方法[J]. 智能系统学报, 2022, 17(4): 714–720
KANG Jie, LIU Wei. A crossmodal retrieval method for intelligent matching of decoration cases[J]. CAAI transactions on intelligent systems, 2022, 17(4): 714–720
[25] DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-11-11)[2022-01-01]. https://arxiv.org/abs/1810.04805.pdf.
[26] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]// International Conference on Learning Representations. Cambridge: MIT Press, 2015: 1768-1776.
[27] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
[28] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2414-2423.
[29] WANG Zihao, LIU Xihui, LI Hongsheng, et al. CAMP: cross-modal adaptive message passing for text-image retrieval[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2020: 5763-5772.
[30] FUKUI A, PARK D H, YANG D, et al. Multimodal compact bilinear pooling for visual question answering and visual grounding[EB/OL]. (2016-06-06)[2022-01-01].https://arxiv.org/abs/1606.01847.pdf.
- 相似文献/References:
-
[1]亢洁,刘威.面向装修案例智能匹配的跨模态检索方法[J].智能系统学报,2022,17(4):714.[doi:10.11992/tis.202106012]
KANG Jie,LIU Wei.A crossmodal retrieval method for intelligent matching of decoration cases[J].CAAI Transactions on Intelligent Systems,2022,17():714.[doi:10.11992/tis.202106012]
备注/Memo
收稿日期:2022-07-20。
基金项目:陕西省重点研发计划项目（2021GY-022）.
作者简介:亢洁，副教授，主要研究方向为机器学习、模式识别。近几年主持和参与教学科研项目20余项，授权发明专利2项，发表学术论文20余篇。 E-mail：kangjie@sust.edu.cn;刘威，硕士研究生，主要研究方向为数字图像处理、多模态表示学习。E-mail：535473833@qq.com
通讯作者:亢洁. E-mail：kangjie@sust.edu.cn
更新日期/Last Update:
1900-01-01