[1]LIU Zhuokun,LIU Huaping,HUANG Wenmei,et al.Audiovisual cross-modal retrieval for surface material[J].CAAI Transactions on Intelligent Systems,2019,14(3):423-429.[doi:10.11992/tis.201804030]
Copy

Audiovisual cross-modal retrieval for surface material

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 14 Number of periods: 2019 3 Page number: 423-429 Column: 学术论文—智能系统 Public date: 2019-05-05
Title:
Audiovisual cross-modal retrieval for surface material
Author(s):
LIU Zhuokun1;  LIU Huaping2;  HUANG Wenmei1;  WANG Bowen1;  SUN Fuchun2
1. State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, China;
2. State Key Lab of Intelligent Technology and Systems, Tsinghua University, Beijing 100084, China
Keywords:
cross-modal retrieval; feature extraction; canonical correlation analysis; subspace mapping; material analysis; convolutional neural network; Mel-frequency cepstral coefficients; Euclidean distance
CLC:
TP391
DOI:
10.11992/tis.201804030
Abstract:
Text and image features sometimes do not allow for true and accurate analysis of the material. To solve this problem, a cross-modal method for surface material retrieval in an audiovisual field is proposed. First, the sound feature is extracted using mel frequency cepstral coefficients (MFCCs), and the image feature is extracted using convolutional neural network (CNN). Then, these two features are mapped to the subspace using canonical correlation analysis and are further retrieved via Euclidean distance. Experimental validation performed using the tactile texture dataset of the Technical University of Munich showed that the proposed method has a good application effect on material retrieval.
References:
[1] MANDAL D, BISWAS S. Query specific re-ranking for improved cross-modal retrieval[J]. Pattern Recognition Letters, 2017, 98:110-116.
[2] WANG Kaiye, HE Ran, WANG Liang, et al. Joint feature selection and subspace learning for cross-modal retrieval[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 38(10):2010-2023.
[3] DENG Cheng, TANG Xu, YAN Junchi, et al. Discriminative dictionary learning with common label alignment for cross-modal retrieval[J]. IEEE transactions on multimedia, 2016, 18(2):208-218.
[4] ZHANG Liang, MA Bingpeng, LI Guorong, et al. Metric based on multi-order spaces for cross-modal retrieval[C]//Proceedings of 2017 IEEE International Conference on Multimedia and Expo. Hong Kong, China, 2017:1374-1379.
[5] 张毅, 谢延义, 罗元, 等. 一种语音特征提取中Mel倒谱系数的后处理算法[J]. 智能系统学报, 2016, 11(2):208-215 ZHANG Yi, XIE Yanyi, LUO Yuan, et al. Postprocessing method of MFCC in speech feature extraction[J]. CAAI transactions on intelligent systems, 2016, 11(2):208-215
[6] WEI Yunchao, ZHAO Yao, LU Canyi, et al. Cross-modal retrieval with CNN visual features:a new baseline[J]. IEEE transactions on cybernetics, 2017, 47(2):449-460.
[7] RANJAN V, RASIWASIA N, JAWAHAR C V. Multi-label cross-modal retrieval[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile, 2015:4094-4102.
[8] SHARMA A, KUMAR A, DAUME H, et al. Generalized multiview analysis:a discriminative latent space[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA, 2012:2160-2167.
[9] HARDOON D R, SZEDMAK S, SHAWE-TAYLOR J. Canonical correlation analysis:an overview with application to learning methods[J]. Neural Computation, 2004, 16(12):2639-2664.
[10] CHEN Yongming, WANG Liang, WANG Wei, et al. Continuum regression for cross-modal multimedia retrieval[C]//Proceedings of the 19th IEEE International Conference on Image Processing. Orlando, USA, 2013:1949-1952.
[11] MANDAL D, BISWAS S. Generalized coupled dictionary learning approach with applications to cross-modal matching[J]. IEEE transactions on image processing, 2016, 25(8):3826-3837.
[12] STRESE M, SCHUWERK C, IEPURE A, et al. Multimodal feature-based surface material classification[J]. IEEE transactions on haptics, 2017, 10(2):226-239.
[13] CAO Jiuwen, ZHAO Tuo, WANG Jianzhong, et al. Excavation equipment classification based on improved MFCC features and ELM[J]. Neurocomputing, 2017, 261:231-241.
[14] RASIWASIA N, MAHAJAN D, MAHADEVAN V, et al. Cluster canonical correlation analysis[C]//Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics. Reykjavik, Iceland, 2014:823-831.
[15] STRESE M, BOECK Y, STEINBACH E. Content-based surface material retrieval[C]//Proceedings of 2017 IEEE World Haptics Conference. Munich, Germany, 2017:352-357.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems