基于宽度学习方法的多模态信息融合

PDF下载HTML

Title:

Multi-modal information fusion based on broad learning method

作者:

贾晨¹, 刘华平^2,3, 续欣莹¹, 孙富春^2,3

1. 太原理工大学 电气与动力工程学院, 山西 太原 030600;
2. 清华大学 计算机科学与技术系, 北京 100084;
3. 清华大学 智能技术与系统国家重点实验室, 北京 100084

Author(s):

JIA Chen¹, LIU Huaping^2,3, XU Xinying¹, SUN Fuchun^2,3

1. College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030600, China;
2. Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China;
3. State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing 100084, China

关键词:

宽度学习方法; 多模态融合; 相关性分析; 特征提取; 非线性变换; 目标识别; 神经网络; RGB-D图像分类

Keywords:

broad learning method; multi-modal fusion; correlation analysis; feature extraction; nonlinear transformation; object recognition; neural networks; RGB-D images classification

分类号:

TP391

DOI:

10.11992/tis.201803022

摘要:

多模态机器学习通过有效学习各个模态的丰富特征来解决不同模态数据的融合问题。考虑到模态间的差异性，基于宽度学习方法提出了一个能够学习和融合两种模态特征的框架，首先利用宽度学习方法分别提取不同模态的抽象特征，然后将高维特征表示在同一个特征空间进行相关性学习，并通过非线性融合得到最后的特征表达，输入分类器进行目标识别。相关实验建立在康奈尔大学抓取数据集和华盛顿大学RGB-D数据集上，实验结果验证了相比于传统的融合方法，所提出的方法具有更好的稳定性和快速性。

Abstract:

Multi-modal machine learning solves the fusion problem that arises in data with different modalites by effectively learning their rich characteristics. Considering the differences between various modalities, we propose a framework that can learn and fuse two kinds of modal characteristics based on the broad learning method. This method first extracts different abstract characteristics, then represents the high-dimension features in the same space to determine their correlation. We obtain a final representation of these characteristics by nonlinear fusion and inputs these characteristics into a classifier for target recognition. Relevant experiments are conducted on the Cornell Grasping Dataset and the Washington RGB-D Object Dataset, and our experimental results confirm that, compared with traditional fusion methods, the proposed algorithm has greater stability and rapidity.