机器人视觉听觉融合的感知操作系统

PDF下载HTML

Title:

A perceptual manipulation system for audio-visual fusion of robots

作者:

王业飞¹, 葛泉波², 刘华平³, 陆振宇⁴

1. 南京信息工程大学 电子与信息工程学院，江苏 南京 210044;
2. 南京信息工程大学 自动化学院，江苏 南京 210044;
3. 清华大学 计算机科学与技术系，北京 100084;
4. 南京信息工程大学 人工智能学院，江苏 南京 210044

Author(s):

WANG Yefei¹, GE Quanbo², LIU Huaping³, LU Zhenyu⁴

1. School of Electronic and Information Engineering, Nanjing University of Information Science and Technology, Nanjing 210044, China;
2. School of Automation, Nanjing University of Information Science and Technology, Nanjing 210044, China;
3. Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China;
4. School of AI, Nanjing University of Information Science and Technology, Nanjing 210044, China

关键词:

视觉定位; 音频识别; 深度学习; 视觉感知; 听觉感知; 视听融合; 多模态数据; 主动操作

Keywords:

visual positioning; audio recognition; deep learning; visual perception; auditory perception; audio-visual fusion; multi-modal data; active operation

分类号:

TP391

DOI:

10.11992/tis.202111036

摘要:

智能机器人面对复杂环境的操作能力一直是机器人应用领域研究的前沿问题，指称表达是人类对指定对象定位通用的表述方式，因此这种方式常被利用到机器人的交互当中，但是单一视觉模态并不足以满足现实世界中的所有任务。因此本文构建了一种基于视觉和听觉融合的机器人感知操作系统，该系统利用深度学习算法的模型实现了机器人的视觉感知和听觉感知，捕获自然语言操作指令和场景信息用于机器人的视觉定位，并为此收集了12类的声音信号数据用于音频识别。实验结果表明：该系统集成在UR机器人上有良好的视觉定位和音频预测能力，并最终实现了基于指令的视听操作任务，且验证了视听数据优于单一模态数据的表达能力。

Abstract:

The ability of intelligent robots to function in complex environments has been a longstanding challenge in the field of robotic applications. Referential expressions are frequently utilized for object positioning, making this method a common approach in robot interactions. However, relying on a single visual modality alone is not adequate for all tasks in real-world scenarios. This study proposes a robot perception system based on the fusion of visual and auditory modalities. The system employs a deep learning algorithm model to realize the visual and auditory perceptions of the robot, and it processes natural language and scene information for visual positioning and collects data from 12 types of sound signals for audio recognition. The experimental results indicate that the system integrated into the UR robot has a strong visual positioning ability and audio prediction, and it has successfully carried out an instruction-based audio-visual operation task. The results confirm that audio-visual data has a higher expressive capability than single-modal data.