Multi-Modal Robotic Learning, Reasoning and Planning

Building an intelligent robot that is capable of collaborating with humans in daily tasks is a challenging problem. Although recent artificial intelligence research shows remarkable results in classical tasks, there is still a long way to achieve human-level intelligent robots. We need to start developing methods in terms of perception, learning, reasoning, and planning. In this dissertation, we study multi-modal robotic learning, reasoning, and planning from three different perspectives: (i) robot imitation learning: we first introduce a series of works including hardware prototype, data collection, modeling human demonstration, and planning for robot imitation learning. (ii) multi-modal reasoning: we study multi-modal reasoning in two different tasks. We develop a dataset and models for visual abstraction reasoning with human IQ test. Additionally, we propose a visual language reasoning method for outside knowledge visual question answering. (iii) robot planning: we show our attempts in robot planning. We introduce a physically realistic virtual testbed where robots can interact with humans. In addition, we show a hierarchical reinforcement learning method for robot planning.