Towards multimodal and condition-invariant vision-based registration for robot positioning on changing surfaces

Autonomously positioning a robot arm toolpoint, mobile base or UAV relative to a surface of interest is a fundamental capability in many applications such as infrastructure monitoring and planetary rock inspection. Robots performing this task in uncontrolled real world environments face many challenges such as adverse weather conditions, changing illumination or even the need to perform positioning using multiple sensing modalities. In this paper, we address this key robotics challenge in the context of the Mars2020 rover mission. Scientists require the ability to manually identify a target on a rocky outcrop using a centrally mounted RGB sensor and command the rover to autonomously position a robot arm toolpoint at that target on the following day, relying only on the x-ray sensor mounted on the arm toolpoint itself. We use an adapted sequence-based technique for multi-modal image registration that builds on recent appearance-invariant robotic place recognition algorithms. We introduce a new large, custom dataset of rock samples with multimodal sensing observations, and compare the performance of the proposed technique to a convolutional neural network (CNN)-based approach as well as a traditional Speeded-Up Robust Features (SURF)-based approach. Finally, we demonstrate the entire system performing vision-based positioning of a robot arm tool point on actual rock samples with significant illumination change.