Deep 3D perception of people and their mobility aids.

Abstract Robots operating in populated environments, such as hospitals, office environments or airports, encounter a large variety of people with some of them having an advanced need for cautious interaction because of their advanced age or motion impairments. To provide appropriate assistance and support robot helpers require the ability to recognize people and their potential requirements. In this article, we present a people detection framework that distinguishes people according to the mobility aids they use. Our framework uses a deep convolutional neural network for detecting people in image data. For human-aware robots it is necessary to know where people are in a 3D world reference frame instead of only locating them in a 2D image, therefore we add a 3D centroid regression output to the network to predict the Cartesian position of people. We further use a probabilistic class, position and velocity tracker to account for false detections and occlusions. Our framework comes in two variants: The depth only variant targets high privacy demands, while the RGB only framework provides improved detection performance for non-critical applications. Both variants do not require additional geometric information about the environment. We demonstrate our approach using a dedicated dataset acquired with the support of a mobile robotic platform. The dataset contains five classes: pedestrian, person in wheelchair, pedestrian pushing a person in a wheelchair, person using crutches and person using a walking frame. Our framework achieves an mAP of 0.87 for RGB and 0.79 for depth images at a detection distance threshold of 0.5 m on our dataset, with a runtime of 53 ms per image. The annotated dataset is publicly available and our framework is made open source as a ROS people detector. Highlights • Deep learning-based framework for 3D people detection. • Perception of people according to their mobility aids. • Flexible with respect to the employed sensor modality: RGB or depth. • Probabilistic position, velocity and class estimation module enhances detector. • Enables service robots to assist people according to their impairments. [ABSTRACT FROM AUTHOR]