Object detection and localization in 3D environment by fusing raw fisheye image and attitude data.

Highlights • Use single fisheye camera to cover a hemisphere FOV of MAV. • Use original fisheye images to implement object detection. • Propose a detector that is more accurate and faster than baselines on TX2. • Fuse fisheye model, detection results, attitude and height to localize objects. Abstract In robotic systems, the fisheye camera can provide a large field of view (FOV). Usually, the traditional restoring algorithms are needed, which are computational heavy and will introduce noise into original data, since the fisheye images are distorted. In this paper, we propose a framework to detect objects from the raw fisheye images without restoration, then locate objects in the real world coordinate by fusing attitude information. A deep neural network architecture based on the MobileNet and feature pyramid structure is designed to detect targets directly on the fisheye raw images. Then, the target can be located based on the fisheye visual model and the attitude of the camera. Compared to traditional approaches, this approach has advantages in computational efficiency and accuracy. This approach is validated by experiments with a fisheye camera and an onboard computer on a micro-aerial vehicle (MAV). [ABSTRACT FROM AUTHOR]