A novel approach for the 3D localization of branch picking points based on deep learning applied to longan harvesting UAVs.

• An RGB-D camera is mounted on the UAV to collect RGB and depth images. • Combine object detection with semantic segmentation. • A method combining CNN and RGB-D image information. • The 3D localization of branch picking points based on deep learning. • The performance of the whole scheme was evaluated by orchard experiments. Longan is a famous speciality fruit and cultivated medicinal plant that has important edible and medicinal value; how to improve productivity in harvest is an important issue. At present, longan is mainly planted in hilly areas. For complex site conditions and tall trees, the ground harvesting machineries cannot work normally. In this study, aiming at harvesting longan fruit using unmanned aerial vehicles, a method combining an improved YOLOv5s, improved DeepLabv3+ model and depth image information is proposed, which is used for the three-dimensional (3D) positioning of branch picking points in complex natural environments. First, the improved YOLOv5s model is used to quickly detect longan fruit skewers and the main fruit branches from a complex orchard environment. The correct main fruit branch is obtained according to its relative position relationship and is extracted as the input to the semantic segmentation model. Second, using the improved DeepLabv3+ model, the image extracted in the previous step is semantically segmented to obtain the 2D coordinate information of the main longan fruit branches. Finally, combined with the growth characteristics of a longan fruit string, RGB-D information fusion is carried out on the main fruit branches in 3D space to obtain the central axis and pose information of the main fruit branches, and the 3D coordinates of the picking points are calculated, which provides destination information for a longan harvesting drone. To verify the effectiveness of the proposed method, an experiment for identifying and locating the main fruit branches and picking points was carried out in a longan orchard. The experimental results show that the longan string fruit and main fruit branch detection accuracy is 85.50%, and the main fruit branch semantic segmentation accuracy is 94.52%. The whole algorithm takes 0.58 s in the actual scene and can quickly and accurately locate the picking points. In summary, this paper fully exploits the advantages of the combination of a convolutional neural network and RGB-D image information, further improving the efficiency of longan harvesting drones in accurately positioning picking points in 3D space. [ABSTRACT FROM AUTHOR]