Search

• An RGB-D camera is mounted on the UAV to collect RGB and depth images. • Combine object detection with semantic segmentation. • A method combining CNN and RGB-D image information. • The 3D localization of branch picking points based on deep learning. • The performance of the whole scheme was evaluated by orchard experiments. Longan is a famous speciality fruit and cultivated medicinal plant that has important edible and medicinal value; how to improve productivity in harvest is an important issue. At present, longan is mainly planted in hilly areas. For complex site conditions and tall trees, the ground harvesting machineries cannot work normally. In this study, aiming at harvesting longan fruit using unmanned aerial vehicles, a method combining an improved YOLOv5s, improved DeepLabv3+ model and depth image information is proposed, which is used for the three-dimensional (3D) positioning of branch picking points in complex natural environments. First, the improved YOLOv5s model is used to quickly detect longan fruit skewers and the main fruit branches from a complex orchard environment. The correct main fruit branch is obtained according to its relative position relationship and is extracted as the input to the semantic segmentation model. Second, using the improved DeepLabv3+ model, the image extracted in the previous step is semantically segmented to obtain the 2D coordinate information of the main longan fruit branches. Finally, combined with the growth characteristics of a longan fruit string, RGB-D information fusion is carried out on the main fruit branches in 3D space to obtain the central axis and pose information of the main fruit branches, and the 3D coordinates of the picking points are calculated, which provides destination information for a longan harvesting drone. To verify the effectiveness of the proposed method, an experiment for identifying and locating the main fruit branches and picking points was carried out in a longan orchard. The experimental results show that the longan string fruit and main fruit branch detection accuracy is 85.50%, and the main fruit branch semantic segmentation accuracy is 94.52%. The whole algorithm takes 0.58 s in the actual scene and can quickly and accurately locate the picking points. In summary, this paper fully exploits the advantages of the combination of a convolutional neural network and RGB-D image information, further improving the efficiency of longan harvesting drones in accurately positioning picking points in 3D space. [ABSTRACT FROM AUTHOR]

• Detect and locate fruits based on deep learning is the key to the success of fruit picking. • UAV fuzzy image preprocessing is the basic condition for longan detection. • Reduce the number of calculations and parameters is vital for target detection task. • The positioning strategy of suitable cutting points on fruit branches is elaborated in detail. • The performance of picking platform is evaluated by performing picking experiments. In agriculture, fruit picking robots on the ground have difficulty adapting to the terrain conditions of mountain orchards and cannot pick longan fruit from tall longan trees. In this paper, aiming to allow picking of longan fruit by unmanned aerial vehicles (UAVs), a deep learning-based scheme to quickly and accurately detect and locate suitable picking points on fruit branches is proposed. The scheme includes a UAV fuzzy image preprocessing method, longan detection based on a convolutional neural network (CNN), red, green, blue and depth (RGB-D) information fusion and an accurate target location strategy. First, the UAV is equipped with an Intel Realsense D455 camera, which collects longan images from the front for training and testing the model. Second, the lightweight MobileNet backbone network is used to improve the performance of the You Only Look Once version 4 (YOLOv4) model in feature extraction. The results for the test set show that compared with the classical feature pyramid network (FPN), YOLOv3 and YOLOv4 models, this model reduces the computation, parameters and detection time of the model. Compared with MobileNet single-shot multibox detector (MobileNet-SSD) and YOLOv4-tiny, this model exhibits improved detection accuracy. Third, according to the target detection result map, a strategy is formulated to accurately determine the suitable picking point on the main branch of the result. Finally, the performance of the improved model and picking platform in the harvest scene is evaluated by performing picking experiments in a longan orchard. In summary, we fully exploit the advantages of the combination of UAVs, RGB-D cameras and CNNs to improve the speed and accuracy of target detection and location for longan picking by UAVs based on vision. [ABSTRACT FROM AUTHOR]