Search

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments. [ABSTRACT FROM AUTHOR]

In the dynamic and unstructured environment where horticultural crops grow, obstacles and interference frequently occur but are rarely addressed, which poses significant challenges for robotic harvesting. This work proposed a tactile-enabled robotic grasping method that combines deep learning, tactile sensing, and soft robots. By integrating fin-ray fingers with embedded tactile sensing arrays and customized perception algorithms, the robot gains the ability to sense and handle branch interference during the harvesting process and thus reduce potential mechanical fruit damage. Through experimental validations, an overall 83.3–87.0% grasping status detection success rate, and a promising interference handling method have been demonstrated. The proposed grasping method can also be extended to broader robotic grasping applications wherever undesirable foreign object intrusion needs to be addressed. [ABSTRACT FROM AUTHOR]

Robotic harvesting research has seen significant achievements in the past decade, with breakthroughs being made in machine vision, robot manipulation, autonomous navigation and mapping. However, the missing capability of obstacle handling during the grasping process has severely reduced harvest success rate and limited the overall performance of robotic harvesting. This work focuses on leaf interference caused slip detection and handling, where solutions to robotic grasping in an unstructured environment are proposed. Through analysis of the motion and force of fruit grasping under leaf interference, the connection between object slip caused by leaf interference and inadequate harvest performance is identified for the first time in the literature. A learning-based perception and manipulation method is proposed to detect slip that causes problematic grasps of objects, allowing the robot to implement timely reaction. Our results indicate that the proposed algorithm detects grasp slip with an accuracy of 94%. The proposed sensing-based manipulation demonstrated great potential in robotic fruit harvesting, and could be extended to other pick-place applications. [ABSTRACT FROM AUTHOR]

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 s. [ABSTRACT FROM AUTHOR]

• Development of a multi-task network DaSNet-v2 for vision sensing in orchards. • Achievement of real-time and accurate detection (F1-score of 0.873) on apples. • Achievement of accurate segmentation (IoU of 0.863) on apples. • Achievement of accurate segmentation (IoU of 0.794) on branches. Development of an accurate and reliable fruit detection system is a challenging task. There are many complex conditions in orchard environments, such as changing illumination, appearance variation, and occlusion. Robotic vision is required to understand the working environments from the sensory data and guide the robotic arm to detach the fruits. In our previous work, a deep neural network DaSNet-v1 was developed to perform detection and segmentation on fruits and branches in orchard environments. However, semantic segmentation returns the mask for each class instead of each object. Segmentation on each fruit is important as it can provide abundant information of each object, especially for those overlapped fruits. This work presents an improved deep neural network DaSNet-v2, which can perform detection and instance segmentation on fruits, and semantic segmentation on branches. DaSNet-v2 is tested and validated by experimental results obtained from field-testing in an apple orchard. From the experiment results, DaSNet-v2 with resnet-101 achieves 0.868, 0.88 and 0.873 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.794 on the accuracy of branches segmentation, respectively. DaSNet-v2 with light-weight backbone resnet-18 achieves 0.85, 0.87 and 0.866 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.775 on the accuracy of branches segmentation, respectively. The average running time and weight size of light-weight DaSNet-v2 are 55 ms and 8.1 M, respectively. Experimental results show DaSNet-v2 can robustly and efficiently perform the vision sensing for robotic harvesting in apple orchards. [ABSTRACT FROM AUTHOR]

• Development of a robust and effective image labelling algorithm. • Development of an efficient deep-learning detector 'LedNet' for apple detection. • Achievement of real-time and accurate apple detection in orchards. • Validation of a fast and effective framework for apple detection. To perform robust and efficient fruit detection in orchards is challenging since there are a number of variances in the working environments. Recently, deep-learning have shown a promising performance in many visual-guided agriculture applications. However, deep-learning based approaches requires labelling on training data, which is a labour-intensive and time-consuming task. In this study, a fast implementation framework of a deep-learning based fruit detector for apple harvesting is developed. The developed framework comprises an auto label generation module and a deep-learning-based fruit detector 'LedNet'. The Label Generation algorithm utilises the multi-scale pyramid and clustering classifier to assist fast labelling of training data. LedNet adopts feature pyramid network and atrous spatial pyramid pooling to improve the detection performance of the model. A light-weight backbone is also developed and utilised to improve computational efficiency. From the experimental results, LedNet achieves 0.821 and 0.853 on recall and accuracy on apple detection in orchards, and its weights size and inference time are 7.4 M and 28 ms, respectively. The experimental results show that LedNet can perform real-time apple detection in orchards robustly and efficiently. [ABSTRACT FROM AUTHOR]