Search

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments. [ABSTRACT FROM AUTHOR]

In an unmanned orchard, various tasks such as seeding, irrigation, health monitoring, and harvesting of crops are carried out by unmanned vehicles. These vehicles need to be able to distinguish which objects are fruit trees and which are not, rather than relying on human guidance. To address this need, this study proposes an efficient and robust method for fruit tree detection in orchard point cloud maps. Feature extraction is performed on the 3D point cloud to form a two-dimensional feature vector containing three-dimensional information of the point cloud and the tree target is detected through the customized deep learning network. The impact of various feature extraction methods such as average height, density, PCA, VFH, and CVFH on the detection accuracy of the network is compared in this study. The most effective feature extraction method for the detection of tree point cloud objects is determined. The ECA attention module and the EVC feature pyramid structure are introduced into the YOLOv8 network. The experimental results show that the deep learning network improves the precision, recall, and mean average precision by 1.5%, 0.9%, and 1.2%, respectively. The proposed framework is deployed in unmanned orchards for field testing. The experimental results demonstrate that the framework can accurately identify tree targets in orchard point cloud maps, meeting the requirements for constructing semantic orchard maps. [ABSTRACT FROM AUTHOR]

How to non-destructively and quickly estimate the storage time of citrus fruit is necessary and urgent for freshness control in the fruit market. As a feasibility study, we present a non-destructive method for storage time prediction of Newhall navel oranges by investigating the characteristics of the rind oil glands in this paper. Through the observation using a digital microscope, the oil glands were divided into three types and the change of their proportions could indicate the rind status as well as the storage time. Images of the rind of the oranges were taken in intervals of 10 days for 40 days, and they were used to train and test the proposed prediction models based on K-Nearest Neighbors (KNN) and deep learning algorithms, respectively. The KNN-based model demonstrated explicit features for storage time prediction based on the gland characteristics and reached a high accuracy of 93.0%, and the deep learning-based model attained an even higher accuracy of 96.0% due to its strong adaptability and robustness. The workflow presented can be readily replicated to develop non-destructive methods to predict the storage time of other types of citrus fruit with similar oil gland characteristics in different storage conditions featuring high efficiency and accuracy. [ABSTRACT FROM AUTHOR]

Defective citrus fruits are manually sorted at the moment, which is a time-consuming and cost-expensive process with unsatisfactory accuracy. In this paper, we introduce a deep learning-based vision system implemented on a citrus processing line for fast on-line sorting. For the citrus fruits rotating randomly on the conveyor, a convolutional neural network-based detector was developed to detect and temporarily classify the defective ones, and a SORT algorithm-based tracker was adopted to record the classification information along their paths. The true categories of the citrus fruits were identified through the tracked historical information, resulting in high detection precision of 93.6%. Moreover, the linear Kalman filter model was applied to predict the future path of the fruits, which can be used to guide the robot arms to pick out the defective ones. Ultimately, this research presents a practical solution to realize on-line citrus sorting featuring low costs, high efficiency, and accuracy. [ABSTRACT FROM AUTHOR]

Accurate fruit localization plays an essential role in many agricultural tasks and is one of the most vital components of an autonomous harvesting robot. Although fruit localization in stationary conditions has been extensively studied, the method to perform fruit localization in dynamic conditions is rarely studied. Fruit localization under dynamic conditions is much more challenging, since movement will introduce continuously irregular ego-motion on visual sensors, which can lead to considerable accuracy degeneration or even system failure. However, fruit localization under dynamic conditions is an essential task since it can significantly improve the harvesting efficiency. In this study, a fruit localization method is proposed to perform accurate fruit localization under dynamic conditions. Firstly, an ORB-SLAM3 SLAM algorithm is utilized to estimate the camera's pose under dynamic conditions. Secondly, YOLO-V5 is used to identify apples in RGB images. Moreover, Lidar is used to measure the depth of each apple accurately. Finally, a clustering technique based on the Gaussian distribution is proposed to mitigate the accuracy loss under dynamic conditions to optimize apple positions. Our method is extensively evaluated in indoor and outdoor experiments. The results of indoor experiments show that the average localization error is 21.1 mm, with 90% of the errors are less than 30 mm. The experimental result in an orchard environment demonstrates that our proposed method can detect and localize apples under dynamic conditions in unstructured natural environments. • Development of a novel real-time dynamic fruit localization method. • Orchard reconstruction by the combination of ORB-SLAM3 and Lidar. • Achieved robust and accurate fruit localization in motion. • Combination of ORB-SLAM3, YOLO-V5, and Lidar. [ABSTRACT FROM AUTHOR]

Accurate depth-sensing is crucial in securing a high success rate of robotic harvesting in natural orchard environments. The solid-state LiDAR technique, a recently introduced LiDAR sensor, can perceive high-resolution geometric information of the scenes, which can be utilised to receive accurate depth information. Meanwhile, the fusion of the sensory data from LiDAR and the camera can significantly enhance the sensing ability of the harvesting robots. This work first introduces a LiDAR-camera fusion-based visual sensing and perception strategy to perform accurate fruit localisation in the apple orchards. Two SOTA LiDAR-camera extrinsic calibration methods are evaluated to obtain the accurate extrinsic matrix between the LiDAR and camera. After that, the point clouds and colour images are fused to perform fruit localisation using a one-stage instance segmentation network. In addition, comprehensive experiments show that LiDAR-camera achieves better visual sensing performance in natural environments. Meanwhile, introducing the LiDAR-camera fusion can largely improve the accuracy and robustness of the fruit localisation. Specifically, the standard deviations of fruit localisation using LiDAR-camera at 0.5, 1.2, and 1.8 m are 0.253, 0.230, and 0.285 cm, respectively, during the afternoon with intensive sunlight. This measurement error is much smaller compared with that from Realsense D455. Lastly, visualised point cloud 2 2 https://drive.google.com/drive/folders/16NV0Bb6N-zlvJC0bFyu-8pl4Gbh9lyOE?usp=sharing. of the apple trees have been attached to demonstrate the highly accurate sensing results of the proposed Lidar-camera fusion method. • A visual sensing approach using high-resolution LiDAR-camera fusion is proposed. • A comprehensive study of extrinsic calibration for LiDAR-camera fusion is performed. • The proposed sensing and perception method is demonstrated in the natural orchards. [ABSTRACT FROM AUTHOR]

• Development of a geometry-aware network A3N for vision sensing in natural orchards. • Combination of A3N and collision modeling as a framework to perceive the workspace. • Implementation of a global-to-local sensing strategy to perform robotic harvesting. Field robotic harvesting is a promising technique in recent development of agricultural industry. It is vital for robots to recognise and localise fruits before the harvesting in natural orchards. However, the workspace of harvesting robots in orchards is complex: many fruits are occluded by branches and leaves. It is important to estimate a proper grasping pose for each fruit before performing the manipulation. In this study, a geometry-aware network, A3N, is proposed to perform end-to-end instance segmentation and grasping estimation using both color and geometry sensory data from a RGB-D camera. Besides, workspace geometry modelling is applied to assist the robotic manipulation. Moreover, we implement a global-to-local scanning strategy, which enables robots to accurately recognise and retrieve fruits in field environments with two consumer-level RGB-D cameras. We also evaluate the accuracy and robustness of proposed network comprehensively in experiments. The experimental results show that A3N achieves 0.873 on instance segmentation accuracy, with an average computation time of 35 ms. The average accuracy of grasping estimation is 0.61 cm and 4.8 ° in centre and orientation, respectively. Overall, the robotic system that utilizes the global-to-local scanning and A3N, achieves success rate of harvesting ranging from 70% to 85% in field harvesting experiments. [ABSTRACT FROM AUTHOR]

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 s. [ABSTRACT FROM AUTHOR]

• Development of a multi-task network DaSNet-v2 for vision sensing in orchards. • Achievement of real-time and accurate detection (F1-score of 0.873) on apples. • Achievement of accurate segmentation (IoU of 0.863) on apples. • Achievement of accurate segmentation (IoU of 0.794) on branches. Development of an accurate and reliable fruit detection system is a challenging task. There are many complex conditions in orchard environments, such as changing illumination, appearance variation, and occlusion. Robotic vision is required to understand the working environments from the sensory data and guide the robotic arm to detach the fruits. In our previous work, a deep neural network DaSNet-v1 was developed to perform detection and segmentation on fruits and branches in orchard environments. However, semantic segmentation returns the mask for each class instead of each object. Segmentation on each fruit is important as it can provide abundant information of each object, especially for those overlapped fruits. This work presents an improved deep neural network DaSNet-v2, which can perform detection and instance segmentation on fruits, and semantic segmentation on branches. DaSNet-v2 is tested and validated by experimental results obtained from field-testing in an apple orchard. From the experiment results, DaSNet-v2 with resnet-101 achieves 0.868, 0.88 and 0.873 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.794 on the accuracy of branches segmentation, respectively. DaSNet-v2 with light-weight backbone resnet-18 achieves 0.85, 0.87 and 0.866 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.775 on the accuracy of branches segmentation, respectively. The average running time and weight size of light-weight DaSNet-v2 are 55 ms and 8.1 M, respectively. Experimental results show DaSNet-v2 can robustly and efficiently perform the vision sensing for robotic harvesting in apple orchards. [ABSTRACT FROM AUTHOR]

• Development of a robust and effective image labelling algorithm. • Development of an efficient deep-learning detector 'LedNet' for apple detection. • Achievement of real-time and accurate apple detection in orchards. • Validation of a fast and effective framework for apple detection. To perform robust and efficient fruit detection in orchards is challenging since there are a number of variances in the working environments. Recently, deep-learning have shown a promising performance in many visual-guided agriculture applications. However, deep-learning based approaches requires labelling on training data, which is a labour-intensive and time-consuming task. In this study, a fast implementation framework of a deep-learning based fruit detector for apple harvesting is developed. The developed framework comprises an auto label generation module and a deep-learning-based fruit detector 'LedNet'. The Label Generation algorithm utilises the multi-scale pyramid and clustering classifier to assist fast labelling of training data. LedNet adopts feature pyramid network and atrous spatial pyramid pooling to improve the detection performance of the model. A light-weight backbone is also developed and utilised to improve computational efficiency. From the experimental results, LedNet achieves 0.821 and 0.853 on recall and accuracy on apple detection in orchards, and its weights size and inference time are 7.4 M and 28 ms, respectively. The experimental results show that LedNet can perform real-time apple detection in orchards robustly and efficiently. [ABSTRACT FROM AUTHOR]

The tower plays a crucial role in wind turbine systems. However, the design and optimization of configuration parameters have traditionally been lacking in intelligent methods. This study proposes a multi-objective parameter optimization framework that incorporates artificial intelligence models. Specifically, the diameters and thicknesses of the tower are the design parameters that strongly influence two conflicting optimization objectives: mass and top deflection. The nonlinear relationship between these parameters is predicted using surrogate models, such as the Convolutional Neural Network (CNN), Back-propagation Neural Network (BPNN), and Support Vector Machine (SVM), which serve as optimization functions. Additionally, the solutions must meet the requirements for frequency, stress, and buckling. In this study, two reference wind turbines, namely, IEA-15-240 and IEA-22-280, are selected as case studies, and the open-source software WISDEM is utilized to construct the training and testing datasets. Bayesian optimization is used to fine-tune the hyperparameters. Results show that the CNN model outperforms others with larger datasets. Leveraging Deep Learning in the design of offshore wind turbines can significantly reduce mass and deflection while maintaining integrity and performance. • An AI-model assisted NSGA-II framework is proposed. • Optimization functions were established by learning model CNN, BPNN, and SVM. • The prediction performances of CNN, BPNN, and SVM are compared and analyzed. • The tower mass and top deflection of IEA-15-240 and IEA-22-240 are minimizing. • CNN can handle large size of dataset with 62 inputs and 2 outputs. [ABSTRACT FROM AUTHOR]