Search

Improving the effectiveness of harvesting robots requires quick and accurate apple detection in natural environments. The colour and shape features of apples are corrupted due to the reflected light and the incomplete coverage of the fruit bag, bringing difficulties to apple detection. To address this issue, the Coordinate Attention You Only Look Once version 5 (CA-YOLOv5) is designed to simultaneously detect bagged and unbagged apples in the natural environment. Firstly, 1525 apple images are collected from apple orchards to build a dataset. Secondly, to solve the reflected light problem, all C3 modules in the Backbone are substituted for Coordinate Attention modules which can improve the feature representation of objects. Finally, to solve the incomplete bagging problem, the Path Aggregation Network in the Neck is replaced by a Bidirectional Feature Pyramid Network which can better fuse the features of various sizes. The CA-YOLOv5 network reaches 82.7%, 89.8%, 48.6%, and 87.0% for recall, mAP@0.5, mAP@0.5:0.95, and F1 score, respectively, which is 2.3%,1.2%,1.9%, and 2.9% higher than the YOLOv5. The results reveal that CA-YOLOv5 has much superior detection performance than the original YOLOv5, and it can serve as a technical benchmark for the development of automatic orchard-picking robots. [ABSTRACT FROM AUTHOR]

With the development of apple-picking robots, deep learning models have become essential in apple detection. However, current detection models are often disrupted by complex backgrounds, leading to low recognition accuracy and slow speeds in natural environments. To address these issues, this study proposes an improved model, YOLOv8s-CFB, based on YOLOv8s. This model introduces partial convolution (PConv) in the backbone network, enhances the C2f module, and forms a new architecture, CSPPC, to reduce computational complexity and improve speed. Additionally, FocalModulation technology replaces the original SPPF module to enhance the model’s ability to recognize key areas. Finally, the bidirectional feature pyramid (BiFPN) is introduced to adaptively learn the importance of weights at each scale, effectively retaining multi-scale information through a bidirectional context information transmission mechanism, and improving the model’s detection ability for occluded targets. Test results show that the improved YOLOv8 network achieves better detection performance, with an average accuracy of 93.86%, a parameter volume of 8.83 M, and a detection time of 0.7 ms. The improved algorithm achieves high detection accuracy with a small weight file, making it suitable for deployment on mobile devices. Therefore, the improved model can efficiently and accurately detect apples in complex orchard environments in real time. [ABSTRACT FROM AUTHOR]

After apple fruit maturation, the optimal harvest period is short, and the picking robot is expected to improve harvesting efficiency. While it is common for apples to be overlapped and occluded by branches and leaves, which pose challenges to the robot's apple harvesting. Therefore, precise and swift identification and localization of the target fruit is crucial. To this end, this paper proposes a lightweight apple detection method, YOLOv5s-ShuffleNetV2-DWconv-Add, or "YOLOv5s-SDA" for short. The red and green apple datasets in natural environment were collected by a mobile phone, which were divided into four categories: red and green apples that can be directly grasped and cannot be directly grasped, in order to avoid damage to the robotic arm. Different deep learning object detection models were compared, with the YOLOv5s algorithm providing superior recognition performance. To improve harvest efficiency and portability of hardware devices, modifications are made to the YOLOv5s algorithm, replacing the Focus, C3, and Conv structures within the backbone with 3 × 3 Conv structures and ShuffleNetV2, removing SPP and C3 structures; substituting the C3 in the Neck portion with DWConv modules; and replacing two Concat layers in the PANet structure with smaller computational Add layers. Results demonstrate that the model achieved a mAP of 94.6% on the test set, doubled the detection speed, and compressed the model weight to 11.8% of its original value, while maintaining model accuracy. This new method exhibits promising performance in fruit target recognition in natural scenes, providing an effective means of visual acquisition for fruit picking robots. [ABSTRACT FROM AUTHOR]

Accurately recognizing apples in complex environments is essential for automating apple picking operations, particularly under challenging natural conditions such as cloudy, snowy, foggy, and rainy weather, as well as low-light situations. To overcome the challenges of reduced apple target detection accuracy due to branch occlusion, apple overlap, and variations between near and far field scales, we propose the Rep-ViG-Apple algorithm, an advanced version of the YOLO model. The Rep-ViG-Apple algorithm features a sophisticated architecture designed to enhance apple detection performance in difficult conditions. To improve feature extraction for occluded and overlapped apple targets, we developed the inverted residual multi-scale structural reparameterized feature extraction block (RepIRD Block) within the backbone network. We also integrated the sparse graph attention mechanism (SVGA) to capture global feature information, concentrate attention on apples, and reduce interference from complex environmental features. Moreover, we designed a feature extraction network with a CNN-GCN architecture, termed Rep-Vision-GCN. This network combines the local multi-scale feature extraction capabilities of a convolutional neural network (CNN) with the global modeling strengths of a graph convolutional network (GCN), enhancing the extraction of apple features. The RepConvsBlock module, embedded in the neck network, forms the Rep-FPN-PAN feature fusion network, which improves the recognition of apple targets across various scales, both near and far. Furthermore, we implemented a channel pruning algorithm based on LAMP scores to balance computational efficiency with model accuracy. Experimental results demonstrate that the Rep-ViG-Apple algorithm achieves precision, recall, and average accuracy of 92.5%, 85.0%, and 93.3%, respectively, marking improvements of 1.5%, 1.5%, and 2.0% over YOLOv8n. Additionally, the Rep-ViG-Apple model benefits from a 22% reduction in size, enhancing its efficiency and suitability for deployment in resource-constrained environments while maintaining high accuracy. [ABSTRACT FROM AUTHOR]

Improving the effectiveness of harvesting robots requires quick and accurate apple detection in natural environments. The colour and shape features of apples are corrupted due to the reflected light and the incomplete coverage of the fruit bag, bringing difficulties to apple detection. To address this issue, the Coordinate Attention You Only Look Once version 5 (CA-YOLOv5) is designed to simultaneously detect bagged and unbagged apples in the natural environment. Firstly, 1525 apple images are collected from apple orchards to build a dataset. Secondly, to solve the reflected light problem, all C3 modules in the Backbone are substituted for Coordinate Attention modules which can improve the feature representation of objects. Finally, to solve the incomplete bagging problem, the Path Aggregation Network in the Neck is replaced by a Bidirectional Feature Pyramid Network which can better fuse the features of various sizes. The CA-YOLOv5 network reaches 82.7%, 89.8%, 48.6%, and 87.0% for recall, mAP@0.5, mAP@0.5:0.95, and F1 score, respectively, which is 2.3%,1.2%,1.9%, and 2.9% higher than the YOLOv5. The results reveal that CA-YOLOv5 has much superior detection performance than the original YOLOv5, and it can serve as a technical benchmark for the development of automatic orchard-picking robots.

Apple detection plays a critical role in enabling the functionality of harvesting robots within natural orchard environments. To address challenges related to low detection accuracy, slow inference speed, and high parameter count, we present PcMNet, a lightweight detection model based on an improved YOLOv8 network. Initially, we employed Partial Convolution (Pconv) to construct a PR module, forming the Pconv-block, which subsequently replaced the original C2f feature extraction module within the YOLOv8n backbone. This replacement led to improvements in both detection accuracy and speed, while simultaneously reducing computational complexity (FLOPs), parameter count, and model size. Furthermore, the Cross-Scale Feature Fusion (CCFF) module was refined into Faster-Cross-Scale Feature Fusion (Faster-CCFF) with the integration of Pconv-block, significantly enhancing the model's feature extraction and fusion capabilities. Additionally, we introduced Mixed Local Channel Attention (MLCA) to further strengthen the model's capacity to capture essential features while effectively suppressing background noise. Experimental results demonstrate that PcMNet achieved a detection accuracy of 92.8 % and an mAP@0.5 of 95.5 %, representing improvements of 1.4 and 0.7 percentage points, respectively, over YOLOv8n. Moreover, PcMNet successfully reduced FLOPs, parameter count, and model size to 5.1 G, 1.4 M, and 3.2 MB, respectively. The per-image detection time was reduced to 2.3 ms, indicating reductions of 37.80 %, 53.33 %, 49.21 %, and 56.60 % in FLOPs, parameters, model size, and detection time compared to YOLOv8n. When deployed on edge computing devices with TensorRT acceleration, PcMNet achieved a detection rate of 92 FPS. Field validation experiments conducted in natural orchard environments confirmed PcMNet's superior ability to detect apples under challenging conditions, such as occlusions and varying lighting conditions. Its lightweight design and rapid detection capabilities provide a valuable reference for achieving real-time apple detection in automated and intelligent harvesting robots, thereby contributing to advancements in smart agriculture.

The apple industry, recognized as a pivotal sector in agriculture, increasingly emphasizes the mechanization and intelligent advancement of picking technology. This study innovatively applies a mist simulation algorithm to apple image generation, constructing a dataset of apple images under mixed sunny, cloudy, drizzling and foggy weather conditions called DNE-APPLE. It introduces a lightweight and efficient target detection network called DNE-YOLO. Building upon the YOLOv8 base model, DNE-YOLO incorporates the CBAM attention mechanism and CARAFE up-sampling operator to enhance the focus on apples. Additionally, it utilizes GSConv and the dynamic non-monotonic focusing mechanism loss function WIOU to reduce model parameters and decrease reliance on dataset quality. Extensive experimental results underscore the efficacy of the DNE-YOLO model, which achieves a detection accuracy (precision) of 90.7%, a recall of 88.9%, a mean accuracy (mAP50) of 94.3%, a computational complexity (GFLOPs) of 25.4G, and a parameter count of 10.46M across various environmentally diverse datasets. Compared to YOLOv8, it exhibits superior detection accuracy and robustness in sunny, drizzly, cloudy, and misty environments, making it especially suitable for practical applications such as apple picking for agricultural robots. The code for this model is open source at https://github.com/wuhaitao2178827/DNE-YOLO.

This paper proposes an improved You Only Look Once (YOLOv3) algorithm for automatically detecting damaged apples to promote the automation of the fruit processing industry. In the proposed method, a clustering method based on Rao-1 algorithm is introduced to optimize anchor box sizes. The clustering method uses the intersection over the union to form the objective function and the most representative anchor boxes are generated for normal and damaged apple detection. To verify the feasibility and effectiveness of the proposed method, real apple images collected from the Internet are employed. Compared with the generic YOLOv3 and Fast Region-based Convolutional Neural Network (Fast R-CNN) algorithms, the proposed method yields the highest mean average precision value for the test dataset. Therefore, it is practical to apply the proposed method for intelligent apple detection and classification tasks.

The Fruit Harvest Helper, a mobile application developed by Northwest Nazarene University’s (NNU) Robotics Vision Lab, aims to assist farmers in estimating fruit yield for apple orchards. Currently, farmers manually estimate the fruit yield for an orchard, which is a laborious task. The Fruit Harvest Helper seeks to simplify their process by detecting apples on images of apple trees. Once the number of apples is detected, a correlation can then be applied to this value to obtain a usable yield estimate for an apple tree. While prior research efforts at NNU concentrated on developing an iOS app for blossom detection, this current research aims to adapt that smart farming application for apple detection across multiple platforms, iOS and Android. Borrowing ideas from the former iOS app, the new application was designed with an intuitive user interface that is easy for farmers to use, allowing for quick image selection and processing. Unlike before, the adapted app uses a color ratio-based image-segmentation algorithm written in C++ to detect apples. This algorithm detects apples within apple tree images that farmers select for processing by using OpenCV functions and C++ code. The results of testing the algorithm on a dataset of images indicate an 8.52% Mean Absolute Percentage Error (MAPE) and a Pearson correlation coefficient of 0.6 between detected and actual apples on the trees. These findings were obtained by evaluating the images from both the east and west sides of the trees, which was the best method to reduce the error of this algorithm. The algorithm’s processing time was tested for Android and iOS, yielding an average performance of 1.16 s on Android and 0.14 s on iOS. Although the Fruit Harvest Helper shows promise, there are many opportunities for improvement. These opportunities include exploring alternative machine-learning approaches for apple detection, conducting real-world testing without any human assistance, and expanding the app to detect various types of fruit. The Fruit Harvest Helper mobile application is among the many mobile applications contributing to precision agriculture. The app is nearing readiness for farmers to use for the purpose of yield monitoring and farm management within Pink Lady apple orchards.

The current apple detection algorithms fail to accurately differentiate obscured apples from pickable ones, thus leading to low accuracy in apple harvesting and a high rate of instances where apples are either mispicked or missed altogether. To address the issues associated with the existing algorithms, this study proposes an improved YOLOv5s-based method, named YOLOv5s-BC, for real-time apple detection, in which a series of modifications have been introduced. First, a coordinate attention block has been incorporated into the backbone module to construct a new backbone network. Second, the original concatenation operation has been replaced with a bi-directional feature pyramid network in the neck network. Finally, a new detection head has been added to the head module, enabling the detection of smaller and more distant targets within the field of view of the robot. The proposed YOLOv5s-BC model was compared to several target detection algorithms, including YOLOv5s, YOLOv4, YOLOv3, SSD, Faster R-CNN (ResNet50), and Faster R-CNN (VGG), with significant improvements of 4.6%, 3.6%, 20.48%, 23.22%, 15.27%, and 15.59% in mAP, respectively. The detection accuracy of the proposed model is also greatly enhanced over the original YOLOv5s model. The model boasts an average detection speed of 0.018 s per image, and the weight size is only 16.7 Mb with 4.7 Mb smaller than that of YOLOv8s, meeting the real-time requirements for the picking robot. Furthermore, according to the heat map, our proposed model can focus more on and learn the high-level features of the target apples, and recognize the smaller target apples better than the original YOLOv5s model. Then, in other apple orchard tests, the model can detect the pickable apples in real time and correctly, illustrating a decent generalization ability. It is noted that our model can provide technical support for the apple harvesting robot in terms of real-time target detection and harvesting sequence planning. [ABSTRACT FROM AUTHOR]

The Fruit Harvest Helper, a mobile application developed by Northwest Nazarene University's (NNU) Robotics Vision Lab, aims to assist farmers in estimating fruit yield for apple orchards. Currently, farmers manually estimate the fruit yield for an orchard, which is a laborious task. The Fruit Harvest Helper seeks to simplify their process by detecting apples on images of apple trees. Once the number of apples is detected, a correlation can then be applied to this value to obtain a usable yield estimate for an apple tree. While prior research efforts at NNU concentrated on developing an iOS app for blossom detection, this current research aims to adapt that smart farming application for apple detection across multiple platforms, iOS and Android. Borrowing ideas from the former iOS app, the new application was designed with an intuitive user interface that is easy for farmers to use, allowing for quick image selection and processing. Unlike before, the adapted app uses a color ratio-based image-segmentation algorithm written in C++ to detect apples. This algorithm detects apples within apple tree images that farmers select for processing by using OpenCV functions and C++ code. The results of testing the algorithm on a dataset of images indicate an 8.52% Mean Absolute Percentage Error (MAPE) and a Pearson correlation coefficient of 0.6 between detected and actual apples on the trees. These findings were obtained by evaluating the images from both the east and west sides of the trees, which was the best method to reduce the error of this algorithm. The algorithm's processing time was tested for Android and iOS, yielding an average performance of 1.16 s on Android and 0.14 s on iOS. Although the Fruit Harvest Helper shows promise, there are many opportunities for improvement. These opportunities include exploring alternative machine-learning approaches for apple detection, conducting real-world testing without any human assistance, and expanding the app to detect various types of fruit. The Fruit Harvest Helper mobile application is among the many mobile applications contributing to precision agriculture. The app is nearing readiness for farmers to use for the purpose of yield monitoring and farm management within Pink Lady apple orchards. [ABSTRACT FROM AUTHOR]

The key technology of automated apple harvesting is detecting apples quickly and accurately. The traditional detection methods of apple detection are often slow and inaccurate in unstructured orchards. Therefore, this article proposes an improved YOLOv5s-GBR model for orchard apple detection under complex natural conditions. First, the researchers collected photos of apples in their natural environments from different angles; then, we enhanced the dataset by changing the brightness, rotating the images, and adding noise. In the YOLOv5s network, the following modules were introduced to improve its performance: First, the YOLOv5s model's backbone network was swapped out for the GhostNetV2 module. The goal of this improvement was to lessen the computational burden on the YOLOv5s algorithm while increasing the detection speed. Second, the bi-level routing spatial attention module (BRSAM), which combines spatial attention (SA) with bi-level routing attention (BRA), was used in this study. By strengthening the model's capacity to extract important characteristics from the target, its generality and robustness were enhanced. Lastly, this research replaced the original bounding box loss function with a repulsion loss function to detect overlapping targets. This model performs better in detection, especially in situations involving occluded and overlapping targets. According to the test results, the YOLOv5s-GBR model improved the average precision by 4.1% and recall by 4.0% compared to those of the original YOLOv5s model, with an impressive detection accuracy of 98.20% at a frame rate of only 101.2 fps. The improved algorithm increases the recognition accuracy by 12.7%, 10.6%, 5.9%, 2.7%, 1.9%, 0.8%, 2.6%, and 5.3% compared to those of YOLOv5-lite-s, YOLOv5-lite-e, yolov4-tiny, YOLOv5m, YOLOv5l, YOLOv8s, Faster R-CNN, and SSD, respectively, and the YOLOv5s-GBR model can be used to accurately recognize overlapping or occluded apples, which can be subsequently deployed in picked robots to meet the realistic demand of real-time apple detection. [ABSTRACT FROM AUTHOR]

Accurate and rapid identification and location of apples contributes to speeding up automation harvesting. However, in unstructured orchard environments, it is common for apples to be overlapped and occluded by branches and leaves, which interferes with apple identification and localization. In order to quickly reconstruct the fruits under overlapping and occlusion conditions, an adaptive radius selection strategy based on random sample consensus algorithm (ARSS-RANSAC) was proposed. Firstly, the edge of apple in the image was obtained by using image preprocessing method. Secondly, an adaptive radius selection strategy was proposed, which is based on fruit shape characteristics. The fruit initial radius was obtained through horizontal or vertical scanning. Then, combined with RANSAC algorithm to select effective contour points by the determined radius, and the circle center coordinates were obtained. Finally, fitting the circle according to the selected valid contour and achieving the recognition and localization of overlapped and occluded apples. 175 apple images with different overlaps and branches and leaves occlusion were applied to verify the effectiveness of algorithm. The evaluation indicators of overlap rate, average false-positive rate, average false-negative rate, and average segmentation error of ARSS-RANSAC were improved compared with the classical Hough transform method. The detection time of a single image was less than 50 ms, which can meet requirements of real-time target detection. The experimental results show that the ARSS-RANSAC algorithm can quickly and accurately identify and locate occluded and overlapped apples and is expected to be applied to harvesting robots of apple and other round fruits. [ABSTRACT FROM AUTHOR]

Considering precision agriculture, recent technological developments have sparked the emergence of several new tools that can help to automate the agricultural process. For instance, accurately detecting and counting apples in orchards is essential for maximizing harvests and ensuring effective resource management. However, there are several intrinsic difficulties with traditional techniques for identifying and counting apples in orchards. To identify, recognize, and detect apples, apple target detection algorithms, such as YOLOv7, have shown a great deal of reflection and accuracy. But occlusions, electrical wiring, branches, and overlapping pose severe issues for precisely detecting apples. Thus, to overcome these issues and accurately recognize apples and find the depth of apples from drone-based videos in complicated backdrops, our proposed model combines a multi-head attention system with the YOLOv7 object identification framework. Furthermore, we provide the ByteTrack method for apple counting in real time, which guarantees effective monitoring of apples. To verify the efficacy of our suggested model, a thorough comparison assessment is performed with several current apple detection and counting techniques. The outcomes adequately proved the effectiveness of our strategy, which continuously surpassed competing methods to achieve exceptional accuracies of 0.92, 0.96, and 0.95 with respect to precision, recall, and F1 score, and a low MAPE of 0.027, respectively. [ABSTRACT FROM AUTHOR]

Apple detection in complex orchard environments holds significant research importance for yield estimation. Although convolutional neural networks have been widely used in the field of object detection, they also have certain limitations. One of the major drawbacks is their inductive biases of locality and scale invariance, which often pose challenges in capturing global and long-term dependencies. In this work, we replace the backbone network with a moving window transformer based on Faster RCNN, fusing features from different stages and introducing an enhanced smoothing loss function called Faster RFormer. We created an apple detection dataset called AD-2023 to validate the reliability of the model. The results indicate that the proposed method in this paper achieved impressive results with 0.692 mAP, 0.796 AP@0.75 and 0.941 AP@0.5, surpassing existing algorithms. More importantly, our study not only provides a reliable idea for the inadequacy of convolutional neural networks for detection tasks in complex environments, but also establishes a new benchmark in apple detection methodologies, showcasing the potential for broader applications in agricultural automation and robotics.

Apple orchards, as an important center of economic activity in forestry special crops, can achieve yield prediction and automated harvesting by detecting and locating apples. Small apples, occlusion, dim lighting at night, blurriness, cluttered backgrounds, and other complex scenes significantly affect the automatic harvesting and yield estimation of apples. To address these issues, this study proposes an apple detection algorithm, "YOLOv5-ACS (Apple in Complex Scenes)", based on YOLOv5s. Firstly, the space-to-depth-conv module is introduced to avoid information loss, and a squeeze-and-excitation block is added in C3 to learn more important information. Secondly, the context augmentation module is incorporated to enrich the context information of the feature pyramid network. By combining the shallow features of the backbone P2, the low-level features of the object are retained. Finally, the addition of the context aggregation block and CoordConv aggregates the spatial context pixel by pixel, perceives the spatial information of the feature map, and enhances the semantic information and global perceptual ability of the object. We conducted comparative tests in various complex scenarios and validated the robustness of YOLOv5-ACS. The method achieved 98.3% and 74.3% for mAP@0.5 and mAP@0.5:0.95, respectively, demonstrating excellent detection capabilities. This paper creates a complex scene dataset of apples on trees and designs an improved model, which can provide accurate recognition and positioning for automatic harvesting robots to improve production efficiency. [ABSTRACT FROM AUTHOR]

Efficient and accurate detection of apples is critical for the successful implementation of harvesting robots in orchards. However, due to limited memory resources on robotic platforms, it is imperative to develop lightweight detection algorithms that can operate in real-time. To address this challenge, we propose an ultralight convolutional neural network, U-DPnet, based on depth-separable convolution. Our approach incorporates the cross-stage deep separable module (CDM) and the multi-cascade deep separable module (MDM) in the backbone for nonlinear unit addition and attention mechanisms, which reduce the volume of the network while improving the feature representation capability. A simplified bi-directional feature pyramid network (BiFPN) is constructed in the neck for multi-scale feature fusion, and Adaptive feature propagation (AFP) is designed between the neck and the backbone for smooth feature transitions across different scales. To further reduce the network volume, we develop a uniform channel downsampling and network weight-sharing strategy. Multiple loss functions and label assignment strategies are used to optimize the training process. The performance of U-DPnet is verified on a homemade Apple dataset. Experimental results demonstrate that U-DPnet achieves detection accuracy and speed comparable to that of the 7 SOTA models. Moreover, U-DPnet exhibits an absolute advantage in model volume and computations (only 1.067M Params and 0.563G FLOPs, 39.79% and 36.36% less than yolov5-n). [ABSTRACT FROM AUTHOR]

Automated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments that involve varying lighting conditions, fruit clustering and foliage/branch occlusions. Apples are often grown in clusters on trees, which may be mis-identified as a single apple and thus causes issues in fruit localization for subsequent robotic harvesting operations. In this paper, we present the development of a novel deep learning-based apple detection framework, called the Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in clustered situations. A comprehensive dataset of RGB images were collected for two varieties of apples under different lighting conditions (overcast, direct lighting, and back lighting) with varying degrees of apple occlusions, and the images were annotated and made available to the public. A novel occlusion-aware network was developed for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations of the developed O2RNet were performed using the collected images, which outperformed 12 other state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection. O2RNet provides an enhanced method for robust detection of clustered apples, which is critical to accurate fruit localization for robotic harvesting.

Considering precision agriculture, recent technological developments have sparked the emergence of several new tools that can help to automate the agricultural process. For instance, accurately detecting and counting apples in orchards is essential for maximizing harvests and ensuring effective resource management. However, there are several intrinsic difficulties with traditional techniques for identifying and counting apples in orchards. To identify, recognize, and detect apples, apple target detection algorithms, such as YOLOv7, have shown a great deal of reflection and accuracy. But occlusions, electrical wiring, branches, and overlapping pose severe issues for precisely detecting apples. Thus, to overcome these issues and accurately recognize apples and find the depth of apples from drone-based videos in complicated backdrops, our proposed model combines a multi-head attention system with the YOLOv7 object identification framework. Furthermore, we provide the ByteTrack method for apple counting in real time, which guarantees effective monitoring of apples. To verify the efficacy of our suggested model, a thorough comparison assessment is performed with several current apple detection and counting techniques. The outcomes adequately proved the effectiveness of our strategy, which continuously surpassed competing methods to achieve exceptional accuracies of 0.92, 0.96, and 0.95 with respect to precision, recall, and F1 score, and a low MAPE of 0.027, respectively.

Horticulture and agriculture are considered as the important pillars of any economy. Current technological advancements have led to the development of several new technologies which are useful in atomizing the agriculture process. Apple farming has a significant role in Italy's agriculture domain where manual labor is widely employed for apple picking which can be replaced by automated robot mechanisms. However, these mechanisms are based on computer vision methods. These methods focus on detection, localization and tracking the apple fruits in given video frames. Later, appropriate actions can be taken to enhance the production and harvesting. Several techniques have been presented for apple detection, but complex background, noise and image blurriness are the major causes which can deteriorate the performance of the system. Thus, in this work, we present a deep learning-based scheme to detect apples which uses Yolov5 architecture in live apple farm images. We further improve the Yolov5 architecture by incorporating an adaptive pooling scheme and attribute augmentation model. This model detects the smaller objects and improves the feature quality to detect the apples in complex backgrounds. Moreover, a loss function is also incorporated to obtain the accurate bounding box which helps to maximize the detection accuracy. The comparative study shows that the proposed approach with the improved Yolov5 architecture achieves overall accuracy of 0.97, 0.99, and 0.98 in terms of precision, recall, and F1-score, respectively. [ABSTRACT FROM AUTHOR]

Automatic identification picking robot is an important research content of agricultural modernization development. In order to overcome the difficulty of picking robots for accurate visual inspection and positioning of apples in a complex orchard, a detection method based on an instance segmentation model is proposed. To reduce the number of model parameters and improve the detection speed, the backbone feature extraction network is replaced from the Resnet101 network to the lightweight GhostNet network. Spatial Pyramid Pooling (SPP) module is used to increase the receptive field to enhance the semantics of the output network. Compared with Resnet101, the parameter quantity of the model is reduced by 90.90%, the detection speed is increased from 5 frames/s to 10 frames/s, and the detection speed is increased by 100%. The detection result is that the accuracy rate is 91.67%, the recall rate is 97.82%, and the mAP value is 91.68%. To solve the repeated detection of fruits due to the movement of the camera, the Deepsort algorithms was used to solve the multi-tracking problems. Experiments show that the algorithm can effectively detect the edge position information and categories of apples in different scenes. It can be an automated apple-picking robot. The vision system provides strong technical support. [ABSTRACT FROM AUTHOR]

In this paper, 317 literature in the Web of Science (WoS) related to research on apple by near‐infrared spectroscopy (NIRS) were drawn on the knowledge map of the number of literature, the co‐occurrence network of authors and institutions, the co‐occurrence and clustering of keywords based on CiteSpace. And a related analysis was carried out. Combined with the results of visual analysis and related literature, the research hotspots were sorted out and discussed. This paper provides a certain reference for relevant researchers to study in this field and provides a new method for macroscopically grasping the current status of apple quality detection research, which helps new researchers to quickly integrate into this field and obtain more valuable scientific information. [ABSTRACT FROM AUTHOR]

Apple harvesting is a highly seasonal and labor-intensive activity in modern agriculture. Fortunately, a harvesting robot is of great significance to improve the productivity and quality of apples, further alleviating the labor shortage in orchards. Among them, the detection model of the harvesting robot is also required to accurately and rapidly detect the target apples in the complex and changing orchard environment. It is a high demand for the small size to be deployed in the embedded device. This study aims to improve the speed and comprehensive performance of apple detection in a complex orchard environment. A Lightweight apple detection YOLOX-Tiny Network (Lad-YXNet) model was proposed to reduce the size of the original model. Some images of “Yanfu” and “Micui” apples were obtained during the apple harvest season in 2021. The images were uniformly clipped to the 1024×1024 pixels. As such, 1 200 images were selected to make the dataset, including the fruits with shaded branches and leaves, fruit clusters, varying degrees of illumination, blurred motion, and high density. This model was then used to optimize the topology of the single-stage detection network YOLOX-Tiny. Two lightweight visual attention modules were added to the model, including Efficient Channel Attention (ECA), and Shuffle Attention (SA). The Shuffle attention and double convolution layer (SDCLayer) was constructed to extract the background and fruit features. Swish and Leaky Rectified Linear Unit (Leaky-ReLU) was identified as the activation functions for the backbone and feature fusion network. A series of ablation experiments were carried out to evaluate the effectiveness of Mosaic enhancement in the model training. The average precision of the Lad-YXNet model decreased by 0.89 percent and 3.81 percent, respectively, after removing random image flipping and random image length width distortion. The F1-socre also decreased by 0.91 percent and 1.95 percent, respectively, where the precision decreased by 2.21 percent and 2.99 percent, respectively. There was a similar regularity of the YOLOX-Tiny model. After removing the image random combination, the average precision of the Lad-YXNet and the YOLOX-Tiny model decreased by 0.56 percent and 0.07 percent, the F1-socre decreased by 0.68 percent and 1.15 percent, as well as the recall rate decreased by 2.35 percent and 4.49 percent, respectively. The results showed that the random distortion of image length and width greatly contributed to the performance of model detection. But the random color gamut transformation of the image decreased the performance of model detection, due to the change of apple color in the training set. Two specific tests were conducted to explore the effectiveness of visual attention mechanisms in convolution networks. Specifically, one was to remove the visual attention modules from the Lad-YXNet, and another was to exchange the position of visual attention modules from the Lad-YXNet. Compared with the Lad-YXNet, the precision of the improved model to exchange the position of the visual attention modules only increased by 0.04 percent, while the recall, F1-socre, and average precision decreased by 0.78 percent, 0.39 percent, and 0.13 percent, respectively. The precision, recall, F1-socre, and average precision of the models without the attention module were reduced by 1.15 percent, 0.64 percent, 0.89 percent, and 0.46 percent, respectively, compared with the Lad-YXNet. Consequently, the SA and ECA enhanced the ability of the Lad-YXNet to extract the apple features, in order to improve the comprehensive detection accuracy of the model. The main feature maps of Lad-YXNet's backbone, feature fusion, and detection network were extracted by the feature visualization technology. A systematic investigation was made to determine the process of detecting apples with the Lad-YXNet in the complex natural environment, particularly from the point of feature extraction. As such, improved interpretability was achieved in the apple detection with the Lad-YXNet model. The Lad-YXNet was trained to be an average accuracy of 94.88% in the test set, which was 3.10 percent, 2.02 percent, 2.00 percent, and 0.51 percent higher than SSD, YOLOV4-Tiny, YOLOV5-Lite, and YOLOX-Tiny models, respectively. The detection time of an image was achieved in 10.06 ms with a model size of 16.6 MB, which was 20.03% and 18.23% less than YOLOX-Tiny, respectively. Therefore, the Lad-YXNet was well balanced with the size, precision, and speed of the apple detection model. The finding can provide a theoretical basis to accurately and quickly detect the apples for the harvesting robot in the complex orchard environment. [ABSTRACT FROM AUTHOR]

The apple industry, recognized as a pivotal sector in agriculture, increasingly emphasizes the mechanization and intelligent advancement of picking technology. This study innovatively applies a mist simulation algorithm to apple image generation, constructing a dataset of apple images under mixed sunny, cloudy, drizzling and foggy weather conditions called DNE-APPLE. It introduces a lightweight and efficient target detection network called DNE-YOLO. Building upon the YOLOv8 base model, DNE-YOLO incorporates the CBAM attention mechanism and CARAFE up-sampling operator to enhance the focus on apples. Additionally, it utilizes GSConv and the dynamic non-monotonic focusing mechanism loss function WIOU to reduce model parameters and decrease reliance on dataset quality. Extensive experimental results underscore the efficacy of the DNE-YOLO model, which achieves a detection accuracy (precision) of 90.7%, a recall of 88.9%, a mean accuracy (mAP50) of 94.3%, a computational complexity (GFLOPs) of 25.4G, and a parameter count of 10.46M across various environmentally diverse datasets. Compared to YOLOv8, it exhibits superior detection accuracy and robustness in sunny, drizzly, cloudy, and misty environments, making it especially suitable for practical applications such as apple picking for agricultural robots. The code for this model is open source at https://github.com/wuhaitao2178827/DNE-YOLO. [ABSTRACT FROM AUTHOR]

A fast and accurate detection is one of the most important prerequisites for the apple harvest robots. However, there are many factors that can make apple detection difficult in a real orchard scene, such as complex backgrounds, fruit overlap, and leaf/branch occlusion. In this study, a fast and stable network was proposed for apple detection using an improved RetinaNet. A picking strategy was also developed for the harvest robot. Specifically, once the apples occluded by branches/wires were regarded as the picking targets, the robot arm would be injured at the same time. Therefore, the apples were labeled with multiple classes, according to different types of occlusions. The Res2Net module was also embedded in the ResNet50, in order to improve the ability of the backbone network to extract the multi-scale features. Furthermore, the BiFPN instead of FPN was used as a feature fusion network in the neck of the network. A weight fusion of feature maps was also made at different scales for the apples with different sizes, thus improving the detection accuracy of the network. After that, a loss function was combined with the Focal loss and Efficient Intersection over Union (EIoU) loss. Among them, Focal loss was used for the classification loss function, further reducing the errors from the imbalance of positive and negative sample ratios. By contrast, the EIoU loss was used for the regression loss function of the bounding box, in order to maintain a fast and accurate regression. Particularly, there were some different relative positions in the prediction and the ground truth box, such as overlap, disjoint and inclusion. Finally, the classification and regression were carried out on the feature map of five scales to realize a better detection of apple. In addition, the original dataset consisted of 800 apple images with complex backgrounds of dense orchards. A data enhancement was conducted to promote the generalization ability of the model. The dataset was then expanded to 4 800 images after operations, such as rotating, adjusting brightness, and adding noise. There was also a balance between the detection accuracy and speed. A series of experimental statistics were obtained on the number of BiFPN stacks in the network. Specifically, the BiFPN was stacked five times in the improved RetinaNet. The ablation experiments showed that each improvement of the model enhanced the accuracy of the network for the apple detection, compared with the original. The average precision of the improved RetinaNet reached 94.02%, 86.74%, 89.42%, and 94.84% for the leaf occlusion, branch/wire occlusion, fruit occlusion, and no occlusion apples, respectively. The mean Average Precision (mAP) reached 91.26%, which was 5.02 percentage points higher than that of the traditional RetinaNet. The improved RetinaNet took only 42.72 ms to process an apple image on average. Correspondingly, each fruit picking cycle was 2 780 ms, indicating that the detection speed fully met the harsh requirement of the picking robot. Only when the apples were large or rarely occluded, both improved and traditional RetinaNet were used to accurately detect them. By contrast, the improved RetinaNet performed the best to detect all apple fruits, when the apples were under a complex environment in an orchard, such as the leaf-, fruit-, or branch/wire-occluded background. The reason was that the traditional RetinaNet often appeared to miss the detection in this case. Consequently, the best comprehensive performance was achieved to verify the effectiveness of the improvements, compared with the state-of-the-art detection network, such as the Faster RCNN and YOLOv4. Overall, all the apples in the different classes can be effectively detected for the apple harvest. The finding can greatly contribute to the picking strategy of the robot, further avoiding the potential damage by the branches and wires during harvesting. [ABSTRACT FROM AUTHOR]

In order to enable the picking robot to detect and locate apples quickly and accurately in the orchard natural environment, we propose an apple object detection method based on Shufflenetv2-YOLOX. This method takes YOLOX-Tiny as the baseline and uses the lightweight network Shufflenetv2 added with the convolutional block attention module (CBAM) as the backbone. An adaptive spatial feature fusion (ASFF) module is added to the PANet network to improve the detection accuracy, and only two extraction layers are used to simplify the network structure. The average precision (AP), precision, recall, and F1 of the trained network under the verification set are 96.76%, 95.62%, 93.75%, and 0.95, respectively, and the detection speed reaches 65 frames per second (FPS). The test results show that the AP value of Shufflenetv2-YOLOX is increased by 6.24% compared with YOLOX-Tiny, and the detection speed is increased by 18%. At the same time, it has a better detection effect and speed than the advanced lightweight networks YOLOv5-s, Efficientdet-d0, YOLOv4-Tiny, and Mobilenet-YOLOv4-Lite. Meanwhile, the half-precision floating-point (FP16) accuracy model on the embedded device Jetson Nano with TensorRT acceleration can reach 26.3 FPS. This method can provide an effective solution for the vision system of the apple picking robot. [ABSTRACT FROM AUTHOR]

• We developed a fast apple object detection model for edge detection devices in orchard. • Our newly proposed PDWConv demonstrated superior computational efficiency. • Introduced PDWFasterNet and DWSConv greatly reduced the model's computation. • Introduced EIoU loss was effective to improve the model's accuracy. • Our Faster-YOLO-AP model achieved exceptional speed without compromising on accuracy. Object detection is a critical technology for apple harvesting robots. For efficient deployment on edge computing devices with limited processing power, we proposed a new lightweight algorithm for apple detection named Faster-YOLO-AP based on YOLOv8. First, by adjusting the network scaling factors of YOLOv8, we created an even smaller scale network named YOLOv8pico (YOLOv8p). Second, we proposed a partial depthwise convolution (PDWConv) with less computation and used it to build the PDWFasterNet module, which replaced YOLOv8p's original C2F feature extraction module. Third, to further lighten the network, we replaced the regular convolution with the lightweight depthwise separable convolution (DWSConv) for downsampling. Last, to offset the impact of these lightweight improvements on detection accuracy, we introduced the EIoU loss for bounding box regression. The results showed that the parameters and floating-point operations (FLOPs) of Faster-YOLO-AP were reduced to 0.66 M and 2.29 G, respectively, while achieving an mAP@.50:.95 of 84.12 % to our dataset. On edge computing devices, the Faster-YOLO-AP model demonstrated superior performance in both speed and accuracy compared with other lightweight models. The developed algorithm, distinguished by its lightweight and fast inference speed, provides a valuable reference for deploying real-time object detection on edge computing devices in apple harvesting robots. [ABSTRACT FROM AUTHOR]

Intelligent production and robotic oporation are the efficient and sustainable agronomic route to cut down economic and environmental costs and boosting orchard productivity. In the actual scene of the orchard, high performance visual perception system is the premise and key for accurate and reliable operation of the automatic cultivation platform. Most of the existing apple detection models, however, are difficult to be used on the platforms with limited hardware resources in terms of computing power and storage capacity due to too many parameters and large model volume. In order to improve the performance and adaptability of the existing apple detection model under the condition of limited hardware resources, while maintaining detection accuracy, reducing the calculation of the model and the model computing and storage footprint, shorten detection time, this method improved the lightweight MobileNetV3 and combined the object detection network which was based on keypoint prediction (CenterNet) to build a lightweight anchor-free model (M-CenterNet) for apple detection. The proposed model used heatmap to search the center point (keypotint) of the object, and predict whether each pixel was the center point of the apple, and the local offset of the keypoint and object size of the apple were estimated based on the extracted center point without the need for grouping or Non-Maximum Suppression (NMS). In view of its advantages in model volume and speed, improved MobileNetV3 which was equipped with transposed convolutional layers for the better semantic information and location information was used as the backbone of the network. Compared with CenterNet and SSD (Single Shot Multibox Detector), the comprehensive performance, detection accuracy, model capacity and running speed of the model were compared. The results showed that the average precision, error rate and miss rate of the proposed model were 88.9%, 10.9% and 5.8%, respectively, and its model volume and frame rate were 14.2MB and 8.1fps. The proposed model is of strong environmental adaptability and has a good detection effect under the circumstance of various light, different occlusion, different fruits’ distance and number. By comparing the performance of the accuracy with the CenterNet and the SSD models, the results showed that the proposed model was only 1/4 of the size of CenterNet model while has comparable detection accuracy. Compared with the SSD model, the average precision of the proposed model increased by 3.9%, and the model volume decreased by 84.3%. The proposed model runs almost twice as fast using CPU than the CenterNet and SSD models. This study provided a new approach for the research of lightweight model in fruit detection with orchard mobile platform under unstructured environment.

It is a challenging problem to detect the apple in natural environment using traditional object recognition algorithms due to occlusion, fluctuating illumination and complex backgrounds. Deep learning methods for object detection make impressive progress, which can automatically extract the number, pixel position, size and other features of apples from the images. In this paper, four deep learning recognition models, Faster RCNN based on AlexNet, Faster RCNN based on ResNet101, YOLOv3 based on DarkNet53 and improved YOLOv3 were employed to carry out recognition experiments on red and green apple under three illumination and two image sharpness conditions, with the transfer learning to accelerate the training process. The results showed that improved YOLOv3 model had the best recognition effect among the four detection models. F1 value of red apple recognition was 95.0%, 94.6% and 94.1% for normal, insufficient and excessive illumination, respectively, and F1 value of green apple recognition was 94.9%, 94.0% and 91.1%. There were F1 value of 92.8% and 92.1% for red and green apple recognition in blurred images, respectively. Moreover, improved YOLOv3 algorithm still had the better performance for occlusion, spot, overlap and incomplete apples, with a recognition recall rate higher than 88.5%. It can be concluded that improved YOLOv3 algorithm can provide a more efficient way for apple detection in natural environment.

In order to enable the picking robot to detect and locate apples quickly and accurately in the orchard natural environment, we propose an apple object detection method based on Shufflenetv2-YOLOX. This method takes YOLOX-Tiny as the baseline and uses the lightweight network Shufflenetv2 added with the convolutional block attention module (CBAM) as the backbone. An adaptive spatial feature fusion (ASFF) module is added to the PANet network to improve the detection accuracy, and only two extraction layers are used to simplify the network structure. The average precision (AP), precision, recall, and F1 of the trained network under the verification set are 96.76%, 95.62%, 93.75%, and 0.95, respectively, and the detection speed reaches 65 frames per second (FPS). The test results show that the AP value of Shufflenetv2-YOLOX is increased by 6.24% compared with YOLOX-Tiny, and the detection speed is increased by 18%. At the same time, it has a better detection effect and speed than the advanced lightweight networks YOLOv5-s, Efficientdet-d0, YOLOv4-Tiny, and Mobilenet-YOLOv4-Lite. Meanwhile, the half-precision floating-point (FP16) accuracy model on the embedded device Jetson Nano with TensorRT acceleration can reach 26.3 FPS. This method can provide an effective solution for the vision system of the apple picking robot.

Apple is one of the main producing fruits and the main economic crops in many areas. Detecting and grading apples through the image of apple trees under natural environment is helpful to promote the modernization of fruit industry. Combining deep learning with traditional methods, this paper proposes a fruit detection and grading method combining color and apple diameter. In order to improve the detection rate of unobvious targets and the precision of bounding boxes when illumination or fruit coloration is uneven, the convolutional neural network is used to construct an apple detection model and detect apple on feature maps of two scales, b货，(1. 8b-- L*) color components of the image in bounding boxes in CIELAB color space are extracted, the image is binarized, and the target contour is accurately extracted to correct the bounding boxes. Experimental results show that the precision is 91. 60% and the Fl-score value is 87. 62%. According to the image and actual size mapping method, the apple diameter is calculated to achieve the apple grading. Experimental results show that the grading accuracy is 90%. [ABSTRACT FROM AUTHOR]

To enable the apple picking robot to quickly and accurately detect apples under the complex background in orchards, we propose an improved You Only Look Once version 4 (YOLOv4) model and data augmentation methods. Firstly, the crawler technology is utilized to collect pertinent apple images from the Internet for labeling. For the problem of insufficient image data caused by the random occlusion between leaves, in addition to traditional data augmentation techniques, a leaf illustration data augmentation method is proposed in this paper to accomplish data augmentation. Secondly, due to the large size and calculation of the YOLOv4 model, the backbone network Cross Stage Partial Darknet53 (CSPDarknet53) of the YOLOv4 model is replaced by EfficientNet, and convolution layer (Conv2D) is added to the three outputs to further adjust and extract the features, which make the model lighter and reduce the computational complexity. Finally, the apple detection experiment is performed on 2670 expanded samples. The test results show that the EfficientNet-B0-YOLOv4 model proposed in this paper has better detection performance than YOLOv3, YOLOv4, and Faster R-CNN with ResNet, which are state-of-the-art apple detection model. The average values of Recall, Precision, and F1 reach 97.43%, 95.52%, and 96.54% respectively, the average detection time per frame of the model is 0.338 s, which proves that the proposed method can be well applied in the vision system of picking robots in the apple industry.

Since most computer vision approaches are now driven by machine learning, the current bottleneck is the annotation of images. This time-consuming task is usually performed manually after the acquisition of images. In this article, we assess the value of various egocentric vision approaches in regard to performing joint acquisition and automatic image annotation rather than the conventional two-step process of acquisition followed by manual annotation. This approach is illustrated with apple detection in challenging field conditions. We demonstrate the possibility of high performance in automatic apple segmentation (Dice 0.85), apple counting (88 percent of probability of good detection, and 0.09 true-negative rate), and apple localization (a shift error of fewer than 3 pixels) with eye-tracking systems. This is obtained by simply applying the areas of interest captured by the egocentric devices to standard, non-supervised image segmentation. We especially stress the importance in terms of time of using such eye-tracking devices on head-mounted systems to jointly perform image acquisition and automatic annotation. A gain of time of over 10-fold by comparison with classical image acquisition followed by manual image annotation is demonstrated.

Accurate apple recognition is a vital step in the operation of robotic fruit picking. To improve robot recognition ability and perception in three-dimensional (3D) space, an automatic recognition method was proposed to achieve apple recognition from point cloud data. First, an improved 3D descriptor (Color-FPFH) with the fusion of color features and 3D geometry features was extracted from the preprocessed point clouds. Then, a classification category was subdivided into apple, branch, and leaf to provide the system with a more comprehensive perception capability. A classifier based on the support vector machine, optimized using a genetic algorithm, was trained by the three data classes. Finally, the results of recognition and lateral comparison were obtained by comparison with the different 3D descriptors and other classic classifiers. The results showed that the proposed method exhibited better performance. In addition, the feasibility of estimating the occurrence of blocking using proposed method was discussed. [ABSTRACT FROM AUTHOR]

To enhance the working efficiency of apple picking systems by improving the accuracy of object detection in natural scenes, an improved apple detection network based on YOLOX-s is proposed. The self-attention residual module is added to the last layer of the improved YOLOX-s, which is used to add global feature information for small apples. An additional object detection head is added in YOLOX-s to strengthen the feature information of dense objects. Extra convolutional branches are added into the spatial pyramid pooling structure to enhance feature fusion and compensate for lost object location information. The loss function of the network is replaced by the α-CIoU loss, which is used to improve the bbox regression accuracy by up-weighting the loss and gradient of the high IoU prediction box. Experiment results show that the mAP50 value and the recall rate of the improved network reached 91.8% and 97%, respectively. Therefore, the improved network is superior to the existing detection networks in detection accuracy. Meanwhile, the detection time increased slightly, but it still meets the requirements of real-time detection of the picking system. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Agricultural Science & Technology (1008-0864) is the property of Journal of Agricultural Science & Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

• An integrated robotic apple harvesting prototype is presented. • An optimized picking pattern and an anthropomorphic picking pattern were used in the field picking evaluation of the robotic apple harvester. • The success rates using two picking motions were 80.17% and 82.93%, respectively. • Stem-pulled or bruised apples did not appear during picking evaluation using either motion pattern. A robotic apple harvester consisting of a mobile platform, a manipulator, an end-effector, a stereo camera, and a host computer was constructed and evaluated using two picking motions. The field tests showed all apple picking with success rates of 80.17% and 82.93% when using anthropomorphic and "horizontal pull with bending" motions, respectively. The main reasons for picking failure were depth misalignment, detachment failure, and blocked grasp. The "horizontal pull with bending" and anthropomorphic motions took 1.14 s and 3.13 s, respectively. The full picking cycle process using "horizontal pull with bending" motion was 12.53 ± 0.53 s, 4.64 s less than the average picking time when using anthropomorphic picking motion (17.17 ± 0.36 s). The picking process using anthropomorphic motion experienced a lower dynamic payload, meaning less effort would be required by the manipulator joints; however, fruit slipping decreased the overall success rate. The "horizontal pull with bending" picking motion had a superior picking cycle time and success rate. Notably, there were no stem-pulled or bruised apples during picking process using either motion. Based on this study, both picking motions have the potential to be applied in harvesting robots. [ABSTRACT FROM AUTHOR]

utomated harvesting requires accurate detection and recognition of the fruit within a tree canopy in real-time in uncontrolled environments. However, occlusion, variable illumination, variable appearance and texture make this task a complex challenge. Our research discusses the development of a machine vision system, capable of recognizing occluded green apples within a tree canopy. This involves the detection of 'green' apples within scenes of 'green leaves', shadow patterns, branches and other objects found in natural tree canopies. The system uses both thermal infra-red and color image modalities in order to achieve improved performance. Maximization of mutual information is used to find the optimal registration parameters between images from the two modalities. We use two approaches for apple detection based on low and high-level visual features. High-level features are global attributes captured by image processing operations, while low-level features are strong responses to primitive parts-based filters (such as Haar wavelets). These features are then applied separately to color and thermal infra-red images to detect apples from the background. These two approaches are compared and it is shown that the low-level feature-based approach is superior (74% recognition accuracy) over the high-level visual feature approach (53.16% recognition accuracy). Finally, a voting scheme is used to improve the detection results, which drops the false alarms with little effect on the recognition rate. The resulting classifiers acting independently can partially recognize the on-tree apples, however, when combined the recognition accuracy is increased. [ABSTRACT FROM AUTHOR]

• A fruit counting method based on video of orchard to estimate yield is studied. • Mean average precision of fruit and trunk based on YOLOv4-tiny was 99.35%. • Detected trunk is tracked to obtain reference displacement in consecutive frames. • Matching same fruits by the minimum Euclidean distance to assign them unique ID. • Proposed method can be implemented on CPU at 2–5 fps with accuracy of 91.49%. Accurate count of fruits is important for producers to make adequate decisions in production management. Although some algorithms based on machine vision have been developed to count fruits which were all implemented by tracking fruits themselves, those algorithms often make mismatches or even lose targets during the tracking process due to the large number of highly similar fruits in appearance. This study aims to develop an automated video processing method for improving the counting accuracy of apple fruits in orchard environment with modern vertical fruiting-wall architecture. As the trunk is normally larger than fruits and appears clearly in the video, the trunk is thus selected as a single-object tracking target to reach a higher accuracy and higher speed tracking than the commonly used method of fruit-based multi-object tracking. This method was trained using a YOLOv4-tiny network integrated with a CSR-DCF (channel spatial reliability-discriminative correlation filter) algorithm. Reference displacement between consecutive frames was calculated according to the frame motion trajectory for predicting possible fruit locations in terms of previously detected positions. The minimum Euclidean distance of detected fruit position and the predicted fruit position was calculated to match the same fruits between consecutive video frames. Finally, a unique ID was assigned to each fruit for counting. Results showed that mean average precision of 99.35% for fruit and trunk detection was achieved in this study, which could provide a good basis for fruit accurate counting. A counting accuracy of 91.49% and a correlation coefficient R2 of 0.9875 with counting performed by manual counting were reached in orchard videos. Besides, proposed counting method can be implemented on CPU at 2 ∼ 5 frames per second (fps). These promising results demonstrate the potential of this method to provide yield data for apple fruits or even other types of fruits. [ABSTRACT FROM AUTHOR]

To enable the apple picking robot to quickly and accurately detect apples under the complex background in orchards, we propose an improved You Only Look Once version 4 (YOLOv4) model and data augmentation methods. Firstly, the crawler technology is utilized to collect pertinent apple images from the Internet for labeling. For the problem of insufficient image data caused by the random occlusion between leaves, in addition to traditional data augmentation techniques, a leaf illustration data augmentation method is proposed in this paper to accomplish data augmentation. Secondly, due to the large size and calculation of the YOLOv4 model, the backbone network Cross Stage Partial Darknet53 (CSPDarknet53) of the YOLOv4 model is replaced by EfficientNet, and convolution layer (Conv2D) is added to the three outputs to further adjust and extract the features, which make the model lighter and reduce the computational complexity. Finally, the apple detection experiment is performed on 2670 expanded samples. The test results show that the EfficientNet-B0-YOLOv4 model proposed in this paper has better detection performance than YOLOv3, YOLOv4, and Faster R-CNN with ResNet, which are state-of-the-art apple detection model. The average values of Recall, Precision, and F1 reach 97.43%, 95.52%, and 96.54% respectively, the average detection time per frame of the model is 0.338 s, which proves that the proposed method can be well applied in the vision system of picking robots in the apple industry. [ABSTRACT FROM AUTHOR]