Search

ObjectiveThe fruit load estimation of fruit tree is essential for horticulture management. Traditional estimation method by manual sampling is not only labor-intensive and time-consuming but also prone to errors. Most existing models can not apply to edge computing equipment with limited computing resources because of their high model complexity. This study aims to develop a lightweight model for edge computing equipment to estimate fruit load automatically in the orchard.MethodsThe experimental data were captured using the smartphone in the citrus orchard in Jiangnan district, Nanning city, Guangxi province. In the dataset, 30 videos were randomly selected for model training and other 10 for testing. The general idea of the proposed algorithm was divided into two parts: Detecting fruits and extracting ReID features of fruits in each image from the video, then tracking fruit and estimating the fruit load. Specifically, the CSPDarknet53 network was used as the backbone of the model to achieve feature extraction as it consumes less hardware computing resources, which was suitable for edge computing equipment. The path aggregation feature pyramid network PAFPN was introduced as the neck part for the feature fusion via the jump connection between the low-level and high-level features. The fused features from the PAFPN were fed into two parallel branches. One was the fruit detection branch and another was the identity embedding branch. The fruit detection branch consisted of three prediction heads, each of which performed 3×3 convolution and 1×1 convolution on the feature map output by the PAFPN to predict the fruit's keypoint heat map, local offset and bounding box size, respectively. The identity embedding branch distinguished between different fruit identity features. In the fruit tracking stage, the byte mechanism from the ByteTrack algorithm was introduced to improve the data association of the FairMOT method, enhancing the performance of fruit load estimation in the video. The Byte algorithm considered both high-score and low-score detection boxes to associate the fruit motion trajectory, then matches the identity features' similarity of fruits between frames. The number of fruit IDs whose tracking duration longer than five frames was counted as the amount of citrus fruit in the video.Results and DiscussionsAll experiments were conducted on edge computing equipment. The fruit detection experiment was conducted under the same test dataset containing 211 citrus tree images. The experimental results showed that applying CSPDarkNet53+PAFPN structure in the proposed model achieved a precision of 83.6%, recall of 89.2% and F1 score of 86.3%, respectively, which were superior to the same indexes of FairMOT (ResNet34) model, FairMOT (HRNet18) model and Faster RCNN model. The CSPDarkNet53+PAFPN structure adopted in the proposed model could better detect the fruits in the images, laying a foundation for estimating the amount of citrus fruit on trees. The model complexity experimental results showed that the number of parameters, FLOPs (Floating Point Operations) and size of the proposed model were 5.01 M, 36.44 G and 70.2 MB, respectively. The number of parameters for the proposed model was 20.19% of FairMOT (ResNet34) model's and 41.51% of FairMOT (HRNet18) model's. The FLOPs for the proposed model was 78.31% less than FairMOT (ResNet34) model's and 87.63% less than FairMOT (HRNet18) model's. The model size for the proposed model was 23.96% of FairMOT (ResNet34) model's and 45.00% of FairMOT (HRNet18) model's. Compared with the Faster RCNN, the model built in this study showed advantages in the number of parameters, FLOPs and model size. The low complexity proved that the proposed model was more friendly to edge computing equipment. Compared with the lightweight backbone network EfficientNet-Lite, the CSPDarkNet53 applied in the proposed model's backbone performed better fruit detection and model complexity. For fruit load estimation, the improved tracking strategy that integrated the Byte algorithm into the FairMOT positively boosted the estimation accuracy of fruit load. The experimental results on the test videos showed that the AEP (Average Estimating Precision) and FPS (Frames Per Second) of the proposed model reached 91.61% and 14.76 f/s, which indicated that the proposed model could maintain high estimation accuracy while the FPS was 2.4 times and 4.7 times of the comparison models, respectively. The RMSE (Root Mean Square Error) of the proposed model was 4.1713, which was 47.61% less than FairMOT (ResNet34) model's and 22.94% less than FairMOT (HRNet18) model's. The R2 of the determination coefficient between the algorithm-measured value and the manual counted value was 0.9858, which was superior to other comparison models. The proposed model revealed better performance in estimating fruit load and lower model complexity than other comparatives.ConclusionsThe experimental results proved the validity of the proposed model for fruit load estimation on edge computing equipment. This research could provide technical references for the automatic monitoring and analysis of orchard productivity. Future research will continue to enrich the data resources, further improve the model's performance, and explore more efficient methods to serve more fruit tree varieties.

Accurate detection and recognition of plant diseases is the key technology to early diagnosis and intelligent monitoring of plant diseases, and is the core of accurate control and information management of plant diseases and insect pests. Deep learning can overcome the disadvantages of traditional diagnosis methods and greatly improve the accuracy of diseases detection and recognition, and has attracted a lot of attention of researchers. This paper collected the main public plant diseases image data sets all over the world, and briefly introduced the basic information of each data set and their websites, which is convenient to download and use. And then, the application of deep learning in plant disease detection and recognition in recent years was systematically reviewed. Plant disease target detection is the premise of accurate classification and recognition of plant disease and evaluation of disease hazard level. It is also the key to accurately locate plant disease area and guide spray device of plant protection equipment to spray drug on target. Plant disease recognition refers to the processing, analysis and understanding of disease images to identify different kinds of disease objects, which is the main basis for the timely and effective prevention and control of plant diseases. The research progress in early disease detection and recognition algorithm was expounded based on depth of learning research, as well as the advantages and existing problems of various algorithms were described. It can be seen from this review that the detection and recognition algorithm based on deep learning is superior to the traditional detection and recognition algorithm in all aspects. Based on the investigation of research results, it was pointed out that the illumination, sheltering, complex background, different disorders with similar symptoms, different changes of disease symptoms in different periods, and overlapping coexistence of multiple diseases were the main challenges for the detection and recognition of plant diseases. At the same time, the establishment of a large-scale and more complex data set that meets the specific research needs is also a difficulty that need to face together. And at further, we point out that the combination of the better performance of the neural network, large-scale data set and agriculture theoretical basis is a major trend of the development of the future. It is also pointed out that multimodal data can be used to identify early plant diseases, which is also one of the future development direction.

Intelligent production and robotic oporation are the efficient and sustainable agronomic route to cut down economic and environmental costs and boosting orchard productivity. In the actual scene of the orchard, high performance visual perception system is the premise and key for accurate and reliable operation of the automatic cultivation platform. Most of the existing apple detection models, however, are difficult to be used on the platforms with limited hardware resources in terms of computing power and storage capacity due to too many parameters and large model volume. In order to improve the performance and adaptability of the existing apple detection model under the condition of limited hardware resources, while maintaining detection accuracy, reducing the calculation of the model and the model computing and storage footprint, shorten detection time, this method improved the lightweight MobileNetV3 and combined the object detection network which was based on keypoint prediction (CenterNet) to build a lightweight anchor-free model (M-CenterNet) for apple detection. The proposed model used heatmap to search the center point (keypotint) of the object, and predict whether each pixel was the center point of the apple, and the local offset of the keypoint and object size of the apple were estimated based on the extracted center point without the need for grouping or Non-Maximum Suppression (NMS). In view of its advantages in model volume and speed, improved MobileNetV3 which was equipped with transposed convolutional layers for the better semantic information and location information was used as the backbone of the network. Compared with CenterNet and SSD (Single Shot Multibox Detector), the comprehensive performance, detection accuracy, model capacity and running speed of the model were compared. The results showed that the average precision, error rate and miss rate of the proposed model were 88.9%, 10.9% and 5.8%, respectively, and its model volume and frame rate were 14.2MB and 8.1fps. The proposed model is of strong environmental adaptability and has a good detection effect under the circumstance of various light, different occlusion, different fruits’ distance and number. By comparing the performance of the accuracy with the CenterNet and the SSD models, the results showed that the proposed model was only 1/4 of the size of CenterNet model while has comparable detection accuracy. Compared with the SSD model, the average precision of the proposed model increased by 3.9%, and the model volume decreased by 84.3%. The proposed model runs almost twice as fast using CPU than the CenterNet and SSD models. This study provided a new approach for the research of lightweight model in fruit detection with orchard mobile platform under unstructured environment.

The variation of facial appearance due to the viewpoint (/pose) degrades face recognition systems considerably, which is one of the bottlenecks in face recognition. One of the possible solutions is generating virtual frontal view from any given nonfrontal view to obtain a virtual gallery/probe face. Following this idea, this paper proposes a simple, but efficient, novel locally linear regression (LLR) method, which generates the virtual frontal view from a given nonfrontal face image. We first justify the basic assumption of the paper that there exists an approximate linear mapping between a nonfrontal face image and its frontal counterpart. Then, by formulating the estimation of the linear mapping as a prediction problem, we present the regression-based solution, i.e., globally linear regression. To improve the prediction accuracy in the case of coarse alignment, LLR is further proposed. In LLR, we first perform dense sampling in the nonfrontal face image to obtain many overlapped local patches. Then, the linear regression technique is applied to each small patch for the prediction of its virtual frontal patch. Through the combination of all these patches, the virtual frontal view is generated. The experimental results on the CMU PIE database show distinct advantage of the proposed method over Eigen light-field method.