Search

ObjectiveCurrently, pig farming facilities mainly rely on manual counting for tracking slaughtered and stored pigs. This is not only time-consuming and labor-intensive, but also prone to counting errors due to pig movement and potential cheating. As breeding operations expand, the periodic live asset inventories put significant strain on human, material and financial resources. Although methods based on electronic ear tags can assist in pig counting, these ear tags are easy to break and fall off in group housing environments. Most of the existing methods for counting pigs based on computer vision require capturing images from a top-down perspective, necessitating the installation of cameras above each hogpen or even the use of drones, resulting in high installation and maintenance costs. To address the above challenges faced in the group pig counting task, a high-efficiency and low-cost pig counting method was proposed based on improved instance segmentation algorithm and WeChat public platform.MethodsFirstly, a smartphone was used to collect pig image data in the area from a human view perspective, and each pig's outline in the image was annotated to establish a pig count dataset. The training set contains 606 images and the test set contains 65 images. Secondly, an efficient global attention module was proposed by improving convolutional block attention module (CBAM). The efficient global attention module first performed a dimension permutation operation on the input feature map to obtain the interaction between its channels and spatial dimensions. The permuted features were aggregated using global average pooling (GAP). One-dimensional convolution replaced the fully connected operation in CBAM, eliminating dimensionality reduction and significantly reducing the model's parameter number. This module was integrated into the YOLOv8 single-stage instance segmentation network to build the pig counting model YOLOv8x-Ours. By adding an efficient global attention module into each C2f layer of the YOLOv8 backbone network, the dimensional dependencies and feature information in the image could be extracted more effectively, thereby achieving high-accuracy pig counting. Lastly, with a focus on user experience and outreach, a pig counting WeChat mini program was developed based on the WeChat public platform and Django Web framework. The counting model was deployed to count pigs using images captured by smartphones.Results and DiscussionsCompared with existing methods of Mask R-CNN, YOLACT(Real-time Instance Segmentation), PolarMask, SOLO and YOLOv5x, the proposed pig counting model YOLOv8x-Ours exhibited superior performance in terms of accuracy and stability. Notably, YOLOv8x-Ours achieved the highest accuracy in counting, with errors of less than 2 and 3 pigs on the test set. Specifically, 93.8% of the total test images had counting errors of less than 3 pigs. Compared with the two-stage instance segmentation algorithm Mask R-CNN and the YOLOv8x model that applies the CBAM attention mechanism, YOLOv8x-Ours showed performance improvements of 7.6% and 3%, respectively. And due to the single-stage design and anchor-free architecture of the YOLOv8 model, the processing speed of a single image was only 64 ms, 1/8 of Mask R-CNN. By embedding the model into the WeChat mini program platform, pig counting was conducted using smartphone images. In cases where the model incorrectly detected pigs, users were given the option to click on the erroneous location in the result image to adjust the statistical outcomes, thereby enhancing the accuracy of pig counting.ConclusionsThe feasibility of deep learning technology in the task of pig counting was demonstrated. The proposed method eliminates the need for installing hardware equipment in the breeding area of the pig farm, enabling pig counting to be carried out effortlessly using just a smartphone. Users can promptly spot any errors in the counting results through image segmentation visualization and easily rectify any inaccuracies. This collaborative human-machine model not only reduces the need for extensive manpower but also guarantees the precision and user-friendliness of the counting outcomes.

ObjectiveThe fruit load estimation of fruit tree is essential for horticulture management. Traditional estimation method by manual sampling is not only labor-intensive and time-consuming but also prone to errors. Most existing models can not apply to edge computing equipment with limited computing resources because of their high model complexity. This study aims to develop a lightweight model for edge computing equipment to estimate fruit load automatically in the orchard.MethodsThe experimental data were captured using the smartphone in the citrus orchard in Jiangnan district, Nanning city, Guangxi province. In the dataset, 30 videos were randomly selected for model training and other 10 for testing. The general idea of the proposed algorithm was divided into two parts: Detecting fruits and extracting ReID features of fruits in each image from the video, then tracking fruit and estimating the fruit load. Specifically, the CSPDarknet53 network was used as the backbone of the model to achieve feature extraction as it consumes less hardware computing resources, which was suitable for edge computing equipment. The path aggregation feature pyramid network PAFPN was introduced as the neck part for the feature fusion via the jump connection between the low-level and high-level features. The fused features from the PAFPN were fed into two parallel branches. One was the fruit detection branch and another was the identity embedding branch. The fruit detection branch consisted of three prediction heads, each of which performed 3×3 convolution and 1×1 convolution on the feature map output by the PAFPN to predict the fruit's keypoint heat map, local offset and bounding box size, respectively. The identity embedding branch distinguished between different fruit identity features. In the fruit tracking stage, the byte mechanism from the ByteTrack algorithm was introduced to improve the data association of the FairMOT method, enhancing the performance of fruit load estimation in the video. The Byte algorithm considered both high-score and low-score detection boxes to associate the fruit motion trajectory, then matches the identity features' similarity of fruits between frames. The number of fruit IDs whose tracking duration longer than five frames was counted as the amount of citrus fruit in the video.Results and DiscussionsAll experiments were conducted on edge computing equipment. The fruit detection experiment was conducted under the same test dataset containing 211 citrus tree images. The experimental results showed that applying CSPDarkNet53+PAFPN structure in the proposed model achieved a precision of 83.6%, recall of 89.2% and F1 score of 86.3%, respectively, which were superior to the same indexes of FairMOT (ResNet34) model, FairMOT (HRNet18) model and Faster RCNN model. The CSPDarkNet53+PAFPN structure adopted in the proposed model could better detect the fruits in the images, laying a foundation for estimating the amount of citrus fruit on trees. The model complexity experimental results showed that the number of parameters, FLOPs (Floating Point Operations) and size of the proposed model were 5.01 M, 36.44 G and 70.2 MB, respectively. The number of parameters for the proposed model was 20.19% of FairMOT (ResNet34) model's and 41.51% of FairMOT (HRNet18) model's. The FLOPs for the proposed model was 78.31% less than FairMOT (ResNet34) model's and 87.63% less than FairMOT (HRNet18) model's. The model size for the proposed model was 23.96% of FairMOT (ResNet34) model's and 45.00% of FairMOT (HRNet18) model's. Compared with the Faster RCNN, the model built in this study showed advantages in the number of parameters, FLOPs and model size. The low complexity proved that the proposed model was more friendly to edge computing equipment. Compared with the lightweight backbone network EfficientNet-Lite, the CSPDarkNet53 applied in the proposed model's backbone performed better fruit detection and model complexity. For fruit load estimation, the improved tracking strategy that integrated the Byte algorithm into the FairMOT positively boosted the estimation accuracy of fruit load. The experimental results on the test videos showed that the AEP (Average Estimating Precision) and FPS (Frames Per Second) of the proposed model reached 91.61% and 14.76 f/s, which indicated that the proposed model could maintain high estimation accuracy while the FPS was 2.4 times and 4.7 times of the comparison models, respectively. The RMSE (Root Mean Square Error) of the proposed model was 4.1713, which was 47.61% less than FairMOT (ResNet34) model's and 22.94% less than FairMOT (HRNet18) model's. The R2 of the determination coefficient between the algorithm-measured value and the manual counted value was 0.9858, which was superior to other comparison models. The proposed model revealed better performance in estimating fruit load and lower model complexity than other comparatives.ConclusionsThe experimental results proved the validity of the proposed model for fruit load estimation on edge computing equipment. This research could provide technical references for the automatic monitoring and analysis of orchard productivity. Future research will continue to enrich the data resources, further improve the model's performance, and explore more efficient methods to serve more fruit tree varieties.

Accurate detection and recognition of plant diseases is the key technology to early diagnosis and intelligent monitoring of plant diseases, and is the core of accurate control and information management of plant diseases and insect pests. Deep learning can overcome the disadvantages of traditional diagnosis methods and greatly improve the accuracy of diseases detection and recognition, and has attracted a lot of attention of researchers. This paper collected the main public plant diseases image data sets all over the world, and briefly introduced the basic information of each data set and their websites, which is convenient to download and use. And then, the application of deep learning in plant disease detection and recognition in recent years was systematically reviewed. Plant disease target detection is the premise of accurate classification and recognition of plant disease and evaluation of disease hazard level. It is also the key to accurately locate plant disease area and guide spray device of plant protection equipment to spray drug on target. Plant disease recognition refers to the processing, analysis and understanding of disease images to identify different kinds of disease objects, which is the main basis for the timely and effective prevention and control of plant diseases. The research progress in early disease detection and recognition algorithm was expounded based on depth of learning research, as well as the advantages and existing problems of various algorithms were described. It can be seen from this review that the detection and recognition algorithm based on deep learning is superior to the traditional detection and recognition algorithm in all aspects. Based on the investigation of research results, it was pointed out that the illumination, sheltering, complex background, different disorders with similar symptoms, different changes of disease symptoms in different periods, and overlapping coexistence of multiple diseases were the main challenges for the detection and recognition of plant diseases. At the same time, the establishment of a large-scale and more complex data set that meets the specific research needs is also a difficulty that need to face together. And at further, we point out that the combination of the better performance of the neural network, large-scale data set and agriculture theoretical basis is a major trend of the development of the future. It is also pointed out that multimodal data can be used to identify early plant diseases, which is also one of the future development direction.

Intelligent production and robotic oporation are the efficient and sustainable agronomic route to cut down economic and environmental costs and boosting orchard productivity. In the actual scene of the orchard, high performance visual perception system is the premise and key for accurate and reliable operation of the automatic cultivation platform. Most of the existing apple detection models, however, are difficult to be used on the platforms with limited hardware resources in terms of computing power and storage capacity due to too many parameters and large model volume. In order to improve the performance and adaptability of the existing apple detection model under the condition of limited hardware resources, while maintaining detection accuracy, reducing the calculation of the model and the model computing and storage footprint, shorten detection time, this method improved the lightweight MobileNetV3 and combined the object detection network which was based on keypoint prediction (CenterNet) to build a lightweight anchor-free model (M-CenterNet) for apple detection. The proposed model used heatmap to search the center point (keypotint) of the object, and predict whether each pixel was the center point of the apple, and the local offset of the keypoint and object size of the apple were estimated based on the extracted center point without the need for grouping or Non-Maximum Suppression (NMS). In view of its advantages in model volume and speed, improved MobileNetV3 which was equipped with transposed convolutional layers for the better semantic information and location information was used as the backbone of the network. Compared with CenterNet and SSD (Single Shot Multibox Detector), the comprehensive performance, detection accuracy, model capacity and running speed of the model were compared. The results showed that the average precision, error rate and miss rate of the proposed model were 88.9%, 10.9% and 5.8%, respectively, and its model volume and frame rate were 14.2MB and 8.1fps. The proposed model is of strong environmental adaptability and has a good detection effect under the circumstance of various light, different occlusion, different fruits’ distance and number. By comparing the performance of the accuracy with the CenterNet and the SSD models, the results showed that the proposed model was only 1/4 of the size of CenterNet model while has comparable detection accuracy. Compared with the SSD model, the average precision of the proposed model increased by 3.9%, and the model volume decreased by 84.3%. The proposed model runs almost twice as fast using CPU than the CenterNet and SSD models. This study provided a new approach for the research of lightweight model in fruit detection with orchard mobile platform under unstructured environment.

The North China type cucumber, characterized by its dense spines and top flowers, is susceptible to damage during the grading process, affecting its market value. Moreover, traditional manual grading methods are time-consuming and labor-intensive. To address these issues, this paper proposes a cucumber quality grader based on machine vision and deep learning. In the electromechanical aspect, a novel fixed tray type grading mechanism is designed to prevent damage to the vulnerable North China type cucumbers during the grading process. In the vision grading algorithm, a new convolutional neural network is introduced named MassNet, capable of predicting cucumber mass using only a top-view image. After obtaining the cucumber mass prediction, mass grading is achieved. Experimental validation includes assessing the electromechanical performance of the grader, comparing MassNet with different models in predicting cucumber mass, and evaluating the online grading performance of the integrated algorithm. Experimental results indicate that the designed cucumber quality grader achieves a maximum capacity of 2.3 t/hr. In comparison with AlexNet, MobileNet, and ResNet, MassNet demonstrates superior cucumber mass prediction, with a MAPE of 3.9% and RMSE of 6.7 g. In online mass grading experiments, the grading efficiency of the cucumber quality grader reaches 93%. [ABSTRACT FROM AUTHOR]

Veins are a critical component of the plant growth and development system, playing an integral role in supporting and protecting leaves, as well as transporting water, nutrients, and photosynthetic products. A comprehensive understanding of the form and function of veins requires a dual approach that combines plant physiology with cutting-edge image recognition technology. The latest advancements in computer vision and machine learning have facilitated the creation of algorithms that can identify vein networks and explore their developmental progression. Here, we review the functional, environmental, and genetic factors associated with vein networks, along with the current status of research on image analysis. In addition, we discuss the methods of venous phenotype extraction and multi-omics association analysis using machine learning technology, which could provide a theoretical basis for improving crop productivity by optimizing the vein network architecture. [ABSTRACT FROM AUTHOR]

Citrus harvesting is a labor‐intensive and time‐intensive task. As the global population continues to age, labor costs are increasing dramatically. Therefore, the citrus‐harvesting robot has attracted considerable attention from the business and academic communities. However, robotic harvesting in unstructured and natural citrus orchards remains a challenge. This study aims to address some challenges faced in commercializing citrus‐harvesting robots. We present a fully integrated, autonomous, and innovative solution for citrus‐harvesting robots to overcome the harvesting difficulties derived from the natural growth characteristics of citrus. This solution uses a fused simultaneous localization and mapping algorithm based on multiple sensors to perform high‐precision localization and navigation for the robot in the field orchard. Besides, a novel visual method for estimating fruit poses is proposed to cope with the randomization of citrus growth orientations. Further, a new end‐effector is designed to improve the success and conformity rate of citrus stem cutting. Finally, a fully autonomous harvesting robot system has been developed and integrated. Field evaluations showed that the robot could harvest citrus continuously with an overall success rate of 87.2% and an average picking time of 10.9 s/fruit. These efforts provide a solid foundation for the future commercialization of citrus‐harvesting robots. [ABSTRACT FROM AUTHOR]

In this paper, a Multi-scale Contrastive Graph Convolutional Network (MC-GCN) method is proposed for unconstrained face recognition with image sets, which takes a set of media (orderless images and videos) as a face subject instead of single media (an image or video). Due to factors such as illumination, posture, media source, etc., there are huge intra-set variances in a face set, and the importance of different face prototypes varies considerably. How to model the attention mechanism according to the relationship between prototypes or images in a set is the main content of this paper. In this work, we formulate a framework based on graph convolutional network (GCN), which considers face prototypes as nodes to build relations. Specifically, we first present a multi-scale graph module to learn the relationship between prototypes at multiple scales. Moreover, a Contrastive Graph Convolutional (CGC) block is introduced to build attention control model, which focuses on those frames with similar prototypes (contrastive information) between pair of sets instead of simply evaluating the frame quality. The experiments on IJB-A, YouTube Face, and an animal face dataset clearly demonstrate that our proposed MC-GCN outperforms the state-of-the-art methods significantly. [ABSTRACT FROM AUTHOR]

Visual sign language recognition is an interesting and challenging problem. To create a discriminative representation, a hierarchical Grassmann covariance matrix (HGCM) model is proposed for sign description. Furthermore, a multi-temporal belief propagation (MTBP) based segmentation approach is presented for continuous sequence spotting. Concretely speaking, a sign is represented by multiple covariance matrices, followed by evaluating and selecting their most significant singular vectors. These covariance matrices are transformed into a more compact and discriminative HGCM, which is formulated on the Grassmann manifold. Continuous sign sequences can be recognized frame by frame using the HGCM model, before being optimized by MTBP, which is a carefully designed graphic model. The proposed method is thoroughly evaluated on isolated and synthetic and real continuous sign datasets as well as on HDM05. Extensive experimental results convincingly show the effectiveness of our proposed framework. [ABSTRACT FROM AUTHOR]

Abstract Inspired by the concept of memory mechanism and predictive coding from the cognitive neuroscience, this paper presents a deep memory and prediction neural network (DMPNet) for video prediction. Correspondingly, memory and error propagation units are designed in DMPNet to capture the previous spatial-temporal information and compute current predictive error which is forwarded to the prediction unit for correcting the subsequent video prediction. Subsequently, prediction unit takes the information stored in memory unit and predictive error of previous frame as input to predict the next frame. We evaluate our method on two public real-world datasets and demonstrate that the proposed DMPNet outperforms some state-of-the-art methods quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

In this paper, we propose a method for robust Sign Language Recognition from RGB-D data. A Sparse Observation (SO) description is proposed to character each sign in terms of the typical hand postures. Concretely speaking, the SOs are generated by considering the typical posture fragments, where hand motions are relatively slow and hand shapes are stable. Thus the matching between two sign words is converted to measure the similarity computing between two aligned SO sequences. The alignment is formulated as a variation of Stable Marriage Problem (SMP). The classical “propose-engage” idea is extended to get the order preserving matched SO pairs. In the training stage, the multiple instances from one sign are fused to generate single SO template. In the recognition stage, SOs of each probe sign “propose” to SOs of the templates for the purpose of reasonable similarity computing. To further speed up the SO alignment, hand posture relationship map is constructed as a strong prior to generate the distinguished low-dimensional feature of SO. Moreover, to get much better performance, the motion trajectory feature is integrated. Experiments on two large datasets and an extra Chalearn Multi-modal Gesture Dataset demonstrate that our algorithm has much higher accuracy with only 1/10 time cost compared with the HMM and DTW based methods. [ABSTRACT FROM AUTHOR]

Exact yield estimation of fruits on plants guaranteed fine and timely decisions on harvesting and marketing practices. Automatic yield estimation based on unmanned agriculture offers a viable solution for large orchards. Recent years have witnessed notable progress in computer vision with deep learning for yield estimation. Yet, the current practice of vision-based yield estimation with successive frames may engender fairly great error because of the double counting of repeat fruits in different images. The goal of this study is to provide a wise framework for fruit yield estimation in sequence images. Specifically, the anchor-free detection architecture (CenterNet) is utilized to detect fruits in sequence images from videos collected in the apple orchard and orange orchard. In order to avoid double counts of a single fruit between different images in an image sequence, the patch matching model is designed with the Kuhn–Munkres algorithm to optimize the paring process of repeat fruits in a one-to-one assignment manner for the sound performance of fruit yield estimation. Experimental results show that the CenterNet model can successfully detect fruits, including apples and oranges, in sequence images and achieved a mean Average Precision (mAP) of 0.939 under an IoU of 0.5. The designed patch matching model obtained an F1-Score of 0.816 and 0.864 for both apples and oranges with good accuracy, precision, and recall, which outperforms the performance of the reference method. The proposed pipeline for the fruit yield estimation in the test image sequences agreed well with the ground truth, resulting in a squared correlation coefficient of R2apple = 0.9737 and R2orange = 0.9562, with a low Root Mean Square Error (RMSE) for these two varieties of fruit. [ABSTRACT FROM AUTHOR]

Abstract: The dissimilarity between a target descriptor and a particle descriptor is a crucial parameter in the particle filtering (PF), while the widely used Bhattacharyya dissimilarity (BD) is not discriminative enough. This paper presents an incremental Bhattacharyya dissimilarity (IBD) for measuring histogram based descriptors (HBDs) used for particle weight estimation. IBD is defined by incorporating an incremental similarity matrix (ISM) into the BD. Such an ISM imposes the incremental similarity beliefs on the matched bin patches of two input histograms and enables a cross-bin interaction during the comparison, which yields the enhanced capability of discriminating the particles located in the object from those positioned in the background. We propose a robust approach to compute the ISM by jointly utilizing the spatial and temporal attributes. Also, to handle target appearance changes and deformations, a classification-inspired target model update strategy is presented. These components lead to an effective and robust tracking algorithm. Experimental results demonstrate that IBD shows promising discriminative capability in comparison with other state of the art dissimilarity measures. Moreover, the IBD based PF-tracker also exhibits competitive tracking performance, especially under scenarios of partial occlusion and background clutter. [Copyright &y& Elsevier]

Due to the misalignment of image features, the performance of many conventional face recognition methods degrades considerably in across pose scenario. To address this problem, many image matching-based methods are proposed to estimate semantic correspondence between faces in different poses. In this paper, we aim to solve two critical problems in previous image matching-based correspondence learning methods: 1) fail to fully exploit face specific structure information in correspondence estimation and 2) fail to learn personalized correspondence for each probe image. To this end, we first build a model, termed as morphable displacement field (MDF), to encode face specific structure information of semantic correspondence from a set of real samples of correspondences calculated from 3D face models. Then, we propose a maximal likelihood correspondence estimation (MLCE) method to learn personalized correspondence based on maximal likelihood frontal face assumption. After obtaining the semantic correspondence encoded in the learned displacement, we can synthesize virtual frontal images of the profile faces for subsequent recognition. Using linear discriminant analysis method with pixel-intensity features, state-of-the-art performance is achieved on three multipose benchmarks, i.e., CMU-PIE, FERET, and MultiPIE databases. Owe to the rational MDF regularization and the usage of novel maximal likelihood objective, the proposed MLCE method can reliably learn correspondence between faces in different poses even in complex wild environment, i.e., labeled face in the wild database.

The variation of facial appearance due to the viewpoint (/pose) degrades face recognition systems considerably, which is one of the bottlenecks in face recognition. One of the possible solutions is generating virtual frontal view from any given nonfrontal view to obtain a virtual gallery/probe face. Following this idea, this paper proposes a simple, but efficient, novel locally linear regression (LLR) method, which generates the virtual frontal view from a given nonfrontal face image. We first justify the basic assumption of the paper that there exists an approximate linear mapping between a nonfrontal face image and its frontal counterpart. Then, by formulating the estimation of the linear mapping as a prediction problem, we present the regression-based solution, i.e., globally linear regression. To improve the prediction accuracy in the case of coarse alignment, LLR is further proposed. In LLR, we first perform dense sampling in the nonfrontal face image to obtain many overlapped local patches. Then, the linear regression technique is applied to each small patch for the prediction of its virtual frontal patch. Through the combination of all these patches, the virtual frontal view is generated. The experimental results on the CMU PIE database show distinct advantage of the proposed method over Eigen light-field method.