Search

Fruit picking has been one of the most important operations in modern agriculture. Manual picking cannot fully meet large-scale apple production in recent years, due to the labor-intensive and time-consuming task. Alternatively, robot picking can be expected to replace manual work. However, the conventional semi-automatic picking machine can seriously vibrate the branches and damage the fresh apples during batch harvesting in the existing orchard. It is a high demand to detect apples under complex environments, and then pick them without damaging the fruit or trees. The picking robots are also required a compatible auxiliary platform during operations in an orchard. The performance of picking robots can be assessed using the picking speed, picking success, and fruit damage rates. These parameters depend on three key technologies: target recognition, localization, and fruit separation. In this paper, an overview was provided of the history and current status of commercial apple-picking robots. Specifically, the performance and robustness were highlighted, in terms of the picking efficiency, structural composition, the type of orchard, and adaptability to the external environment. The main technical features were then summarized from the application of apple-picking robots. Since the apple picking was operated in an unstructured orchard environment, the most serious challenge remained in the application of target recognition, localization, and fruit separation techniques. Therefore, the recognition techniques were reviewed for the apples in the complex natural environments with changing light, branch and leaf shading, overlapping fruit, night-time environments, and same-colored apples. Deep learning-based apple recognition algorithms were the far-reaching prospects in complex environments, particularly in multiple scenarios. The accuracy and speed of apple recognition were improved using deep learning recognition algorithms in the field of apple picking. The second most challenge was the application of the apple localization for the fruit occlusion and overlap, while the fruit displacement by wind during the imaging process. Therefore, four aspects were introduced in the apple localization, including 2D information acquisition, depth information acquisition, apple pose acquisition and oscillating fruit localization. Point cloud processing algorithms were highlighted to achieve the precise localization of occluded apples. In addition, the applications of end-effectors were reviewed in the apple-picking robots with high efficiency and non-destructive picking. Three types of the end-effector structure (such as hand-jaw, vacuum, and shear) were summarized under fruit separation. It was found that the hand-jaw was the most commonly-used type of end-effector. Current research on the end-effectors was focused on the flexible materials for the finger surfaces, under-driven, flexible structures or control systems, such as the soft hands and perception-based servo control. Therefore, it is necessary to improve the picking speed and success rate with the low fruit damage and cost saving in the apple picking robot, in order to accelerate the process of apple picking automation. Future development was also proposed for the apple-picking robot in the urgent commercial need of the pioneering innovation in the combination of agricultural machinery and agronomy. Some optimization can be expected to be performed on the recognition algorithm, multi-sensor fusion, multi-arm cooperation, human-machine collaboration, extending device versatility, and integrating 5G and IoT technology. This finding can provide a strong reference for the current status, future development trends, and equipment innovation of apple-picking robots, together with the key technologies. [ABSTRACT FROM AUTHOR]

A method for inverse kinematics analysis based on screw theory was presented in this paper, which can directly map the position and orientation of the working object to the joint variables of the manipulator with its application to a full inverse kinematics analysis of forest-fruit harvesting manipulator characterized by a hybrid kinematic structure, 2P4R. The solution of inverse kinematics modeling derived by screw theory was commonly realized by Paden-Kahan sub-problem method, which decomposes a full kinematics problem into sub-problem with obviously geometrical meaning through choosing appropriate point, usually, intersection of adjacent joint, such as wrist joint, to reduce the number of the variable quantity, and then close-form solution can be easily obtained. However, in practice, it is hard to gain the position of those points through measure because of their absence before end-effector actually moving to the desired position. And few researchers mentioned this issue in the relevant literature. In order to discuss this problem, firstly, a geometrical method was proposed for this issue to obtain the position of the required point, wrist joint, according to the orientation of end-effector and its geometric properties and geometric relationships through using the vector algebra method. Furthermore, a mapping between driving and driven join was gained in order to simplify the solving process of the equation set at a later, according to the solution of the structural equation of the manipulator derived by the product-of-exponentials (POEs) formula and structural character of manipulator. Meanwhile, the closed-form solution for each driving joint variables was derived by employing the proposed method with Paden-Kahan sub-problem method. A mapping relationship between the plücker coordinates of the object and the location information of end-effector was derived through an algebraic method according to the principle of minimum displacement and its operating mode in which the gripper of end-effector should reach the position of the trunk with two labels detected by the robot vision system and be perpendicular to the orientation of the trunk. In addition, the problem of multiple solutions in the inverse kinematics analysis for the harvesting manipulator was solved according to the range of joint variables. Finally, the real-world experiment was performed under laboratory environment. In order to vertify the correctness and obtain the accuracy of the method proposed in this paper. A wooden stick with two markers was placed in the kinematics test platform as the object, which consisted of a laser tracker and a harvesting manipulator. Then, the values of each joint variable could be calculated via the proposed method according to the plücker coordinate data of the markers measured in the object. The results showed that the forest-fruit harvesting manipulator was driven by the solution of inverse kinematics to the position on the stick that its end-effect reached and normal to the stick, which meant this method could meet the requirements of the operating mode. Then ten sets of joint variable values were randomly generated where the positions were measured and the manipulator was sequentially driven by that. The joint variable values were calculated according to the positions through the method proposed in this paper. At last, the calculated results were re-inputted into the controller to drive the manipulator to the new positions. The two measure results on different positions driven by joint variable values generated and calculated were used to obtain the error. The results showed that the maximum position error of end-effector was 6.597 mm, far less than the open size of its gripper, 200 mm, and no more than 3.30%, with the maximum orientation error of 0.975°. The method in this paper was not limited by the specific structure, therefore it is versatile. [ABSTRACT FROM AUTHOR]

Image Jacobian matrix is core portion of robot vision servo system. During estimating image Jacobian matrix with traditional methodology of KF (Kalman Filter) and UKF (Unscented Kalman Filter), the system noises of unknown statistical properties is determined according to priori knowledge, which is remained unchanged in system. However, this methodology of dealing system noises would make image Jacobian matrix’s estimation inaccuracy because of time-varying system noises. Based on traditional methodology of KF and UKF, in this paper, we structured propagator of system noises estimation with the difference value of system state predicted value and updated value, and raised Adoption Unscented Kalman Filter to estimate image Jacobian matrix. Two points on the tomato string’s stem, clamping point and picking point, were chosen as feature points in the image feature space. The center of mass of fruit stem was extracted as the holding point. According to the vertical growth characteristics of tomato clusters and the distance between the holding point on the end and the cutting point on the image, a point above the holding point was extracted as the cutting point. Image Jacobian matrix was used to transform information of pixel difference in image feature space to change in pose of end-effector in manipulator motion space. First, results of image feature space localization experiment showed that six image feature errors were smaller and the average image feature error of visual servo system based on AUKF was 4.978 pixes/mm in condition of unknown system noises. Compared with visual servo system based on KF and UKF, average image feature error reduced 54.44% and 24.78%, respectively. Second, three dimensional space positioning experiment between end-effector and target picking point showed that the final distance between end-effector and target picking point was 4.19 mm, reduced by 79.74% and 80.38% compared to methods of KF and UKF. Third, the response time of visual servo control system based on KF method, UKF method and AUKF method to estimate image Jacobian matrix was 2, 1.5 and 1.2 s, respectively. The picking experiment was carried out on the visual servo test platform of the seven-degree-of-freedom picking manipulator. The visual system based on each visual servo control method was tested for 30 times. For the visual servo system based on AUKF method estimation, the success rate of picking the static target was 90%, and the success rate of picking the dynamic target was 83%. Compared with KF methodology and UKF methodology, the success rate of static picking test was increased by 17 and 10 percentage points, respectively, and the success rate of dynamic picking test was increased by 16 and 10 percentage points, respectively. The results showed that the static picking time based on AUKF methodology was the 18 s, compared with the visual servo system based on KF methodology and UKF methodology, the time of static picking was reduced by 35.71% and 25%, respectively. The average dynamic picking time of visual servo system based on AUKF was 22 s and the test time of dynamic picking was reduced by 35.29% and 26.67%, respectively compared with the visual servo system based on KF methodology and UKF methodology. The image Jacobian matrix based on AUKF was suitable for the dynamic picking of tomato clusters, and the accuracy of visual servo control under dynamic condition was lower than that under static condition. Test results suggested that the proposed AUKF method with KF method and the UKF method compared to estimating by image Jacobin matrix, and AUKF method to estimate the image Jacobin matrix, reduced the process of picking robot visual servo control system noise interference, which made the picking robot visual servo control system of processing noise adapted to the change of the visual servo system, resulting in higher positioning precision for picking robot visual servo control system . [ABSTRACT FROM AUTHOR]

It is difficult to locate the apple in fog and haze environment for apple harvest robot. This paper proposed a new method to apply the principle of DCP (dark channel prior) to remove fog and haze on images which were collected from apple orchard. We adopted a new route to achieve the value of atmospheric light coefficient. Scan the hazed image with a 15×15 window, and get the smallest value of the 225 pixels from every window. All the smallest values constituted a dark channel image. The values of dark channel were stored in a matrix at the first step, and then the 1/1000 largest elements and their locations were calculated and stored in a new matrix which had the same shape with the dark channel matrix. Extract the matrix of red (R) channel of the hazed image at the next step. At the third step, the corresponding values in the matrix of R channel were obtained according to the position information in the new matrix. Finally, the average value of these values was calculated as the value of atmospheric light coefficient. According to the requirement of apple harvest robot, we took the haze-removal strength parameter of DCP algorithm as 1. In order to speed up the running of the algorithm, we calculated the transmission radio with guided filter. Image segmentation method used in the study had 3 stages: binaryzation, de-noising and dilation. Firstly, the grey image was obtained by calculating a special linear combination with red (R) channel, green (G) channel and blue (B) channel. This method emphasized the value of red channel in color images, which was conducive to separate the apple better at the next step. Secondly, binary image was obtained by Otsu method based on the grey image. Finally, after the process of de-noising and dilation, a better segmentation results could be obtained. We developed an experimental software with Microsoft Vision Studio 2010 and OpenCV (Open Source Computer Vision Library) to test the haze-removal effects in apple harvest robot vision system. The graphical user interface of the program was developed based on MFC (Microsoft Foundation Classes) library. The software had achieved the following functions: reading the image, calculating the dark channel value, calculating the value of atmospheric light coefficient based on the R channel, calculating the transmission radio, and so on. Based on this software we compared some haze-removal methods including MSR (multiscale retinex), AHE (adaptive histogram equalization) and DCP with different parameters. Hardware platform of the experiment was X230, which is a notebook computer produced by Lenovo Inc. We took Nikon D7100 camera as the image acquisition equipment and fixed it with tripod when it worked. Experimental images were collected at the apple base in Changping District of Beijing. Image acquisition dates were some days with heavy fog and haze in November 2015. Twelve images were selected as the experimental materials. After the analysis of the experimental data, this paper got the following results: 1) The average contrast value of the images was 64.04 with our method; the AHE method was faster, but the contrast value was 35.46 with the AHE; the histogram obtained by our method had the characteristics of Gaussian distribution, which showed that our method could get better image quality; 2) Testing the 640×480 images pixels, our method required 36.46 ms computing time, the MSR method required 126.43 ms, and the AHE required 28.58 ms. The time performance of our method was not as good as the AHE, but it was better than the MSR; 3) The average location accuracy was 94.8% with our method, which was higher than other methods. The experiments show that our method can get better balance between efficiency and performance. It is a feasible method for the actual apple harvest operation. [ABSTRACT FROM AUTHOR]

Fruit and vegetable picking grippers are important technology to achieve rapid and labor-saving harvest. However, most of the existing fruit and vegetable picking grippers still use traditional rigid or underactuated grippers, which often cause fruits and vegetables damage by the heavy mass and lack of high-precision control, and have poor compliance in the operation process. In recent years, inspired by soft creature’s tentacles, soft robotic grippers have appeared and been used in robotics due to the emergence of soft robots. Soft robotic gripper, which is made of flexible material, is a new type of gripper for general purposes with grasping and holding capabilities enabled by a simple control scheme. Under the infinite degrees of freedom, the soft robotic gripper can change its shape and size corresponding to the load in a large range. These advantages overcome the defects of traditional fruit and vegetable picking robots, such as rigidity and poor adaptability. Moreover, soft robotic gripper is easy to manufacture and can be integrated with manual operation without any large-scale safe requirement. This paper gives a research on limitations of the traditional grippers and summarizes the characteristics of ideal picking grippers of fruit and vegetable. In addition, the concept of soft robotic grippers is introduced in detail, and the current development status about soft robotic grippers is described. The manuscript also summarizes the progress and superiority of soft robotic grippers in fruit and vegetable picking as well as strong adaptability to the environment. The characteristics of the soft robotic grippers driven by the common driving mode including pneumatic, cable, shape memory alloy and electroactive polymer mode have been analyzed in the process of fruit and vegetable grasping and picking. Compared with the rigid grippers, in addition to the simple control and mechanism, the soft robotic grippers have high-degree flexibility, adaptability and versatility. Based on the related work, the problems of modeling and control of fruit and vegetable picking devices have been discussed, and possible solutions of soft robotic grippers are also summarized by means of analysis and classification. We can choose the appropriate control methods according to the surface characteristics of picked fruits and vegetables. Finally, we conclude that multi-sensing, variable stiffness, multi-functional composite materials, as well as control strategies of fusion intelligence will be future development directions of fruit and vegetable picking grippers with the progress of micro-sensors and biomaterials. This research will provide theoretical and technical guidance for the development of fruit and vegetable picking grippers. It is expected that more fruit and vegetable picking operations could be carried out by soft robotic grippers, and the application will effectively reduce the damage rate of fruit and vegetable picking. As a new-generation operating device, the soft robotic grippers involve the development of materials, chemistry, machinery and other multidiscipline. Further studies are required to improve its design, controllability, manipulating methods, and so on. [ABSTRACT FROM AUTHOR]