Improved multi-classes kiwifruit detection in orchard to avoid collisions during robotic picking

Deep learning has achieved kiwifruit detection with high accuracy and fast speed. However, all the kiwifruits have been labeled and detected as only one class in most researches for robotic fruit picking, where fruits occluded by branches or wires have been detected as pickable targets. End-effectors or robots may be damaged by the branches or wires when they are forced to pick those fruits. Therefore, kiwifruits are labeled, trained, and detected in multi-classes based on their occlusions to avoid detecting fruits occluded by branches or wires as pickable targets. Fruits are classified into four classes and five classes according to robotic picking strategy and field occlusions, respectively. Well-known YOLOv3 and recently released YOLOv4 are employed to do transfer learning for multi-classes kiwifruit detection. Results show that mAP (mean average precision) of fruits in the five-classes is higher than that in the four-classes, while mAP of YOLOv4 is higher than YOLOv3. The mAP of YOLOv4 and YOLOv3 in the five-classes and four-classes are 91.9%, 91.5%, 91.1%, and 89.5%, respectively. The results demonstrate that fruits labeled and trained in more classes can achieve higher mAP. There are significant differences in average detection speed in YOLOv3 and YOLOv4, but no in the four-classes and five-classes. Overall, the highest mAP of 91.9% was achieved by YOLOv4 in the five-classes, which cost 25.5 ms on average to process a 2352 × 1568 image. The results illustrate that multi-classes kiwifruit detection is helpful for avoiding damage to the end-effectors or robots.