Tapping line detection and rubber tapping pose estimation for natural rubber trees based on improved YOLOv8 and RGB-D information fusion

Abstract Tapping line detection and rubber tapping pose estimation are challenging tasks in rubber plantation environments for rubber tapping robots. This study proposed a method for tapping line detection and rubber tapping pose estimation based on improved YOLOv8 and RGB-D information fusion. Firstly, YOLOv8n was improved by introducing the CFB module into the backbone, adding an output layer into the neck, fusing the EMA attention mechanism into the neck, and modifying the loss function as NWD to realize multi-object detection and segmentation. Secondly, the trunk skeleton line was extracted by combining level set and ellipse fitting. Then, the new tapping line was located by combining edge detection and geometric analysis. Finally, the rubber tapping pose was estimated based on the trunk skeleton line and the new tapping line. The detection results from 597 test images showed the improved YOLOv8n’s detection mAP0.5, segmentation mAP0.5, and model size were 81.9%, 72.9%, and 6.06 MB, respectively. The improved YOLOv8n’s effect and efficiency were superior compared to other networks, and it could better detect and segment natural rubber tree image targets in different scenes. The pose estimation results from 300 new tapping lines showed the average success rate and average time consumed for rubber tapping pose estimation were 96% and 0.2818 s, respectively. The positioning errors in x, y, and z directions were 0.69 ± 0.51 mm, 0.73 ± 0.4 mm, and 1.07 ± 0.56 mm, respectively. The error angles in a, o, and n directions were 1.65° ± 0.68°, 2.53° ± 0.88°, and 2.26° ± 0.89°, respectively. Therefore, this method offers an effective solution for rubber tapping pose estimation and provides theoretical support for the development of rubber tapping robots.