Search

We propose a 6D pose estimation method that introduces an edge attention mechanism into the bidirectional feature fusion network. Our method constructs an end-to-end network model by sharing weights between the edge detection encoder and the encoder of the RGB branch in the feature fusion network, effectively utilizing edge information and improving the accuracy and robustness of 6D pose estimation. Experimental results show that this method achieves an accuracy of nearly 100% on the LineMOD dataset, and it also achieves state-of-the-art performance on the YCB-V dataset, especially on objects with significant edge information. [ABSTRACT FROM AUTHOR]

6D pose estimation using RGB-D data has been widely utilized in various scenarios, with keypoint-based methods receiving significant attention due to their exceptional performance. However, these methods still face numerous challenges, especially when the object is heavily occluded or truncated. To address this issue, we propose a novel cross-modal fusion network. Specifically, our approach initially employs object detection to identify the potential position of the object and randomly samples within this region. Subsequently, a specially designed feature extraction network is utilized to extract appearance features from the RGB image and geometry features from the depth image respectively; these features are then implicitly aggregated through cross-modal fusion. Finally, keypoints are employed for estimating the pose of the object. The proposed method undergoes extensive testing on Occlusion Linemod and Truncation Linemod datasets. Experimental results demonstrate that our method has made significant advancements, thereby validating the effectiveness of cross-modal feature fusion strategy in enhancing the accuracy of RGB-D image pose estimation based on keypoints. [ABSTRACT FROM AUTHOR]

In the agricultural sector, utilizing robots for tasks such as fruit harvesting poses significant challenges, particularly in achieving accurate 6D pose estimation of the target objects, which is essential for precise and efficient harvesting. Particularly, fruit harvesting relies heavily on manual labor, leading to issues with an unstable labor supply and rising costs. To solve these problems, agricultural harvesting robots are gaining attention. However, effective harvesting necessitates accurate 6D pose estimation of the target object. This study proposes a method to enhance the performance of fruit-harvesting robots, including the development of a dataset named HWANGMOD, which was created using both virtual and real environments with tools such as Blender and BlenderProc. Additionally, we present methods for training an EfficientPose-based model for 6D pose estimation and ripeness classification, and an algorithm for determining the optimal harvest sequence among multiple fruits. Finally, we propose a multi-object tracking method using coordinates estimated by deep learning models to improve the robot's performance in dynamic environments. The proposed methods were evaluated using metrics such as A D D and A D D S , showing that the deep learning model for agricultural harvesting robots excelled in accuracy, robustness, and real-time processing. These advancements contribute to the potential for commercialization of agricultural harvesting robots and the broader field of agricultural automation technology. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Due to the multiple types of objects and the uncertainty of their geometric structures and scales in indoor scenes, the position and pose estimation of point clouds of indoor objects by mobile robots has the problems of domain gap, high learning cost, and high computing cost. In this paper, a lightweight 6D pose estimation method is proposed, which decomposes the pose estimation into a viewpoint and the in-plane rotation around the optical axis of the viewpoint, and the improved PointNet + ⁣ + network structure and two lightweight modules are used to construct a codebook, and the 6d pose estimation of the point cloud of the indoor objects is completed by building and querying the codebook. The model was trained on the ShapeNetV2 dataset, and reports the ADD-S metric validation on the YCB-Video and LineMOD datasets, reaching 97.0% and 94.6% respectively. The experiment shows that the model can be trained to estimate the 6d position and pose of the unknown object point cloud with lower computation and storage cost, and the model with fewer parameters and better real-time performance is superior to other high-recision methods. [ABSTRACT FROM AUTHOR]

Instrument pose estimation is a key demand in computer-aided surgery, and its main challenges lie in two aspects: Firstly, the difficulty of obtaining stable corresponding image feature points due to the instruments' high refraction and complicated background, and secondly, the lack of labeled pose data. This study aims to tackle the pose estimation problem of surgical instruments in the current endoscope system using a single endoscopic image. More specifically, a weakly supervised method based on the instrument's image segmentation contour is proposed, with the effective assistance of synthesized endoscopic images. Our method consists of the following three modules: a segmentation module to automatically detect the instrument in the input image, followed by a point inference module to predict the image locations of the implicit feature points of the instrument, and a point back-propagatable Perspective-n-Point module to estimate the pose from the tentative 2D–3D corresponding points. To alleviate the over-reliance on point correspondence accuracy, the local errors of feature point matching and the global inconsistency of the corresponding contours are simultaneously minimized. Our proposed method is validated with both real and synthetic images in comparison with the current state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Object 6D pose estimation, as a key technology in applications such as augmented reality (AR), virtual reality (VR), robotics, and autonomous driving, requires the prediction of the 3D position and 3D pose of objects robustly from complex scene images. However, complex environmental factors such as occlusion, noise, weak texture, and lighting changes may affect the accuracy and robustness of object 6D pose estimation. We propose a robust CoS-PVNet (complex scenarios pixel-wise voting network) pose estimation network for complex scenes. By adding a pixel-weight layer based on the PVNet network, more accurate pixel point vectors are selected, and dilated convolution and adaptive weighting strategies are used to capture local and global contextual information of the input feature map. At the same time, the perspective-n-point localization algorithm is used to accurately locate 2D key points to solve the pose of 6D objects, and then, the transformation relationship matrix of 6D pose projection is solved. The research results indicate that on the LineMod and Occlusion LineMod datasets, CoS-PVNet has high accuracy and can achieve stable and robust 6D pose estimation even in complex scenes. [ABSTRACT FROM AUTHOR]

Abstract The six-dimensional (6D) pose object estimation is a key task in robotic manipulation and grasping scenes. Many existing two-stage solutions with a slow inference speed require extra refinement to handle the challenges of variations in lighting, sensor noise, object occlusion, and truncation. To address these challenges, this work proposes a decoupled one-stage network (DON6D) model for 6D pose estimation that improves inference speed on the premise of maintaining accuracy. Particularly, since the RGB images are aligned with the RGB-D images, the proposed DON6D first uses a two-dimensional detection network to locate the interested objects in RGB-D images. Then, a module of feature extraction and fusion is used to extract color and geometric features fully. Further, dual data augmentation is performed to enhance the generalization ability of the proposed model. Finally, the features are fused, and an attention residual encoder–decoder, which can improve the pose estimation performance to obtain an accurate 6D pose, is introduced. The proposed DON6D model is evaluated on the LINEMOD and YCB-Video datasets. The results demonstrate that the proposed DON6D is superior to several state-of-the-art methods regarding the ADD(-S) and ADD(-S) AUC metrics.

In the realm of computer vision and robotics, the pursuit of intelligent robotic grasping and accurate 6D object pose estimation has been a focal point of research. Many modern-world applications, such as robot grasping, manipulation, and palletizing, require the correct pose of objects present in a scene to perform their specific tasks. The estimation of a 6D object pose becomes even more challenging due to inherent complexities, especially when dealing with objects positioned within cluttered scenes and subjected to high levels of occlusion. While prior endeavors have made strides in addressing this issue, their accuracy falls short of the reliability demanded by real-world applications. In this research, we present an architecture that, unlike prior works, incorporates contextual awareness. This novel approach capitalizes on the contextual information attainable about the objects in question. The framework we propose takes a dissection approach, discerning objects by their intrinsic characteristics, namely whether they are symmetric or non-symmetric. Notably, our methodology employs a more profound estimator and refiner network tandem for non-symmetric objects, in contrast to symmetric ones. This distinction acknowledges the inherent dissimilarities between the two object types, thereby enhancing performance. Through experiments conducted on the LineMOD dataset, widely regarded as a benchmark for pose estimation in occluded and cluttered scenes, we demonstrate a notable improvement in accuracy of approximately 3.2% compared to the previous state-of-the-art method, DenseFusion. Moreover, our results indicate that the achieved inference time is sufficient for real-time usage. Overall, our proposed architecture leverages contextual information and tailors the pose estimation process based on object types, leading to enhanced accuracy and real-time performance in challenging scenarios. Code is available at GitHub link [ABSTRACT FROM AUTHOR]

Copyright of Cyber Security & Data Governance is the property of Editorial Office of Information Technology & Network Security and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The six-dimensional (6D) pose object estimation is a key task in robotic manipulation and grasping scenes. Many existing two-stage solutions with a slow inference speed require extra refinement to handle the challenges of variations in lighting, sensor noise, object occlusion, and truncation. To address these challenges, this work proposes a decoupled one-stage network (DON6D) model for 6D pose estimation that improves inference speed on the premise of maintaining accuracy. Particularly, since the RGB images are aligned with the RGB-D images, the proposed DON6D first uses a two-dimensional detection network to locate the interested objects in RGB-D images. Then, a module of feature extraction and fusion is used to extract color and geometric features fully. Further, dual data augmentation is performed to enhance the generalization ability of the proposed model. Finally, the features are fused, and an attention residual encoder–decoder, which can improve the pose estimation performance to obtain an accurate 6D pose, is introduced. The proposed DON6D model is evaluated on the LINEMOD and YCB-Video datasets. The results demonstrate that the proposed DON6D is superior to several state-of-the-art methods regarding the ADD(-S) and ADD(-S) AUC metrics. [ABSTRACT FROM AUTHOR]

The accurate estimation of 6D pose remains a challenging task within the computer vision domain, even when utilizing 3D point cloud data. Conversely, in the manufacturing domain, instances arise where leveraging prior knowledge can yield advancements in this endeavor. This study focuses on the disassembly of starter motors to augment the engineering of product life cycles. A pivotal objective in this context involves the identification and 6D pose estimation of bolts affixed to the motors, facilitating automated disassembly within the manufacturing workflow. Complicating matters, the presence of occlusions and the limitations of single-view data acquisition, notably when motors are placed in a clamping system, obscure certain portions and render some bolts imperceptible. Consequently, the development of a comprehensive pipeline capable of acquiring complete bolt information is imperative to avoid oversight in bolt detection. In this paper, employing the task of bolt detection within the scope of our project as a pertinent use case, we introduce a meticulously devised pipeline. This multi-stage pipeline effectively captures the 6D information with regard to all bolts on the motor, thereby showcasing the effective utilization of prior knowledge in handling this challenging task. The proposed methodology not only contributes to the field of 6D pose estimation but also underscores the viability of integrating domain-specific insights to tackle complex problems in manufacturing and automation. [ABSTRACT FROM AUTHOR]

With advancements in technology, deep learning has become increasingly widespread, particularly in fields like robot control, computer vision, and autonomous driving. In these areas, obtaining pose information of target objects, especially their spatial location, is crucial for robot grasping tasks. Although many effective implementations of six degrees of freedom (6D) pose estimation methods based on RGB images exist, challenges in this domain persist. This paper provides a comprehensive review of traditional 6D pose estimation methods, deep learning approaches, and point cloud techniques by analyzing their advantages and disadvantages. It also discusses evaluation metrics and performance on common datasets for 6D pose estimation. Furthermore, the paper offers a theoretical foundation for robot grasping and explores future directions for 6D pose estimation. Finally, it summarizes the current state and development trends of 6D pose estimation, aiming to help researchers better understand and learn about this field.

In the agricultural sector, utilizing robots for tasks such as fruit harvesting poses significant challenges, particularly in achieving accurate 6D pose estimation of the target objects, which is essential for precise and efficient harvesting. Particularly, fruit harvesting relies heavily on manual labor, leading to issues with an unstable labor supply and rising costs. To solve these problems, agricultural harvesting robots are gaining attention. However, effective harvesting necessitates accurate 6D pose estimation of the target object. This study proposes a method to enhance the performance of fruit-harvesting robots, including the development of a dataset named HWANGMOD, which was created using both virtual and real environments with tools such as Blender and BlenderProc. Additionally, we present methods for training an EfficientPose-based model for 6D pose estimation and ripeness classification, and an algorithm for determining the optimal harvest sequence among multiple fruits. Finally, we propose a multi-object tracking method using coordinates estimated by deep learning models to improve the robot’s performance in dynamic environments. The proposed methods were evaluated using metrics such as ADD and ADDS, showing that the deep learning model for agricultural harvesting robots excelled in accuracy, robustness, and real-time processing. These advancements contribute to the potential for commercialization of agricultural harvesting robots and the broader field of agricultural automation technology.

The capabilities of AUV mutual perception and localization are crucial for the development of AUV swarm systems. We propose the AUV6D model, a synthetic image-based approach to enhance inter-AUV perception through 6D pose estimation. Due to the challenge of acquiring accurate 6D pose data, a dataset of simulated underwater images with precise pose labels was generated using Unity3D. Mask-CycleGAN technology was introduced to transform these simulated images into realistic synthetic images, addressing the scarcity of available underwater data. Furthermore, the Color Intermediate Domain Mapping strategy is proposed to ensure alignment across different image styles at pixel and feature levels, enhancing the adaptability of the pose estimation model. Additionally, the Salient Keypoint Vector Voting Mechanism was developed to improve the accuracy and robustness of underwater pose estimation, enabling precise localization even in the presence of occlusions. The experimental results demonstrated that our AUV6D model achieved millimeter-level localization precision and pose estimation errors within five degrees, showing exceptional performance in complex underwater environments. Navigation experiments with two AUVs further verified the model’s reliability for mutual 6D pose estimation. This research provides substantial technical support for more complex and precise collaborative operations for AUV swarms in the future.

6D pose estimation is very important for supporting future smart technologies. The previous methods show optimal performance on RGB-D images or single objects. However, the problem still occurs in RGB images or multiple objects with occlusion. This study focuses on solving the problem using a deep learning approach. One of the key components of deep learning is the optimization process, which we research to determine its effect on solving the problem. The research methodology includes implementing the optimization techniques in the methods, measuring loss value, measuring performance, observing experimental results, analyzing statistical significance, and comparing the performance of optimizers. We implement Adam, RMSprop, Adagrad, Adadelta, and SGD optimizers and analyze their effects on the EfficientPose and DPOD methods. We use the LineMod-Occluded dataset to measure the performance of the methods using the ADD metric. According to the experiment, the loss value is low and stable in the experimental scenarios with a number of epochs between 200 and 500. The performance is relatively high in those scenarios, where Adadelta's performance outperforms other optimizers on both methods. Based on the analysis of variance, the effect of optimizers on the performance of the methods is low, but the slight performance increase is significant in this case. [ABSTRACT FROM AUTHOR]

Precise 6DoF (6D) object pose estimation is an essential topic for many intelligent applications, for example, robot grasping, virtual reality, and autonomous driving. Lacking depth information, traditional pose estimators using only RGB cameras consistently predict bias 3D rotation and translation matrices. With the wide use of RGB-D cameras, we can directly capture both the depth for the object relative to the camera and the corresponding RGB image. Most existing methods concatenate these two data sources directly, which does not make full use of their complementary relationship. Therefore, we propose an efficient RGB-D fusion network for 6D pose estimation, called EFN6D, to exploit the 2D–3D feature more thoroughly. Instead of directly using the original single-channel depth map, we encode the depth information into a normal map and point cloud data. To effectively fuse the surface texture features and the geometric contour features of the object, we feed the RGB images and the normal map into two ResNets. Besides, the PSP modules and skip connections are used between the two ResNets, which not only enhances cross modal fusion performance of the network but also enhances the network's capability in handling objects at different scales. Finally, the fused features obtained from these two ResNets and the point cloud features are densely fused point by point to further strengthen the fusion of 2D and 3D information at a per-pixel level. Experiments on the LINEMOD and YCB-Video datasets show that our EFN6D outperforms state-of-the-art methods by a large margin. [ABSTRACT FROM AUTHOR]

We propose a 6D pose estimation method that introduces an edge attention mechanism into the bidirectional feature fusion network. Our method constructs an end-to-end network model by sharing weights between the edge detection encoder and the encoder of the RGB branch in the feature fusion network, effectively utilizing edge information and improving the accuracy and robustness of 6D pose estimation. Experimental results show that this method achieves an accuracy of nearly 100% on the LineMOD dataset, and it also achieves state-of-the-art performance on the YCB-V dataset, especially on objects with significant edge information.

The 6D pose estimation using RGBD images plays a pivotal role in robotics applications. At present, after obtaining the RGB and depth modality information, most methods directly concatenate them without considering information interactions. This leads to the low accuracy of 6D pose estimation in occlusion and illumination changes. To solve this problem, we propose a new method to fuse RGB and depth modality features. Our method effectively uses individual information contained within each RGBD image modality and fully integrates cross-modality interactive information. Specifically, we transform depth images into point clouds, applying the PointNet++ network to extract point cloud features; RGB image features are extracted by CNNs and attention mechanisms are added to obtain context information within the single modality; then, we propose a cross-modality feature fusion module (CFFM) to obtain the cross-modality information, and introduce a feature contribution weight training module (CWTM) to allocate the different contributions of the two modalities to the target task. Finally, the result of 6D object pose estimation is obtained by the final cross-modality fusion feature. By enabling information interactions within and between modalities, the integration of the two modalities is maximized. Furthermore, considering the contribution of each modality enhances the overall robustness of the model. Our experiments indicate that the accuracy rate of our method on the LineMOD dataset can reach 96.9%, on average, using the ADD (-S) metric, while on the YCB-Video dataset, it can reach 94.7% using the ADD-S AUC metric and 96.5% using the ADD-S score (<2 cm) metric. [ABSTRACT FROM AUTHOR]

In the context of collaborative robotics, handing over hand-held objects to a robot is a safety-critical task. Therefore, a robust distinction between human hands and presented objects in image data is essential to avoid contact with robotic grippers. To be able to develop machine learning methods for solving this problem, we created the OHO (Object Hand-Over) dataset of tools and other everyday objects being held by human hands. Our dataset consists of color, depth, and thermal images with the addition of pose and shape information about the objects in a real-world scenario. Although the focus of this paper is on instance segmentation, our dataset also enables training for different tasks such as 3D pose estimation or shape estimation of objects. For the instance segmentation task, we present a pipeline for automated label generation in point clouds, as well as image data. Through baseline experiments, we show that these labels are suitable for training an instance segmentation to distinguish hands from objects on a per-pixel basis. Moreover, we present qualitative results for applying our trained model in a real-world application. [ABSTRACT FROM AUTHOR]

Instrument pose estimation is a key demand in computer-aided surgery, and its main challenges lie in two aspects: Firstly, the difficulty of obtaining stable corresponding image feature points due to the instruments’ high refraction and complicated background, and secondly, the lack of labeled pose data. This study aims to tackle the pose estimation problem of surgical instruments in the current endoscope system using a single endoscopic image. More specifically, a weakly supervised method based on the instrument’s image segmentation contour is proposed, with the effective assistance of synthesized endoscopic images. Our method consists of the following three modules: a segmentation module to automatically detect the instrument in the input image, followed by a point inference module to predict the image locations of the implicit feature points of the instrument, and a point back-propagatable Perspective-n-Point module to estimate the pose from the tentative 2D–3D corresponding points. To alleviate the over-reliance on point correspondence accuracy, the local errors of feature point matching and the global inconsistency of the corresponding contours are simultaneously minimized. Our proposed method is validated with both real and synthetic images in comparison with the current state-of-the-art methods.

6D pose recognition has been a crucial factor in the success of robotic grasping, and recent deep learning based approaches have achieved remarkable results on benchmarks. However, their generalization capabilities in real-world applications remain unclear. To overcome this gap, we introduce 6IMPOSE, a novel framework for sim-to-real data generation and 6D pose estimation. 6IMPOSE consists of four modules: First, a data generation pipeline that employs the 3D software suite Blender to create synthetic RGBD image datasets with 6D pose annotations. Second, an annotated RGBD dataset of five household objects was generated using the proposed pipeline. Third, a real-time two-stage 6D pose estimation approach that integrates the object detector YOLO-V4 and a streamlined, real-time version of the 6D pose estimation algorithm PVN3D optimized for time-sensitive robotics applications. Fourth, a codebase designed to facilitate the integration of the vision system into a robotic grasping experiment. Our approach demonstrates the efficient generation of large amounts of photo-realistic RGBD images and the successful transfer of the trained inference model to robotic grasping experiments, achieving an overall success rate of 87% in grasping five different household objects from cluttered backgrounds under varying lighting conditions. This is made possible by fine-tuning data generation and domain randomization techniques and optimizing the inference pipeline, overcoming the generalization and performance shortcomings of the original PVN3D algorithm. Finally, we make the code, synthetic dataset, and all the pre-trained models available on GitHub.

Fast and accurate 6D pose estimation can help a robot arm grab industrial parts efficiently. The previous 6D pose estimation algorithms mostly target common items in daily life. Few algorithms are aimed at texture-less and occluded industrial parts and there are few industrial parts datasets. A novel method called the Industrial Parts 6D Pose Estimation framework based on point cloud repair (IPPE-PCR) is proposed in this paper. A synthetic dataset of industrial parts (SD-IP) is established as the training set for IPPE-PCR and an annotated real-world, low-texture and occluded dataset of industrial parts (LTO-IP) is constructed as the test set for IPPE. To improve the estimation accuracy, a new loss function is used for the point cloud repair network and an improved ICP method is proposed to optimize template matching. The experiment result shows that IPPE-PCR performs better than the state-of-the-art algorithms on LTO-IP. [ABSTRACT FROM AUTHOR]

Estimation of 6D object poses is a key issue in robotic grasping tasks. Recently, many high-performance learning-based methods have been introduced using robust deep learning techniques; however, applying these methods to real robot environments requires many ground truth 6D pose annotations for training. To address this problem, we propose a template matching-based particle filter approach for 6D pose estimation; the proposed method does not require ground truth 6D poses. Although particle filter approaches can stochastically avoid local optima, they require adequate initial pose hypotheses for estimating an accurate 6D object pose. Therefore, we estimated an initial translation of the target object for accurately initializing a particle filter by developing a new deep network. Once the proposed centroid prediction network (CPN) is trained with a specific dataset, no additional training is required for new objects not in the dataset. We evaluated the performance of the CPN and the proposed 6D pose estimation method on benchmark datasets, which demonstrated that the CPN can predict the centroid for any object, including those not in the training data, and that our 6D pose estimation method outperforms existing methods for partially occluded objects. Finally, we tested a grasping task based on our proposed method using a real robot platform to demonstrate an application of our method to a downstream task. This experiment shows that our method can be applied to part assembly, bin picking, and object manipulation without large training datasets with 6D pose annotations. The code and models are available at: https://github.com/oorrppp2/Particle_filter_approach_6D_pose_estimation.

Accurate and efficient object pose estimation holds the indispensable part of virtual/augmented reality (VR/AR) and many other applications. While previous works focus on directly regressing 6D pose from RGB and depth image and thus suffer from the non-linearity of rotation space, we propose an iterative 3D keypoints voting network, named as KVNet. Specifically, our method decouples the pose into separate translation and rotation branch, both estimated by Hough voting scheme. By treating the uncertainty of keypoints' vote as the Lipschitz continuous function of seed points' fused embedding feature, our method is able to adaptively select the optimal keypoints vote. In this way, we argue that KVNet bridges the gap between the non-linear rotation space and linear Euclidean space, which introduces inductive bias for our network to learn the intrinsic pattern and infer 6D pose from RGB and depth images. Furthermore, our model will refine the initial keypoints localization with iterative fashion. Experiments show that across three challenging benchmark datasets (LineMOD, YCB-Video and Occlusion LineMOD), our method exhibits excellent performance. [ABSTRACT FROM AUTHOR]

Visual tracking of objects is a fundamental technology for industry 4.0, allowing the integration of digital content and real-world objects. The industrial operation known as manual cargo packing can benefit from the visual tracking of objects. No dataset exists to evaluate the visual tracking algorithms on manual packing scenarios. To close this gap, this article presents 6D-ViCuT, a dataset of images, and 6D pose ground truth of cuboids in a manual packing operation in intralogistics. The initial release of the dataset comprehends 28 sessions acquired in a space that rebuilds a manual packing zone: indoors, area of (6 × 4 × 2) m3, and warehouse illumination. The data acquisition experiment involves capturing images from fixed and mobile RGBD devices and a motion capture system while an operator performs a manual packing operation. Each session contains between 6 and 18 boxes from an available set of 10 types, with each type varying in height, width, depth, and texture. Each session had a duration in the range of 1 to 5 minutes. Each session exhibits operator speed and box type differences (box texture, size heterogeneity, occlusion).

Unmanned aerial vehicle (UAV) express delivery is facing a period of rapid development and continues to promote the aviation logistics industry due to its advantages of elevated delivery efficiency and low labor costs. Automatic detection, localization, and estimation of 6D poses of targets in dynamic environments are key prerequisites for UAV intelligent logistics. In this study, we proposed a novel vision system based on deep neural networks to locate targets and estimate their 6D pose parameters from 2D color images and 3D point clouds captured by an RGB-D sensor mounted on a UAV. The workflow of this system can be summarized as follows: detect the targets and locate them, separate the object region from the background using a segmentation network, and estimate the 6D pose parameters from a regression network. The proposed system provides a solid foundation for various complex operations for UAVs. To better verify the performance of the proposed system, we built a small dataset called SIAT comprising some household staff. Comparative experiments with several state-of-the-art networks on the YCB-Video dataset and SIAT dataset verified the effectiveness, robustness, and superior performance of the proposed method, indicating its promising applications in UAV-based delivery tasks. [ABSTRACT FROM AUTHOR]

In order to solve the problem of low accuracy of 6D pose estimation for textured models under occlusion, this paper proposes an end-to-end 6D pose estimation algorithm based on local feature representation. Firstly, this paper proposes a Spatial and Coordinate Attention mechanism to obtain accurate localization information. A YOLOv5-CBE detection network forms by adding the attention mechanism to the backbone network and introducing a weighted Bidirectional Feature Pyramid Network in the detection layer. The Precision, Recall and mAP@0.5 of YOLOV5-CBE algorithm rise by 3.6%, 2.8% and 2.5% respectively, and the coordinate error of local feature center point decreases by 25% at most. Secondly, the YOLOv5-CBE network detects the local feature key points and calculates 6D pose of the model with 3 D Harris key points by Singular Value Decomposition, and the algorithm can guarantee 2D reprojection Accuracy and ADD Accuracy above 95% with 70% occlusion, which has a strong robustness. [ABSTRACT FROM AUTHOR]

Six-dimensional pose estimation for non-Lambertian objects, such as metal parts, is essential in intelligent manufacturing. Current methods pay much less attention to the influence of the surface reflection problem in 6D pose estimation. In this paper, we propose a cross-attention-based reflection-aware 6D pose estimation network (CAR6D) for solving the surface reflection problem in 6D pose estimation. We use a pseudo-Siamese network structure to extract features from both an RGB image and a 3D model. The cross-attention layers are designed as a bi-directional filter for each of the inputs (the RGB image and 3D model) to focus on calculating the correspondences of the objects. The network is trained to segment the reflection area from the object area. Training images with ground-truth labels of the reflection area are generated with a physical-based rendering method. The experimental results on a 6D dataset of metal parts demonstrate the superiority of CAR6D in comparison with other state-of-the-art models. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of recognizing multiple objects and multiple instances from point clouds. Whereas existing methods utilize descriptors on 3D fields or pointwise voting to achieve this task, our framework takes advantage of both descriptor-based and voting-based schemes to realize more robust and efficient prediction. Specifically, we propose a novel and robust descriptor called an orientation-enhanced fast point feature histogram (OE-FPFH) to describe points in both the object model and scene, and further to build the correspondence set. The OE-FPFH integrates an orientation vector through mining the geometric tensor of the local structure of a surface point, which is more representative than the original FPFH descriptor. To improve voting efficiency, we devise a novel single-point voting mechanism (SPVM), which constructs a unique local reference frame (LRF) on a single point using the orientation vector. The SPVM takes as input the corresponding point set and can generate a pose candidate for each correspondence. The process is realized by matching LRFs from two corresponding points. All pose candidates are subsequently divided into clusters and aggregated using the K -means clustering algorithm to deduce the poses for different objects or instances in the scene. Experiments on three challenging datasets demonstrate that our method is effective, efficient, and robust to occlusions and multiple instances. • Addresses the problem of recognizing multiple objects and multiple instances from point clouds. • Design a new robust descriptor called Direction-Enhanced Fast Point Feature Histogram (OE-FPFH). • Propose a novel single point voting mechanism (SPVM) that uses direction vectors. • Combines with the advantages of feature matching, the anti-interference and robustness of the attitude voting method are enhanced. [ABSTRACT FROM AUTHOR]

The 6D pose estimation using RGBD images plays a pivotal role in robotics applications. At present, after obtaining the RGB and depth modality information, most methods directly concatenate them without considering information interactions. This leads to the low accuracy of 6D pose estimation in occlusion and illumination changes. To solve this problem, we propose a new method to fuse RGB and depth modality features. Our method effectively uses individual information contained within each RGBD image modality and fully integrates cross-modality interactive information. Specifically, we transform depth images into point clouds, applying the PointNet++ network to extract point cloud features; RGB image features are extracted by CNNs and attention mechanisms are added to obtain context information within the single modality; then, we propose a cross-modality feature fusion module (CFFM) to obtain the cross-modality information, and introduce a feature contribution weight training module (CWTM) to allocate the different contributions of the two modalities to the target task. Finally, the result of 6D object pose estimation is obtained by the final cross-modality fusion feature. By enabling information interactions within and between modalities, the integration of the two modalities is maximized. Furthermore, considering the contribution of each modality enhances the overall robustness of the model. Our experiments indicate that the accuracy rate of our method on the LineMOD dataset can reach 96.9%, on average, using the ADD (-S) metric, while on the YCB-Video dataset, it can reach 94.7% using the ADD-S AUC metric and 96.5% using the ADD-S score (

In the context of collaborative robotics, handing over hand-held objects to a robot is a safety-critical task. Therefore, a robust distinction between human hands and presented objects in image data is essential to avoid contact with robotic grippers. To be able to develop machine learning methods for solving this problem, we created the OHO (Object Hand-Over) dataset of tools and other everyday objects being held by human hands. Our dataset consists of color, depth, and thermal images with the addition of pose and shape information about the objects in a real-world scenario. Although the focus of this paper is on instance segmentation, our dataset also enables training for different tasks such as 3D pose estimation or shape estimation of objects. For the instance segmentation task, we present a pipeline for automated label generation in point clouds, as well as image data. Through baseline experiments, we show that these labels are suitable for training an instance segmentation to distinguish hands from objects on a per-pixel basis. Moreover, we present qualitative results for applying our trained model in a real-world application.