Search

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

Three-dimensional (3-D) object recognition task focuses on detecting the objects of a scene and estimating their 6-DOF pose via effective feature extraction methods. Most recent feature extraction methods are based on the deep neural networks and show good performances. However, these methods require rendering engine to assist in generating a large amount of training data, which need much time to converge and further lead to the block in a rapid industrial production line. Besides, for the common hand-crafted features, the lack of discriminant feature-points amongst various texture-less and surface-smooth objects can cause ambiguity in the process of feature-points matching. To address these challenges above, a hand-crafted 3-D feature descriptor with center offset and pose annotations is proposed in this article, which is called view-specific local projection statistics (VSLPSs). By relying on these annotations as seeds, a voting strategy is then used to transform the feature-points matching problem into the problem of voting an optimal model-view in the 6-DOF space. In this way, the ambiguity of feature-points matching caused by poor feature discrimination is eliminated. To the end, various experiments on three public datasets and our built 3-D bin-picking dataset demonstrate that our proposed VSLPS method performs well in comparison with the state-of-the-art.

3D object classification has been widely applied in both academic and industrial scenarios. However, most state-of-the-art algorithms rely on a fixed object classification task set, which cannot tackle the scenario when a new 3D object classification task is coming. Meanwhile, the existing lifelong learning models can easily destroy the learned tasks performance, due to the unordered, large-scale, and irregular 3D geometry data. To address these challenges, we propose a L ifelong 3D O bject C lassification ( i.e., L3DOC) model, which can consecutively learn new 3D object classification tasks via imitating “human learning”. More specifically, the core idea of our model is to capture and store the cross-task common knowledge of 3D geometry data in a 3D neural network, named as point-knowledge , through employing layer-wise point-knowledge factorization architecture. Afterwards, a task-relevant knowledge distillation mechanism is employed to connect the current task to previous relevant tasks and effectively prevent catastrophic forgetting . It consists of a point-knowledge distillation module and a transforming-space distillation module, which transfers the accumulated point-knowledge from previous tasks and soft-transfers the compact factorized representations of the transforming-space, respectively. To our best knowledge, the proposed L3DOC algorithm is the first attempt to perform deep learning on 3D object classification tasks in a lifelong learning way. Extensive experiments on several point cloud benchmarks illustrate the superiority of our L3DOC model over the state-of-the-art lifelong learning methods.

Human males absent on the first (MOF), a histone acetyltransferase (HAT), forms male-specific lethal (MSL) and non-specific lethal (NSL), two multiprotein HATs, in cells. MSL was originally discovered in dosage compensation study in Drosophila that can specifically acetylate H4K16, while NSL can simultaneously catalyze the H4 at K5, K8, and K16 sites. However, comparative studies of the two HATs in regulating specific biological functions are rarely reported. Here, we present evidence to argue that MSL and NSL function in different ways in the epithelial-to-mesenchymal transition (EMT) process. At first, CRISPR/Cas9-mediated MSL1 (a key subunit of the MSL)-knockout (KO) and NSL3 (a key subunit of the NSL)-KO cells seem to prefer to grow in clusters. Interestingly, the former promotes cell survival and clonal formation, while the latter has the opposite effect on it. Cell staining revealed that MSL1-KO leads to multipolarized spindles, while NSL3-KO causes more lumen-like cells. Furthermore, in Transwell experiments, silencing of MSL1 promotes cell invasion in 293 T, MCF-7, and MDA-MB-231 cells. In contrast, the inhibitory effects on cell invasion are observed in the same NSL3-silenced cells. Consistent with this, mesenchymal biomarkers, like N-cadherin, vimentin, and snail, are negatively correlated with the expression level of MSL1; however, a positive relationship between these proteins and NSL3 in cells has been found. Further studies have clarified that MSL1, but not NSL3, can specifically bind to the E-box-containing Snail promoter region and thereby negatively regulate Snail transactivation. Also, silencing of MSL1 promotes the lung metastasis of B16F10 melanoma cells in mice. Finally, ChIP-Seq analysis indicated that the NSL may be mainly involved in phosphoinositide-mediated signaling pathways. Taken together, the MOF-containing MSL and NSL HATs may regulate the EMT process in different ways in order to respond to different stimuli.

The human males absent on the first (MOF)-containing non-specific lethal (NSL) histone acetyltransferase (HAT) complex acetylates histone H4 at lysine K5, K8, and K16. This complex shares several subunits with other epigenetic regulatory enzymes, which highlights the complexity of its intracellular function. However, the effect of the NSL HAT complex on the genome and target genes in human cells is still unclear. By using a CRISPR/Cas9-mediated NSL3-knockout 293T cell line and chromatin immunoprecipitation-sequencing (ChIP-Seq) approaches, we identified more than 100 genes as NSL HAT transcriptional targets, including several transcription factors, such as Yin Yang 1 (YY1) which are mainly involved in cell proliferation, biological adhesion, and metabolic processes. We found here that the ChIP-Seq peaks of MOF and NSL3 co-localized with H4K16ac, H3K4me2, and H3K4me3 at the transcriptional start site of YY1. In addition, both the mRNA and protein expression levels of YY1 were regulated by silencing or overexpressing NSL HAT. Interestingly, the expression levels of cell division cycle 6, a downstream target gene of YY1, were regulated by MOF or NSL3. In addition, the suppressed clonogenic ability of HepG2 cells caused by siNSL3 was reversed by overexpressing YY1, suggesting the involvement of YY1 in NSL HAT functioning. Additionally, de novo motif analysis of MOF and NSL3 targets indicated that the NSL HAT complex may recognize the specific DNA-binding sites in the promoter region of target genes in order to regulate their transcription.

Accurate 3D object recognition and 6-DOF pose estimation have been pervasively applied to a variety of applications, such as unmanned warehouse, cooperative robots, and manufacturing industry. How to extract a robust and representative feature from the point clouds is an inevitable and important issue. In this paper, an unsupervised feature learning network is introduced to extract 3D keypoint features from point clouds directly, rather than transforming point clouds to voxel grids or projected RGB images, which saves computational time while preserving the object geometric information as well. Specifically, the proposed network features in a stacked point feature encoder, which can stack the local discriminative features within its neighborhoods to the original point-wise feature counterparts. The main framework consists of both offline training phase and online testing phase. In the offline training phase, the stacked point feature encoder is trained first and then generate feature database of all keypoints, which are sampled from synthetic point clouds of multiple model views. In the online testing phase, each feature extracted from the unknown testing scene is matched among the database by using the K-D tree voting strategy. Afterwards, the matching results are achieved by using the hypothesis & verification strategy. The proposed method is extensively evaluated on four public datasets and the results show that ours deliver comparable or even superior performances than the state-of-the-arts in terms of F1-score, Average of the 3D distance (ADD) and Recognition rate.

The 3D mesh is an important representation of geometric data. It is widely used in computer graphics and has attracted more attention in computer vision community recently. However, in the generation of mesh data, geometric deficiencies (e.g., duplicate elements, degenerate faces, isolated vertices, self-intersection, and inner faces) are unavoidable. Geometric deficiencies may violate the topology structure of an object and affect the use of 3D meshes. In this paper, we propose an end-to-end algorithm to eliminate geometric deficiencies effectively and efficiently for 3D meshes in a specific and reasonable order. Specifically, duplicate elements can be first eliminated by assessing appear times of vertices or faces. Then, degenerate faces can be removed according to the outer product of two edges. Next, since isolated vertices do not appear in any face vertices, they can be deleted directly. Afterward, self-intersecting faces are detected and remeshed by using an AABB tree. Finally, we detect and remove an inner face according to whether multiple random rays shooted from a face can reach infinity. Experiments on ModelNet40 dataset illustrate that our method can eliminate the deficiencies of 3D meshes thoroughly.

As an important type of geometric data for 3D shapes, the unique topological connection makes the mesh more powerful than other types of data, but it also introduces complexity and irregularity. In this paper, we propose a Topological Perception Network (TPN) that consumes meshes directly to learn 3D shape representation via informative topology property. More specifically, to tackle the complexity and irregularity problem, a Topological Perception Attention (TPA) is designed that could incorporate local topological information efficiently via focusing on more important edges of the local topological neighborhood. Meanwhile, it could be stacked to produce global shape representation. Compared with the state-of-the-art, the proposed TPN uses less than half of the vertex number to get better performance, while costing less memory and computational time. Experiments on ModelNet40 and ShapeNet Core55 datasets demonstrate the effectiveness of our method on classification and retrieval.

Both OGT1 (O-linked &beta, N-acetylglucosamine (O-GlcNAc) transferase isoform 1) and NSL3 (nonspecific lethal protein 3) are crucial components of the MOF (males absent on the first)/NSL histone acetyltransferase complex. We previously described how global histone H4 acetylation levels were modulated by OGT1/O-GlcNAcylation-mediated NSL3 stability. However, the specific modification site of NSL3 and its molecular mechanism of protein stability remain unknown. Here, we present evidence from biochemical experiments arguing that O-GlcNAcylation of NSL3 at Thr755 is tightly associated with holoenzyme activity of the MOF/NSL complex. Using in vitro O-GlcNAc-transferase assays combined with mass spectrometry, we suppose that the residue Thr755 on NSL3 C-terminus is the major site O-GlcNAc-modified by OGT1. Importantly, O-GlcNAcylation of this site is involved in the regulation of the ubiquitin-degradation of NSL3, because this site mutation (T755A) promotes the ubiquitin-mediated degradation of NSL3. Further in-depth research found that ubiquitin conjugating enzyme E2 S (UBE2S) accelerated the degradation of NSL3 via direct binding to it. Interestingly, OGT1 and UBE2S competitively bind to NSL3, suggesting the coordination of OGT1&ndash, UBE2S in regulating NSL3 stability. Furthermore, O-GlcNAcylation of NSL3 Thr755 site regulates the histone H4 acetylation levels at lysine 5, 8, and 16, suggesting that the O-GlcNAcylation of NSL3 at Thr755 is required for maintaining the integrity and holoenzyme activity of the MOF/NSL complex. In colony formation assays, we found that the integrity of the complex impacts the proliferation of the lung carcinoma type II epithelium-like A549 cells. Taken together, our results provide new insight into the elucidation of the molecular mechanism of the MOF/NSL complex.

Photodynamic therapy (PDT) is one of the non-invasive and selective treatment methodologies for cancer. However, many highly efficient photosensitizers (PSs) are usually low physiological solubility, limited bioavailability and tending aggregation, impeding the effectiveness of PDT, as well as cancer resistance of PDT further reduce its therapeutic effect. Though some smart delivery systems have been developed, the problem of photosensitizer leakage/release has not been completely solved. Herein, we developed a smart therapeutic nanoplatform based on polyphotosensitizer nanogel as novel nanophotosensitizers and drug carriers. Moreover, by loading of histone deacetylase inhibitors (SAHA), it allows for enhanced synergistic therapy strategy of prostate cancer via inhibiting HIF-1α and VEGF pathways of cancer cells involved in PDT resistance. Our study presents the well-designed nanoplatform of nanogel-Ce6, which could serve as a photodynamic agent without Ce6 molecules release in the responsive environment, offering the potential to encapsulate diverse functional components for smart drug release and imaging-guided combination therapy in vitro and in vivo.

In this paper, we develop a novel 3D object recognition algorithm to perform detection and pose estimation jointly. We focus on analyzing the advantages of the 3D point cloud relative to the RGB-D image and try to eliminate the unpredictability of output values that inevitably occurs in regression tasks. To achieve this, we first adopt the Truncated Signed Distance Function (TSDF) to encode the point cloud and extract low compact discriminative feature via unsupervised deep learning network. This approach can not only eliminate the dense scale sampling for offline model training but also reduce the distortion by mapping the 3D shape to the 2D plane and overcome the dependence on color cues. Then, we train a Hough forests to achieve multi-object detection and 6-DoF pose estimation simultaneously. In addition, we propose a robust multilevel verification strategy that effectively reduces the unpredictability of output values which occurs in the hough regression module. Experiments on public datasets demonstrate that our approach provides effective results comparable to the state-of-the-arts.

Robust 3D object detection and pose estimation is still a big challenging for robot vision. In this paper, we propose a new framework for 3D object detection and pose estimation. Rather than using RGB-D image as the original data, we propose to use volumetric representation with the help of unsupervised deep learning network to extract low dimensional feature from 3D point cloud directly. The volumetric representation can not only eliminate the dense scale sampling for offline model training, but also reduce the distortion by mapping the 3D shape to 2D plane and overcome the dependence on texture information. Depending on the Hough forest, we can achieve multi-object detection and pose estimation simultaneously. In compare with the state-of-the-arts using public datasets, we justify the effectiveness of our proposed method.