Search

Scene graphs connect individual objects with visual relationships. They serve as a comprehensive scene representation for downstream multimodal tasks. However, by exploring recent progress in Scene Graph Generation (SGG), we find that the performance of recent works is highly limited by the pairwise relationship modeling by naive feature concatenation. Such pairwise features lack sufficient object interaction due to the mis-aligned object parts, resulting in non-discriminative pairwise features for visual relationship prediction. For example, nave concatenated pairwise feature usually make the model fail to discriminate between riding and feeding for object pair person and horse. To this end, we design a meta-architecture learning-to-align for dynamic object feature concatenation. We call our model: Align R-CNN. Specifically, we introduce a novel attention-based multiple region alignment module that can be jointly optimized with SGG. Experiments on the large-scale SGG benchmark Visual Genome show that the proposed Align R-CNN can replace the naive feature concatenation and thus boost all the existing SGG methods.

Background Spinal cord injury (SCI) is the most serious complication of spinal injury, often leading to severe dysfunction of the limbs below the injured segment. Conventional therapy approaches are becoming less and less effective, and gene therapy is a new research direction by now. Methods The Sprague-Dawley rats were haphazardly assigned to two groups, namely sham group and SCI model group, and lncRNA H19 and miR-370-3p levels were investigated using reverse transcription-quantitative polymerase chain reaction (RT-qPCR). Correlation between lncRNA H19 and miR-370-3p was ascertained by luciferase report assay and RT-qPCR. After transfection with si-H19, miR-370-3p inhibitor, negative controls (NC), or both, primary spinal neurons were subjected to the simulation of lipopolysaccharide (LPS) for inducing in vitro model of SCI. Cell viability, apoptotic rate, caspase-3 activity, Bax and Bcl-2 protein, ROS generation, TNF-α, IL-1β, and IL-6 protein, as well as IκBα and p65 phosphorylation ratio were evaluated adopting 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide (MTT), apoptosis, caspase-3 activity, ROS generation, and western blot assays, thereby searching for the specific action mechanism on LPS-induced spinal never injury. Results SCI resulted in lncRNA H19 higher expression and miR-370-3p lower expression. LPS simulation raised a series of cellular biological changes, such as decreased viability, promoted apoptosis, generated ROS, and released inflammatory factors. lncRNA H19 inhibition reversed above LPS-induced changes. Besides, as the downstream target of lncRNA H19, miR-370-3p was oppositely regulated by lncRNA H19. The above biological changes induced by lncRNA H19 inhibition were reversed by miR-370-3p upregulation. Moreover, lncRNA H19 inhibition could block NF-κB pathway through miR-370-3p upregulation. Conclusion Inhibition of lncRNA H19/miR-370-3p mitigated spinal neuron apoptosis in an in vitro model of SCI. This provided the possibility for clinical use of gene therapy.

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the framework, we find that although the feature/label distillation is storage-efficient, its causal effect is not coherent with the end-to-end feature learning merit, which is however preserved by data replay. To this end, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub&Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72%--9.06%).

We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A. Our visual context tree model, dubbed VCTree, has two key advantages over existing structured object representations including chains and fully-connected graphs: 1) The efficient and expressive binary tree encodes the inherent parallel/hierarchical relationships among objects, e.g., "clothes" and "pants" are usually co-occur and belong to "person"; 2) the dynamic structure varies from image to image and task to task, allowing more content-/task-specific message passing among objects. To construct a VCTree, we design a score function that calculates the task-dependent validity between each object pair, and the tree is the binary version of the maximum spanning tree from the score matrix. Then, visual contexts are encoded by bidirectional TreeLSTM and decoded by task-specific models. We develop a hybrid learning procedure which integrates end-task supervised learning and the tree structure reinforcement learning, where the former's evaluation result serves as a self-critic for the latter's structure exploration. Experimental results on two benchmarks, which require reasoning over contexts: Visual Genome for scene graph generation and VQA2.0 for visual Q&A, show that VCTree outperforms state-of-the-art results while discovering interpretable visual context structures.

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation `person on bike', it is natural to replace `on' with `ride' and infer `person riding bike on a road' even the `road' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph ($\mathcal{G}$) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image ($\mathcal{I}$) and sentence ($\mathcal{S}$). In the textual domain, we use SGAE to learn a dictionary ($\mathcal{D}$) that helps to reconstruct sentences in the $\mathcal{S}\rightarrow \mathcal{G} \rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline, where $\mathcal{D}$ encodes the desired language prior; in the vision-language domain, we use the shared $\mathcal{D}$ to guide the encoder-decoder in the $\mathcal{I}\rightarrow \mathcal{G}\rightarrow \mathcal{D} \rightarrow \mathcal{S}$ pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, e.g., our SGAE-based single-model achieves a new state-of-the-art $127.8$ CIDEr-D on the Karpathy split, and a competitive $125.5$ CIDEr-D (c40) on the official server even compared to other ensemble models.

While traditional PIE (pose, illumination and expression) face variations have been well settled by latest methods, a new kind of variation, cross-age variation, is drawing attention from researchers. Most of the existing methods fail to maintain the effectiveness in real world applications that contain significant gap of age. Cross-age variation is caused by the shape deformation and texture changing of human faces while people getting old. It will result in tremendous intra-personal changes of face feature that deteriorate the performance of algorithms. This paper proposed a deep feature based framework for face retrieval problem. Our framework uses deep CNNs feature descriptor and two well designed post-processing methods to achieve age-invariance. To the best of our knowledge, this is the first deep feature based method in cross-age face retrieval problem. The deep CNNs model we use is firstly trained on traditional PIE datasets and then fine-tuned by cross-age dataset. The feature selection and projection post-processing we propose is also proved to be very effective in eliminating cross-age variation of deep CNNs feature. The experiments conducted on Cross-Age Celebrity Dataset (CACD), which is the largest public dataset containing cross-age variation, show that our framework outperforms previous state-of-the-art methods.

Recent works have achieved near or over human performance in traditional face recognition under PIE (pose, illumination and expression) variation. However, few works focus on the cross-age face recognition task, which means identifying the faces from same person at different ages. Taking human-aging into consideration broadens the application area of face recognition. It comes at the cost of making existing algorithms hard to maintain effectiveness. This paper presents a new reference based approach to address cross-age problem, called Eigen-Aging Reference Coding (EARC). Different from other existing reference based methods, our reference traces eigen faces instead of specific individuals. The proposed reference has smaller size and contains more useful information. To the best of our knowledge, we achieve state-of-the-art performance and speed on CACD dataset, the largest public face dataset containing significant aging information.