Search

Image-text bidirectional retrieval is a significant task within cross-modal learning field. The main issue lies on the jointly embedding learning and accurately measuring image-text matching score. Most prior works make use of either intra-modality methods performing within two separate modalities or inter-modality ones combining two modalities tightly. However, intra-modality methods remain ambiguous when learning visual context due to the existence of redundant messages. And inter-modality methods increase the complexity of retrieval because of unifying two modalities closely when learning modal features. In this research, we propose an eclectic Visual Context Learning based on Textual knowledge Network (VCLTN), which transfers textual knowledge to visual modality for context learning and decreases the discrepancy of information capacity between two modalities. Specifically, VCLTN merges label semantics into corresponding regional features and employs those labels as intermediaries between images and texts for better modal alignment. Contextual knowledge of those labels learned within textual modality is utilized to guide the visual context learning. Besides, considering the homogeneity within each modality, global features are merged into regional features for assisting in the context learning. In order to alleviate the imbalance of information capacity between images and texts, entities together with relations inside the given caption are extracted and an auxiliary caption is sampled for attaching supplementary messages to textual modality. Experiments performed on Flickr30K and MS-COCO reveal that our model VCLTN achieves best results compared with the state-of-the-art methods.

Convolutional Neural Networks (CNNs) have played an important role in saliency detection. How to detect a salient object as a whole is a key issue. However, most existing learning-based methods are not accurate enough to detect salient objects in complex scenes, such as easily overlooked small salient areas in a whole salient object, which is called scale imbalance problem in this paper. To address this issue, Scale Balance Network (SBN) based on fully convolutional network is proposed to accurately recognize and comprehensively detect salient objects. Firstly, to detect more small salient areas, a specially designed backbone instead of common backbone is adopted in this paper, which can capture larger resolution with more spatial features in deeper layers. Secondly, we present a novel progressive pyramid mechanism named Connective Feature Pyramid Module (CFPM), aiming to make the network focus on the balance between the large salient areas and the small ones. Finally, we present an Edge Enhancement Architecture with Various Kernels (EEAVK) to locate the saliency maps and refine the boundary features. Experimental results on five benchmark datasets show that the proposed SBN method achieves consistently superior performance in comparison with other state-of-the-art ones under different evaluation metrics.

Visual Dialog aims to answer multi-round, interactive questions based on the dialog history and image content. Existing methods either consider answer ranking and generating individually or only weakly capture the relation across the two tasks implicitly by two separate models. The research on a universal framework that jointly learns to rank and generate answers in a single model is seldom explored. In this paper, we propose a contrastive learning-based framework UTC to unify and facilitate both discriminative and generative tasks in visual dialog with a single model. Specifically, considering the inherent limitation of the previous learning paradigm, we devise two inter-task contrastive losses i.e., context contrastive loss and answer contrastive loss to make the discriminative and generative tasks mutually reinforce each other. These two complementary contrastive losses exploit dialog context and target answer as anchor points to provide representation learning signals from different perspectives. We evaluate our proposed UTC on the VisDial v1.0 dataset, where our method outperforms the state-of-the-art on both discriminative and generative tasks and surpasses previous state-of-the-art generative methods by more than 2 absolute points on Recall@1., Comment: Accepted in CVPR 2022

Color transfer in image processing usually suffers from misleading color mapping and loss of details. This paper presents a novel directive local color transfer method based on dynamic look-up table (D-DLT) to solve these problems in two steps. First, a directive mapping between the source and the reference image is established based on the salient detection and the color clusters to obtain directive color transfer intention. Then, dynamic look-up tables are created according to the color clusters to preserve the details, which can suppress pseudo contours and avoid detail loss. Subjective and objective assessments are presented to verify the feasibility and the availability of the proposed approach. Experimental results demonstrate that our proposed method has better performance on natural color images than classical color transfer algorithms. Furthermore, the reference image can be extended to color blocks instead of images.

Recent great progress has been made on Salient Object Detection (SOD) by deep Convolutional Neural Networks (CNNs). However, most SOD methods still suffer from scale imbalance issue, which pays more attention on large salient areas but ignores small salient areas though they belong to the same object. To address this issue, this paper proposes a Scale Balance Network (SBN) to accurately locate large salient areas and recognize small salient areas. Firstly, a backbone network specifically designed for object detection is adopted in this paper, which captures larger resolution with more spatial features in deeper layers. Secondly, to focus on the balance between the large salient areas and the small salient areas, this paper proposes a novel Connective Feature Pyramid Module (CFPM) for sufficiently leveraging the multi-scale features and the multi-level features, which includes Feature Coherence Enhancement (FCE) and Feature Progressive Extraction (FPE). FCE is designed to enhance the coherence between high-level and low-level features, and FPE is designed to extract the progressive features in different convolutional layers. Finally, an Edge Enhancement Architecture with Various Kernels (EEAVK) is proposed to refine the edge features. Experimental results on five benchmark datasets show that the proposed method outperforms or achieves consistently superior performance in comparison with other methods under different evaluation metrics.

Color transfer is to change the image color according to one or more reference images to match people’s cognition. Usually, there are two problems deserved to be researched further. One is to find suitable reference colors for a complex image. The other is to lessen the details loss in the process. In the paper, GMM model is improved by considering the geometric position relation of the color clustering to construct color transfer intentions between source image and multiple reference images. The geometric coordinate smoothing parameter is introduced to improve EM algorithm to reduce the details loss. Based on Baidu data set and the reference paper data sets, a comparison experiment between typical color transfer algorithms and our method was presented. The experimental results can demonstrate that the algorithm proposed in the paper is reasonable and effective.