Search

VQA attracts lots of researchers in recent years. It could be potentially applied to the remote consultation of COVID-19. Attention mechanisms provide an effective way of utilizing visual and question information selectively in visual question and answering (VQA). The attention methods of existing VQA models generally focus on spatial dimension. In other words, the attention is modeled as spatial probabilities that re-weights the image region or word token features. However, feature-wise attention cannot be ignored, as image and question representations are organized in both spatial and feature-wise modes. Taking the question “What is the color of the woman’s hair” for example, identifying the hair color attribute feature is as important as focusing on the hair region. In this paper, we propose a novel neural network module named “multimodal feature-wise attention module” (MulFA) to model the feature-wise attention. Extensive experiments show that MulFA is capable of filtering representations for feature refinement and leads to improved performance. By introducing MulFA modules, we construct an effective union feature-wise and spatial co-attention network (UFSCAN) model for VQA. Our evaluation on two large-scale VQA datasets, VQA 1.0 and VQA 2.0, shows that UFSCAN achieves performance competitive with state-of-the-art models.

Image semantic segmentation is a basic and challenging computer vision task, where each pixel in an image is classified into a semantic class. In recent years, deep neural networks have settled many computer vision problems such as semantic segmentation, image classification, object detection. A lot of approaches based on deep neural networks have achieved outstanding performance on different benchmarks. However, most of them are restricted by the scale of elaborate labeled data to train a deep neural network, especially on image semantic segmentation. Collecting pixel-level annotated images is an extremely time-consuming process. Thus, utilization of synthetic data is becoming prevalent. Nevertheless, simply applying the models trained on synthetic data leads to a dramatic performance drop on real images due to the domain shift. In this paper, we propose an adversarial learning method for domain adaptation from the perspectives of both image-level and feature-level. The former adapts images from two domains to appear as if drawn from one domain and the latter attempts to learn similarities between the source and target domains in feature-level common space. To further enhance the adapted model, we introduce a self-training strategy using pseudo-labels. Extensive experiments and ablation studies are conducted under various settings and experiment results indicate our unsupervised method outperforms other state-of-the-art methods.

Towards the large scale face search problem, this paper proposes a fast deep face search method which is realized by combination of deep convolution neural network (CNN), semantic hashing, and hash-based similarity search. First of all, to boost the performance in accuracy of face search, the residual network (Resnet) is exploited to construct a deep face feature model and then train it over the cleaned MS-Celeb-1M, which is used to extract real-valued face feature. Next, by imposing PCA and binarization operations, we convert the real-valued feature into compact hash code used for speeding up the face search. Based on the extracted dual features, the face search can be efficiently performed by adopting two-stage matching (i.e., coarse matching and fine matching) strategy. The coarse matching is implemented under the support of efficient hash indexing technique for yielding a small number of candidates while the fine stage is to filter out the unrelated images by cosine distance comparison of real-valued features. It is worth noting that we offer two coarse matching methods, such as GPU-hash and M-index-hash based matching, which are suitable for tens-of million and billion scale scenarios respectively. The experimental results demonstrate that the proposed method is very effective for large scale face search in both aspects of accuracy and real time property.

For person re-identification, the core task is to find effective representations of a person image. As Multi-Task Learning can achieve great performance in seeking robust features, we propose a novel Multi-Task Learning Network (MTNet) with four different losses for person re-identification (re-ID). Our MTNet is an end-to-end deep learning framework, which all the parameters and losses can be jointly optimized. In our method we combine two tasks closely corresponding to person re-identification, pedestrian identity task and pedestrian attribute task, who provide complementary information from different perspective by integrating multi-context informations. Attribute focuses on some special aspects of a person, while identity pays more attention to overall contour and appearance. Meanwhile, both classification and verification losses are employed to optimize the distance of samples. Identification losses are used to construct a large class space, while verification losses are applied optimize the space by minimizing the distance between similar images and maximizing the distance between dissimilar images. In the experiments, our MTNet achieves the state-of-the-art results on two typical datasets Market1501 [1] and DukeMTMC-reID [2].

Overlapping community detection plays an important role in studying social networks. The existing overlapping community detection methods seldom perform well on networks with complex weight distribution. Density peaks clustering (DPC) is capable of finding communities with arbitrary shape efficiently and accurately. However, DPC fails to be applied to overlapping community detection directly. In this paper, we propose an extended adaptive density peaks clustering for overlapping community detection, called EADP. To handle both weighted and unweighted social networks, EADP takes weights into consideration and incorporates a novel distance function based on common nodes to measure the distance between nodes. Moreover, unlike DPC choosing cluster centers by hand, EADP adopts a linear fitting based strategy to choose cluster centers adaptively. Experiments on real-world social networks and synthetic networks show that EADP is an effective overlapping community detection algorithm. Compared with the state-of-the-art methods, EADP performs better on those networks with complex weight distribution.

Pornographic image recognition is a challenging task due to the subjective definition and complex contextual information. In this paper, we propose an ensemble framework using uncertain inference (EFUI) for pornographic image recognition. The EFUI employs bayesian network (BN) as uncertain inference engine, while prior global confidence and uncertain evidence of local semantic components are acquired by deep learning networks. Specifically, we construct the graphical model of BN based on the internal contextual relationship of local semantic components, which conforms to common sense. The prior global confidence of pornography for candidate image is extracted using GoogleNet/ResNet-50. We extract the uncertain evidence of local semantic components, represented by the probability of visual object presence, using Single Shot MultiBox Detector (SSD). Finally, a novel uncertain belief propagation algorithm is introduced to propagate the belief of extracted uncertain evidence until convergence, then identifying the pornographic possibility of the image according to the final confidence. To evaluate EFUI, we employ the public NPDI Pornography database and collect a practical dataset contains 10 million images. Experimental results on all datasets demonstrate that our EFUI achieves the state-of-the-art performance on all evaluation metrics, and outperforms the best counterparts by up 6.95% and 0.61% for accuracy on the two datasets respectively. It also presents a comparable performance over the state-of-the-art approaches in literature.

Sentiment-based text retrieval is an urgent and valuable task due to the explosive growth of sentiment-expressed reviews from social networks like Twitter, Facebook, Instagram, etc. Social networks within the domain of Cognitive Internet of Things (CIoT) make it much easier to dynamically discover desirable services and valuable information. Information retrieval in social media is a daunting task which requires a lot of technical insights. As a powerful tool for large-scale information retrieval, hashing techniques have also been extensively employed for text retrieval. However, most existing text hashing methods are impractical for sentiment-expressed text retrieval mainly for three reasons: (1) the text representations are captured by shallow machine learning algorithms; (2) sentiment is rarely considered when measuring the similarity of two documents; and (3) unsupervised learning of hash functions is employed due to the lack of hash labels. To address these problems, in this paper, we put forward a general deep sentiment hashing model, which is composed of three steps. First, a hierarchical attention-based Long Short-Term Memory network (LSTM) is trained to obtain sentiment-specific document representations. Second, given the document embeddings, k-Nearest Neighbor (kNN) algorithm is used to construct a Laplacian matrix which is projected into hash labels via Laplacian Eigenmaps (LapEig) later. Third, we build a deep model for hash functions learning, which is supervised by both the generated hash labels and the original sentiment labels. Such joint supervision ensures that the ultimate hash codes produced by the learned hash functions maintain sentiment-level similarity. Experimental results turn out that the proposed approach achieves an effective and outstanding retrieval performance.

Learning-based hashing methods are becoming the mainstream for approximate scalable multimedia retrieval. They consist of two main components: hash codes learning for training data and hash functions learning for new data points. Tremendous efforts have been devoted to designing novel methods for these two components, i.e., supervised and unsupervised methods for learning hash codes, and different models for inferring hashing functions. However, there is little work integrating supervised and unsupervised hash codes learning into a single framework. Moreover, the hash function learning component is usually based on hand-crafted visual features extracted from the training images. The performance of a content-based image retrieval system crucially depends on the feature representation and such hand-crafted visual features may degrade the accuracy of the hash functions. In this paper, we propose a semi-supervised deep learning hashing (DLH) method for fast multimedia retrieval. More specifically, in the first component, we utilize both visual and label information to learn an optimal similarity graph that can more precisely encode the relationship among training data, and then generate the hash codes based on the graph. In the second stage, we apply a deep convolutional network to simultaneously learn a good multimedia representation and a set of hash functions. Extensive experiments on five popular datasets demonstrate the superiority of our DLH over both supervised and unsupervised hashing methods.

We present a summarization approach for egocentric video. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer's day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important video shots which reflect high stable salience, discrimination and representativeness. To accomplish this, we utilize egocentric salience cues, motion cues and a selection model to capture stable salience weight, discriminative weight and representative weight of a video shot respectively. We further combine these weights in a unified framework to predict the importance score of a shot, based on which, important shots are selected for the storyboard. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of shots that have never been seen previously. Experimental results on three video datasets across various genres demonstrate that our proposed approach clearly outperforms several state-of-the-art methods.

With the advance of 3D digitalization techniques, it has produced a large number of digital 3D objects, which are usually present in graph, image or video format. In this paper, we focus on designing a novel feature extraction method towards 2D image of 3D object for recognition task. Motivated by the fact that the responses generated by a classifier for two objects can highly reflect their semantic similarity, we attempt to exploit a set of classifiers to construct feature extraction method. The basic idea is as follows. We first learn a classifier for each class and then combine the outputs of all classifiers as object feature. Due to the label information being considered, the proposed method will be more powerful than the typical methods, such as SIFT based bag-of-feature and sparse coding, in terms of discovering the latent semantic information. This is helpful to improve the accuracy of the object recognition. In addition, to make the proposed method scalable to be trained over the massive data (so as to better its generalization ability), the l 2 - norm logistic regression is selected as the classifier and trained with stochastic gradient ascent. At the aspect of time complexity, the proposed method is linear to the number of image pixels and less expensive than the other two methods. These arguments have been demonstrated by the obtained experimental results, which is performed over four 3D datasets, such as COIL-100, 3Ddata, ETH-80 and RGB-D dataset.

It is a NP-hard problem to optimize the objective function of hash-based similarity search algorithms, such as Spectral Hashing and Self-Taught Hashing. To make the problem solvable, existing methods have relaxed the constraints on hash codes from binary values (discrete) to real values (continuous). Then eigenvalue decomposition was employed to achieve the relaxed real solution. The main problem is that the signs of the relaxed continuous solution are mixed. Such results may deviate severely from the true solution, which has lead to significant semantic loss. Moreover, eigenvalue decomposition confronts singularity problem when the dimension of the data is larger than the sample size. To address these problems, we propose a novel method named Nonnegative Sparse Locality Preserving Hashing (NSLPH). Nonnegative and sparse constraints are imposed for a more accurate solution which preserves semantic information well. Then, we have applied nonnegative quadratic programming and multiplicative updating to solve the optimization problem, which successfully avoids the singularity problem of the eigenvalue decomposition. The extensive experiments presented in this paper demonstrate that the proposed approach outperforms the state-of-the-art algorithms.

Understanding the weaknesses and the limitations of existing digital fingerprinting schemes and designing effective collusion attacks play an important role in the development of digital fingerprinting. In this paper, we propose a collusion attack optimization framework for spread-spectrum (SS) fingerprinting. Our framework is based upon the closed-loop feedback control theory. In the framework, we at first define a measure function to test whether the fingerprint presents in the attacked signal after collusion. Then, an optimization mechanism is introduced to attenuate the fingerprints from the forgery. We evaluate the performance of the proposed framework for three different SS-based embedding methods. The experimental results show that the proposed framework is more effective than the other examined collusion attacks. About three pieces of fingerprinted content are able to interrupt the fingerprinting system which accommodates about 1000 users, if we require the detection probability to be less than 0.9. Meanwhile, a high fidelity of the attacked content is retained.

Among the existing hashing methods, spectral hashing (SpH) and self-taught hashing (STH) are considered as the state-of-the-art works. However, two such methods still have some drawbacks. For example, when generating the extension of out-of-sample, SpH makes assumption that data follows uniform distribution but it is impractical. As to STH, its hash functions are obtained by training SVM classifier bit-by-bit, which will lead to ten-fold increase in training time. Moreover, they both suffer overfitting issue. To conquer those drawbacks, we propose a new hashing method, also called LS_SPH, which adopts a unified objective function to obtain the binary embeddings of training objects and hash functions for predicting hash code of test object. Integrating two such processes together will bring in two advantages: (1) It can highly decrease the time complexity of offline stage for training hash codes and hash function due to not requiring extra time for learning hash function. (2) The overfitting issue can be successfully avoided because the empirical loss function associated with hash function is served as the regularization item in objective function in this method. The extensive experiments show that the LS_SPH is superior to the state-of-the-art hashing methods such as SpH and STH on the whole.

Recently the methods based on bag-of-visual words have become very popular in near-duplicate retrieval and content identification. However, obtaining the visual vocabulary by quantization is very time-consuming and unscalable to large databases. In this paper, we propose a fast copy detection method which uses local image fingerprints to define visual words. To construct the fingerprint, a 32-bit vector is extracted from the local description and then converted into a number which is used to define the visual word. Then, a histogram intersection is employed to measure the similarity between two images. Since the fingerprint building is of low-complexity, this method is very efficient and scalable to very large databases. Furthermore, the fingerprint-defined visual words are more discriminative and precise than the clustering-defined visual words because the vocabulary size could be large enough while maintaining high efficiency. Visual words with strong discriminability can distinguish copies from similar objects, which can reduce the number of false positives and improve the precision and efficiency. The evaluation shows that our approach significantly outperforms state-of-the-art methods.

Among the existing hashing methods, the Self-taught hashing (STH) is regarded as the state-of-the-art work. However, it still suffers the problem of semantic loss, which mainly comes from the fact that the original optimization objective of in-sample data is NP-hard and therefore is compromised into the combination of Laplacian Eigenmaps (LE) and binarization. Obviously, the shape associated with the embedding of LE is quite dissimilar to that of binary code. As a result, binarization of the LE embedding readily leads to significant semantic loss. To overcome this drawback, we combine the constrained nonnegative sparse coding and the Support Vector Machine (SVM) to propose a new hashing method, called nonnegative sparse coding induced hashing (NSCIH). Here, nonnegative sparse coding is exploited for seeking a better intermediate representation, which can make sure that the binarization can be smoothly conducted. In addition, we build an image copy detection scheme based on the proposed hashing methods. The extensive experiments show that the NSCIH is superior to the state-of-the-art hashing methods. At the same time, this copy detection scheme can be used for performing copy detection over very large image database.

The robustness of the localized watermarking methods mainly depends on the robustness of the feature locating the watermark. Based on the mean luminance of the disk, a rotation and scale invariant feature extraction algorithm is proposed. A theoretical verification of the rotation and scale invariance of extracted feature points in the continuous image is further performed. The extracted feature points are used to construct rotation and scale invariant circular regions, where the watermark is embedded after affine normalization. Experimental results show that the constructed regions fit the watermarking applications much better than those in previous feature-based watermarking schemes from the aspect of robustness against common attacks including filtering, JPEG compression, cropping, rotation and scaling, and the proposed localized image watermarking scheme has better robustness than previous feature-based watermarking schemes against common signal process and geometrical attacks while maintaining imperceptibility.

This paper proposes a novel robust video watermarking scheme based on local affine invariant features in the compressed domain. This scheme is resilient to geometric distortions and quite suitable for DCT-encoded compressed video data because it performs directly in the block DCTs domain. In order to synchronize the watermark, we use local invariant feature points obtained through the Harris-Affine detector which is invariant to affine distortions. To decode the frames from DCT domain to the spatial domain as fast as possible, a fast inter-transformation between block DCTs and sub-block DCTs is employed and down-sampling frames in the spatial domain are obtained by replacing each sub-blocks DCT of 2x2 pixels with half of the corresponding DC coefficient. The above-mentioned strategy can significantly save computational cost in comparison with the conventional method which accomplishes the same task via inverse DCT (IDCT). The watermark detection is performed in spatial domain along with the decoded video playing. So it is not sensitive to the video format conversion. Experimental results demonstrate that the proposed scheme is transparent and robust to signal-processing attacks, geometric distortions including rotation, scaling, aspect ratio changes, linear geometric transforms, cropping and combinations of several attacks, frame dropping, and frame rate conversion.