Search

As important side information, attributes have been widely exploited in the existing recommender system for better performance. Prior studies usually use a default value (i.e., ``other") to represent the missing attribute, resulting in sub-optimal performance. To address this problem, in this paper, we present an attribute-aware attentive graph convolution network. In particular, we first construct a graph, where users, items, and attributes are three types of nodes and their associations are edges. Thereafter, we leverage the graph convolution network to characterize the complicated interactions among . Furthermore, to learn the node representation, we adopt the message-passing strategy to aggregate the messages passed from the other directly linked types of nodes (e.g., a user or an attribute). Towards this end, we are capable of incorporating associate attributes to strengthen the user and item representation learning, and thus naturally solve the attribute missing problem. Given that for different users, the attributes of an item have different influence on their preference to this item, we design a novel attention mechanism to filter the message passed from an item to a target user by considering the attribute information. Extensive experiments on several publicly accessible datasets demonstrate the superiority of our model over several state-of-the-art methods.

Online learning has become one of the most important learning styles, yet with the need of supervisors to consistently keep the learners motivated and on-task. Some learners could be supervised by outer factors, and distance learners have to be motivated by themselves. However, online learning engagement is hardly to be assessed by supervisors in real time. With the rapid development of information technology, it is able to remedy the above problem by using intelligent video surveillance techniques. In this paper, we propose a novel framework of learning engagement assessment which introduces facial expression recognition to timely acquire the emotional changes of the learners. Moreover, a new facial expression recognition method is proposed based on domain adaptation, which is suitable for the MOOC scenario. The experiments show the effectiveness of our proposed framework on assessing learners’ learning engagement. The comparisons with the state-of-the-art methods also demonstrate the superiority of our proposed facial emotion recognition method.

Recent studies show that hashing technology can achieve efficient similarity searching and many works have been done on supervised deep hash learning. However, under unsupervised scenarios, there are several issues to be solved when learning hashing codes based on visual features for image retrieval and clustering. In this article, we propose a simple but effective Unsupervised Deep K-means Hashing (UDKH) method to simultaneously alleviate the problems of image retrieval and clustering within a single learning framework. UDKH progressively improves the quality of cluster labels and binary hash codes by minimizing pair-wise supervision loss and optimizing the binary K-means to generate discriminative hash codes under the supervision of the learned cluster labels for effective image retrieval. Since the learned hash codes are discriminative, UDKH also improves the image clustering accuracy. Experiments on test datasets demonstrate its effectiveness for image retrieval and clustering.

The creation of improvised dancing choreographies is an important research field of cross-modal analysis. A key point of this task is how to effectively create and correlate music and dance with a probabilistic one-to-many mapping, which is essential to create realistic dances of various genres. To address this issue, we propose a GAN-based cross-modal association framework, DeepDance, which correlates two different modalities (dance motion and music) together, aiming at creating the desired dance sequence in terms of the input music. Its generator is to predictively produce the dance movements best-fit to current music piece by learning from examples. In another hand, its discriminator acts as an external evaluation from the audience and judges the whole performance. The generated dance movements and the corresponding input music are considered to be well-matched if the discriminator cannot distinguish the generated movements from the training samples according to the estimated probability. By adding motion consistency constraints in our loss function, the proposed framework is able to create long realistic dance sequences. To alleviate the problem of expensive and inefficient data collection, we propose an effective approach to create a large-scale dataset, YouTube-Dance3D, from open data source. Extensive experiments on currently available music-dance datasets and our YouTube-Dance3D dataset demonstrate that our approach effectively captures the correlation between music and dance and can be used to choreograph appropriate dance sequences.

Computed tomography (CT) reconstruction for limited-angle projection data is an ill-posed inverse problem, that often produces artifacts near the edges of an image. In this study, a hybrid minimization model based on a nonlocal low-rank approximation and a prior image in a wavelet tight framework is proposed to improve reconstructions from limited-angle projections. Low-frequency wavelet coefficients of the reconstructed image were estimated using a nonlocal low-rank approximation and the $l_{2}$ norm minimization was applied to the difference between the high-frequency components of a prior image and the reconstructed image. In addition, the alternative direction method of multipliers (ADMM) was used for alternately minimization to solve two regularization terms which produce the most parameters. Experimental results demonstrated that the proposed algorithm offers several advantages over conventional iterative reconstruction techniques, including faster convergence, suppression of limited-angle artifacts, noise reduction, and the preservation of edges and other image details. This study represents the first time that nonlocal low-rank prior information has been applied to limited-angle CT.

In this paper, we work towards linking users’ identities on different social media platforms by exploring the user-generated contents (UGCs). This task is non-trivial due to the following challenges. 1) As UGCs involve multiple modalities (e.g., text and image), how to accurately characterize the user account based on their heterogeneous multi-modal UGCs poses the main challenge. 2) As people tend to post similar UGCs on different social media platforms during the same period, how to effectively model the temporal post correlation is a crucial challenge. And 3) no public benchmark dataset is available to support our user identity linkage based on heterogeneous UGCs with timestamps. Towards this end, we present an attentive time-aware user identity linkage scheme, which seamlessly integrates the temporal post correlation modeling and attentive user similarity modeling. To facilitate the evaluation, we create a comprehensive large-scale user identity linkage dataset from two popular social media platforms: Instagram and Twitter. Extensive experiments have been conducted on our dataset and the results verify the effectiveness of the proposed scheme. As a residual product, we have released the dataset, codes, and parameters to facilitate other researchers.

Social networks allow users to actively upload images and descriptive tags, which has led to an explosive growth in the number of social images. Multi-view hashing is an efficient technique for supporting large-scale social image retrieval because of its desirable capabilities of encoding multi-view features into compact binary hash codes with extremely low storage costs and fast retrieval speeds. However, existing methods require multi-view features to be fully paired at both the offline model training and online query stages. This requirement cannot be easily satisfied for social image retrieval, where social images that lack descriptive tags are common in social networks. In this paper, we propose an Unsupervised Adaptive Partial Multi-view Hashing (UAPMH) method to handle the partial-view hashing problem for efficient social image retrieval. Specifically, the shared and view-specific latent representations of fully paired and partial-view images, respectively, are learned separately by an adaptive partial multi-view matrix factorization module within the identical semantic space. In particular, instead of adopting simple fixed view combination weights, we develop a parameter-free weight learning scheme to adaptively learn the weights to capture the view variations and the discriminative capabilities of different views. With such a design, our model can sufficiently exploit the available partial-view samples with separate hash code learning and effectively preserve the latent relations of images and tags in hash codes with semantic space sharing. Moreover, to avoid relaxing errors and improve the learning efficiency, binary hash codes are directly learned in a fast mode with simple and efficient operations. Finally, we extend UAPMH to the supervised learning paradigm as Supervised Adaptive Partial Multi-view Hashing (SAPMH) with the supervision of pair-wise semantic labels to further enhance the discriminative capability of hash codes. The experiments demonstrate the state-of-the-art performance of the proposed approaches on public social image retrieval datasets. Our source codes and testing datasets can be obtained at https://github.com/ChaoqunZheng/APMH .

According to the occlusions, Person reidentification (ReID) can be divided into holistic person ReID and partial person ReID tasks. Occlusions commonly exist in the partial person ReID task but pose or observation perspective changes often occur in the holistic person ReID task; thus, many different network architectures have been designed for each task. However, this approach increases the cost in practice and hinders the development of ReID techniques. To solve this problem, in this work, a unified framework is proposed for holistic/partial person ReID, which can effectively and efficiently address changes in pose or observation perspective and the occlusions in both tasks. In detail, we first employ a fully convolutional network to generate feature maps for an arbitrarily sized image and then use spatial pyramid pooling to obtain its spatial pyramid feature. Thereafter, to efficiently solve the matching problem between the query image and gallery images, we build a deep spatial pyramid feature collaborative reconstruction model (DCR). Experimental results on two partial person ReID datasets and three holistic person ReID datasets demonstrate that DCR outperforms state-of-the-art approaches on both tasks and all datasets. Specifically, it outperforms all competitors with a large margin and achieves an improvement of 9.07% and 5.95% over DSR on Partial ReID and Partial-iLIDS datasets with Rank-1, respectively. Similarly, it also achieves an improvement of 5.08% over VPM on the DukeMTMC-ReID dataset with Rank-1. Additionally, the running time of our method for each query is more than 7× faster than that of DSR or DuATM methods.

Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O(n^2)$ O ( n 2 ) memory cost ( $n$ n is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art hashing techniques, in terms of both model learning efficiency and retrieval accuracy.

Due to the complex and dynamic environment of social media, user generated contents (UGCs) may inadvertently leak users’ personal aspects, such as the personal attributes, relationships and even the health condition, and thus place users at high privacy risks. Limited research efforts, thus far, have been dedicated to the privacy detection from users’ unstructured data (i.e., UGCs). Moreover, existing efforts mainly focus on applying conventional machine learning techniques directly to traditional hand-crafted privacy-oriented features, ignoring the powerful representing capability of the advanced neural networks. In light of this, in this article, we present a fine-grained privacy detection network (GrHA) equipped with graph-regularized hierarchical attentive representation learning. In particular, the proposed GrHA explores the semantic correlations among personal aspects with graph convolutional networks to enhance the regularization for the UGC representation learning, and, hence, fulfil effective fine-grained privacy detection. Extensive experiments on a real-world dataset demonstrate the superiority of the proposed model over state-of-the-art competitors in terms of eight standard metrics. As a byproduct, we have released the codes and involved parameters to facilitate the research community.

Hashing is an effective technique to address the large-scale recommendation problem, due to its high computation and storage efficiency on calculating the user preferences on items. However, existing hashing-based recommendation methods still suffer from two important problems: 1) Their recommendation process mainly relies on the user-item interactions and single specific content feature. When the interaction history or the content feature is unavailable (the cold-start problem), their performance will be seriously deteriorated. 2) Existing methods learn the hash codes with relaxed optimization or adopt discrete coordinate descent to directly solve binary hash codes, which results in significant quantization loss or consumes considerable computation time. In this paper, we propose a fast cold-start recommendation method, called Multi-Feature Discrete Collaborative Filtering (MFDCF), to solve these problems. Specifically, a low-rank self-weighted multi-feature fusion module is designed to adaptively project the multiple content features into binary yet informative hash codes by fully exploiting their complementarity. Additionally, we develop a fast discrete optimization algorithm to directly compute the binary hash codes with simple operations. Experiments on two public recommendation datasets demonstrate that MFDCF outperforms the state-of-the-arts on various aspects.

Graph-based clustering is one of the major clustering methods. Most of it works in three separate steps: 1) similarity graph construction; 2) clustering label relaxing; and 3) label discretization with ${k}$ -means (KM). Such common practice has three disadvantages: 1) the predefined similarity graph is often fixed and may not be optimal for the subsequent clustering; 2) the relaxing process of cluster labels may cause significant information loss; and 3) label discretization may deviate from the real clustering result since KM is sensitive to the initialization of cluster centroids. To tackle these problems, in this paper, we propose an effective discrete optimal graph clustering framework. A structured similarity graph that is theoretically optimal for clustering performance is adaptively learned with a guidance of reasonable rank constraints. Besides, to avoid the information loss, we explicitly enforce a discrete transformation on the intermediate continuous label, which derives a tractable optimization problem with a discrete solution. Furthermore, to compensate for the unreliability of the learned labels and enhance the clustering accuracy, we design an adaptive robust module that learns the prediction function for the unseen data based on the learned discrete cluster labels. Finally, an iterative optimization strategy guaranteed with convergence is developed to directly solve the clustering results. Extensive experiments conducted on both real and synthetic datasets demonstrate the superiority of our proposed methods compared with several state-of-the-art clustering approaches.

Unsupervised cross-modal hashing learns hash codes without dependence on the semantic labels. It has the desirable advantage of well scalability and thus can effectively support large-scale cross-media retrieval. However, how to directly learn discriminative discrete hash codes under the unsupervised learning paradigm is still an open challenging problem. In this paper, we aim to attack this problem by proposing an Unsupervised Deep Cross-modal Hashing with Virtual Label Regression (UDCH-VLR) method. We propose a novel unified learning framework to jointly perform deep hash function training, virtual label learning and regression. Specifically, we learn unified hash codes via collaborative matrix factorization on the multi-modal deep representations to preserve the multi-modal shared semantics. Further, we incorporate the virtual label learning into the objective functions and simultaneously regress the learned virtual labels to the hash codes. Finally, instead of simply exploiting the existing shallow features and relaxing the binary constraints, we devise an alternative optimization strategy to directly update the deep hash functions and discrete binary codes. Under such circumstance, the discriminative capability of hash codes can be progressively enhanced with iterative learning. Extensive experiments on three publicly available cross-media retrieval datasets demonstrate that our approach outperforms state-of-the-art methods.

Multi-modal hashing methods could support efficient multimedia retrieval by combining multi-modal features for binary hash learning at the both offline training and online query stages. However, existing multi-modal methods cannot binarize the queries, when only one or part of modalities are provided. In this article, we propose a novel Flexible Multi-modal Hashing (FMH) method to address this problem. FMH learns multiple modality-specific hash codes and multi-modal collaborative hash codes simultaneously within a single model. The hash codes are flexibly generated according to the newly coming queries, which provide any one or combination of modality features. Besides, the hashing learning procedure is efficiently supervised by the pair-wise semantic matrix to enhance the discriminative capability. It could successfully avoid the challenging symmetric semantic matrix factorization and O ( n 2 ) storage cost of semantic matrix. Finally, we design a fast discrete optimization to learn hash codes directly with simple operations. Experiments validate the superiority of the proposed approach.

Unsupervised video object segmentation aims to automatically segment moving objects over an unconstrained video without any user annotation. So far, only few unsupervised online methods have been reported in the literature, and their performance is still far from satisfactory because the complementary information from future frames cannot be processed under online setting. To solve this challenging problem, in this paper, we propose a novel unsupervised online video object segmentation (UOVOS) framework by construing the motion property to mean moving in concurrence with a generic object for segmented regions. By incorporating the salient motion detection and the object proposal, a pixel-wise fusion strategy is developed to effectively remove detection noises, such as dynamic background and stationary objects. Furthermore, by leveraging the obtained segmentation from immediately preceding frames, a forward propagation algorithm is employed to deal with unreliable motion detection and object proposals. Experimental results on several benchmark datasets demonstrate the efficacy of the proposed method. Compared to state-of-the-art unsupervised online segmentation algorithms, the proposed method achieves an absolute gain of 6.2%. Moreover, our method achieves better performance than the best unsupervised offline algorithm on the DAVIS-2016 benchmark dataset. Our code is available on the project website: https://www.github.com/visiontao/uovos .

Hashing could significantly accelerate large-scale image search by transforming the high-dimensional features into binary Hamming space, where efficient similarity search can be achieved with very fast Hamming distance computation and extremely low storage cost. As an important branch of hashing methods, multi-view hashing takes advantages of multiple features from different views for binary hash learning. However, existing multi-view hashing methods are either based on shallow models which fail to fully capture the intrinsic correlations of heterogeneous views, or unsupervised deep models which suffer from insufficient semantics and cannot effectively exploit the complementarity of view features. In this paper, we propose a novel Deep Collaborative Multi-view Hashing (DCMVH) method to deeply fuse multi-view features and learn multi-view hash codes collaboratively under a deep architecture. DCMVH is a new deep multi-view hash learning framework. It mainly consists of 1) multiple view-specific networks to extract hidden representations of different views, and 2) a fusion network to learn multi-view fused hash code. DCMVH associates different layers with instance-wise and pair-wise semantic labels respectively. In this way, the discriminative capability of representation layers can be progressively enhanced and meanwhile the complementarity of different view features can be exploited effectively. Finally, we develop a fast discrete hash optimization method based on augmented Lagrangian multiplier to efficiently solve the binary hash codes. Experiments on public multi-view image search datasets demonstrate our approach achieves substantial performance improvement over state-of-the-art methods.

Recent studies have pointed out that many well-developed Visual Question Answering (VQA) models are heavily affected by the language prior problem. It refers to making predictions based on the co-occurrence pattern between textual questions and answers instead of reasoning upon visual contents. To tackle this problem, most existing methods focus on strengthening the visual feature learning capability to reduce this text shortcut influence on model decisions. However, few efforts have been devoted to analyzing its inherent cause and providing an explicit interpretation. It thus lacks a good guidance for the research community to move forward in a purposeful way, resulting in model construction perplexity towards overcoming this non-trivial problem. In this paper, we propose to interpret the language prior problem in VQA from a class-imbalance view. Concretely, we design a novel interpretation scheme whereby the loss of mis-predicted frequent and sparse answers from the same question type is distinctly exhibited during the late training phase. It explicitly reveals why the VQA model tends to produce a frequent yet obviously wrong answer, to a given question whose right answer is sparse in the training set. Based upon this observation, we further propose a novel loss re-scaling approach to assign different weights to each answer according to the training data statistics for estimating the final loss. We apply our approach into six strong baselines and the experimental results on two VQA-CP benchmark datasets evidently demonstrate its effectiveness. In addition, we also justify the validity of the class imbalance interpretation scheme on other computer vision tasks, such as face recognition and image classification.

In industrial enterprises, effective retrieval of three-dimensional (3-D) computer-aided design (CAD) models can greatly save time and cost in new product development and manufacturing, thus, many researchers have focused on it. Recently, many view-based 3D model retrieval methods have been proposed and have achieved state-of-the-art performance. However, most of these methods focus on extracting more discriminative view-level features and effectively aggregating the multi-view images of a 3D model, and the latent relationship among these multi-view images is not fully explored. Thus, we tackle this problem from the perspective of exploiting the relationships between patch features to capture long-range associations among multi-view images. To capture associations among views, in this work, we propose a novel patch convolutional neural network (PCNN ) for view-based 3D model retrieval. Specifically, we first employ a CNN to extract patch features of each view image separately. Second, a novel neural network module named PatchConv is designed to exploit intrinsic relationships between neighboring patches in the feature space to capture long-range associations among multi-view images. Then, an adaptive weighted view layer is further embedded into PCNN to automatically assign a weight to each view according to the similarity between each view feature and the view-pooling feature. Finally, a discrimination loss function is employed to extract the discriminative 3D model feature, which consists of softmax loss values generated by the fusion classifier and the specific classifier. Extensive experimental results on two public 3D model retrieval benchmarks, namely, the ModelNet40, and ModelNet10, demonstrate that our proposed PCNN can outperform state-of-the-art approaches, with mAP values of 93.67%, and 96.23%, respectively.

Orthogonal matching pursuit (OMP) is a popular greedy algorithm because of its simplicity and low computational cost. In the application of OMP, one of the most important issue is to design a stopping condition according to some prior information, such as sparsity and noise level. In this paper, we provide a blind stopping condition for OMP in the presence of Gaussian noise. We theoretically analyze the stopping condition in terms of the mutual incoherence and the minimum component signal-to-noise ratio (SNR), with which OMP can stop with correct support recovery. Different from other alternatives, the proposed stopping condition can be explicitly determined regardless of prior information. So it is much suitable for practical applications. In the application of compressive sensing pulse-Doppler radar, we further analyze the support recovery performance of the proposed condition. By separately recovering the targets belonging to different Doppler frequencies, we remarkably reduce the requirement on the minimum component SNR of all targets, and thus our stopping condition is applicable to low SNR scenes. Numerical results illustrate that the proposed condition has comparable performance to the alternative with a priori information.

In recent years, many studies extract aspects from user reviews and integrate them with ratings for improving the recommendation performance. The common aspects mentioned in a user's reviews and a product's reviews indicate indirect connections between the user and product. However, these aspect-based methods suffer from two problems. First, the common aspects are usually very sparse, which is caused by the sparsity of user-product interactions and the diversity of individual users' vocabularies. Second, a user's interests on aspects could be different with respect to different products, which are usually assumed to be static in existing methods. In this paper, we propose an Attentive Aspect-based Recommendation Model (AARM) to tackle these challenges. For the first problem, to enrich the aspect connections between user and product, besides common aspects, AARM also models the interactions between synonymous and similar aspects. For the second problem, a neural attention network which simultaneously considers user, product and aspect information is constructed to capture a user's attention towards aspects when examining different products. Extensive quantitative and qualitative experiments show that AARM can effectively alleviate the two aforementioned problems and significantly outperforms several state-of-the-art recommendation methods on top-N recommendation task., Comment: Camera-ready manuscript for TOIS

Hashing has been widely exploited in the cross-modal retrieval applications in recent years for its low storage cost and high retrieval efficiency. However, most existing cross-modal hashing methods either fail to capture the discriminative semantics of multi-modal data or suffer from the relatively high training cost. To address these limitations, we propose an efficient Discrete Latent Semantic Hashing (DLSH) method. DLSH first learns the latent semantic representations of different modalities, and then projects them into a shared Hamming space to support the scalable cross-modal retrieval. Because DLSH directly correlates the explicit semantic labels with binary codes, it can enhance the discriminative capability of the learned hashing codes. Furthermore, to obtain binary codes, traditional methods often relax the discrete constraint, resulting in relatively high computation cost as well as quantization loss. In contrast, DLSH directly learns the binary codes with an efficient discrete hash optimization, and thus increases efficiency and reduces the quantization loss in hash optimization. Extensive experiments on several public datasets show that, DLSH outperforms several state-of-the-art cross-modal hashing methods.

Graph Convolution Networks (GCNs) manifest great potential in recommendation. This is attributed to their capability on learning good user and item embeddings by exploiting the collaborative signals from the high-order neighbors. Like other GCN models, the GCN based recommendation models also suffer from the notorious over-smoothing problem - when stacking more layers, node embeddings become more similar and eventually indistinguishable, resulted in performance degradation. The recently proposed LightGCN and LR-GCN alleviate this problem to some extent, however, we argue that they overlook an important factor for the over-smoothing problem in recommendation, that is, high-order neighboring users with no common interests of a user can be also involved in the user's embedding learning in the graph convolution operation. As a result, the multi-layer graph convolution will make users with dissimilar interests have similar embeddings. In this paper, we propose a novel Interest-aware Message-Passing GCN (IMP-GCN) recommendation model, which performs high-order graph convolution inside subgraphs. The subgraph consists of users with similar interests and their interacted items. To form the subgraphs, we design an unsupervised subgraph generation module, which can effectively identify users with common interests by exploiting both user feature and graph structure. To this end, our model can avoid propagating negative information from high-order neighbors into embedding learning. Experimental results on three large-scale benchmark datasets show that our model can gain performance improvement by stacking more layers and outperform the state-of-the-art GCN-based recommendation models significantly., Comment: WWW 2021, 10 pages

Localizing the desired video clip for a given query in an untrimmed video has been a hot research topic for multimedia understanding. Recently, a new task named video relocalization, in which the query is a video clip, has been raised. Some methods have been developed for this task, however, these methods often require dense annotations of the temporal boundaries inside long videos for training. A more practical solution is the weakly-supervised approach, which only needs the matching information between the query and video. Motivated by that, we propose a weakly-supervised video relocalization approach based on an attention-based feature matching method. Specifically, it recognizes the video clip by finding the clip whose frames are the most relevant to the query clip frames based on the matching results of the frame embeddings. In addition, an attention module is introduced to identify the frames containing rich semantic correlations in the query video. Extensive experiments on the ActivityNet dataset demonstrate that our method can outperform several weakly-supervised methods consistently and even achieve competing performance to supervised baselines.

Hashing learns compact binary codes to store and retrieve massive data efficiently. Particularly, unsupervised deep hashing is supported by powerful deep neural networks and has the desirable advantage of label independence. It is a promising technique for scalable image retrieval. However, deep models introduce a large number of parameters, which is hard to optimize due to the lack of explicit semantic labels and brings considerable training cost. As a result, the retrieval accuracy and training efficiency of existing unsupervised deep hashing are still limited. To tackle the problems, in this paper, we propose a simple and efficient Lightweight Augmented Graph Network Hashing (LAGNH) method with a two-pronged strategy. For one thing, we extract the inner structure of the image as the auxiliary semantics to enhance the semantic supervision of the unsupervised hash learning process. For another, we design a lightweight network structure with the assistance of the auxiliary semantics, which greatly reduces the number of network parameters that needs to be optimized and thus greatly accelerates the training process. Specifically, we design a cross-modal attention module based on the auxiliary semantic information to adaptively mitigate the adverse effects in the deep image features. Besides, the hash codes are learned by multi-layer message passing within an adversarial regularized graph convolutional network. Simultaneously, the semantic representation capability of hash codes is further enhanced by reconstructing the similarity graph. Experimental results show that our method achieves significant performance improvement compared with the state-of-the-art unsupervised deep hashing methods in terms of both retrieval accuracy and efficiency. Notably, on MS-COCO dataset, our method achieves more than 10% improvement on retrieval precision and 2.7x speedup on training time compared with the second best result.

A number of studies point out that current Visual Question Answering (VQA) models are severely affected by the language prior problem, which refers to blindly making predictions based on the language shortcut. Some efforts have been devoted to overcoming this issue with delicate models. However, there is no research to address it from the angle of the answer feature space learning, despite of the fact that existing VQA methods all cast VQA as a classification task. Inspired by this, in this work, we attempt to tackle the language prior problem from the viewpoint of the feature space learning. To this end, an adapted margin cosine loss is designed to discriminate the frequent and the sparse answer feature space under each question type properly. As a result, the limited patterns within the language modality are largely reduced, thereby less language priors would be introduced by our method. We apply this loss function to several baseline models and evaluate its effectiveness on two VQA-CP benchmarks. Experimental results demonstrate that our adapted margin cosine loss can greatly enhance the baseline models with an absolute performance gain of 15\% on average, strongly verifying the potential of tackling the language prior problem in VQA from the angle of the answer feature space learning.

In this work, we target cross-domain action recognition (CDAR) in the video domain and propose a novel end-to-end pairwise two-stream ConvNets (PTC) algorithm for real-life conditions, in which only a few labeled samples are available. To cope with the limited training sample problem, we employ pairwise network architecture that can leverage training samples from a source domain and, thus, requires only a few labeled samples per category from the target domain. In particular, a frame self-attention mechanism and an adaptive weight scheme are embedded into the PTC network to adaptively combine the RGB and flow features. This design can effectively learn domain-invariant features for both the source and target domains. In addition, we propose a sphere boundary sample-selecting scheme that selects the training samples at the boundary of a class (in the feature space) to train the PTC model. In this way, a well-enhanced generalization capability can be achieved. To validate the effectiveness of our PTC model, we construct two CDAR data sets (SDAI Action I and SDAI Action II) that include indoor and outdoor environments; all actions and samples in these data sets were carefully collected from public action data sets. To the best of our knowledge, these are the first data sets specifically designed for the CDAR task. Extensive experiments were conducted on these two data sets. The results show that PTC outperforms state-of-the-art video action recognition methods in terms of both accuracy and training efficiency. It is noteworthy that when only two labeled training samples per category are used in the SDAI Action I data set, PTC achieves 21.9% and 6.8% improvement in accuracy over two-stream and temporal segment networks models, respectively. As an added contribution, the SDAI Action I and SDAI Action II data sets will be released to facilitate future research on the CDAR task.

With rising awareness of environment protection and recycling, second-hand trading platforms have attracted increasing attention in recent years. The interaction data on second-hand trading platforms, consisting of sufficient interactions per user but rare interactions per item, is different from what they are on traditional platforms. Therefore, building successful recommendation systems in the second-hand trading platforms requires balancing modeling items? and users? preference, and mitigating the adverse effects of the sparsity, which makes recommendation especially challenging. Accordingly, we proposed a method to simultaneously learn representations of items and users from coarse-grained and fine-grained features, and a multi-task learning strategy is designed to address the issue of data sparsity. Experiments conducted on a real-world second-hand trading platform dataset demonstrate the effectiveness of our proposed model.

In real person re-identification (ReID) tasks, pedestrians are often obscured by other pedestrians or objects; moreover, changes in poses or observation perspectives also commonly exist in partial-person ReID. To the best of our knowledge, few works simultaneously focus on these two issues. In this work, we propose a novel texture semantic alignment (TSA) approach with the visibility-aware for partial person ReID task where the occlusion issue and changes in poses are simultaneously explored in an end-to-end unified framework. Specifically, we first employ a texture alignment scheme with the semantic visibility of a person's image to solve the issue of changes in poses that can enhance the alignment and generalization capability of the models. Second, we design a human pose-based partial region alignment scheme to solve the occlusion problem that makes TSA method emphasize the shared body parts. Finally, these two networks jointly learn these aspects. Extensive experimental results demonstrate that our proposed TSA method is very effective and robust for simultaneously handling occlusion and changes in pose, and it can outperform state-of-the-art approaches by a large margin and achieves an improvement of 5% and 6.4% on the rank-1 accuracy over the visibility-aware part model (VPM) method (published in CVPR 2019) on the Partial ReID and Partial-iLIDS datasets, respectively.

Video moment retrieval targets at retrieving a moment in a video for a given language query. The challenges of this task include 1) the requirement of localizing the relevant moment in an untrimmed video, and 2) bridging the semantic gap between textual query and video contents. To tackle those problems, early approaches adopt the sliding window or uniform sampling to collect video clips first and then match each clip with the query. Obviously, these strategies are time-consuming and often lead to unsatisfied accuracy in localization due to the unpredictable length of the golden moment. To avoid the limitations, researchers recently attempt to directly predict the relevant moment boundaries without the requirement to generate video clips first. One mainstream approach is to generate a multimodal feature vector for the target query and video frames (e.g., concatenation) and then use a regression approach upon the multimodal feature vector for boundary detection. Although some progress has been achieved by this approach, we argue that those methods have not well captured the cross-modal interactions between the query and video frames. In this paper, we propose an Attentive Cross-modal Relevance Matching (ACRM) model which predicts the temporal boundaries based on an interaction modeling. In addition, an attention module is introduced to assign higher weights to query words with richer semantic cues, which are considered to be more important for finding relevant video contents. Another contribution is that we propose an additional predictor to utilize the internal frames in the model training to improve the localization accuracy. Extensive experiments on two datasets TACoS and Charades-STA demonstrate the superiority of our method over several state-of-the-art methods. Ablation studies have been also conducted to examine the effectiveness of different modules in our ACRM model., 12 pages; accepted by IEEE TMM

View-based 3D model retrieval has become an important task in both computer vision and machine learning domains. Although deep learning methods have achieved excellent performances on view-based 3D model retrieval, the intrinsic correlation and the degree of view discrimination among multiple views in a 3D model have not been effectively exploited. To obtain a more efficient feature descriptor for 3D model retrieval, in this work, we propose the pairwise view weighted graph network (abbreviated PVWGN) for view-based 3D model retrieval where non-local graph layers are embedded into the network architecture to automatically mine the intrinsic relationship among multiple views of a 3D model. Furthermore, the view weighted layer is employed in the PVWGN to adaptively assign the weight to each view according to its aggregation information. In addition, the pairwise discrimination loss function is designed to improve the feature discrimination of the 3D model. Most importantly, these three issues are integrated into a unified framework. Extensive experimental results on the ModelNet40 and ModelNet10 3D model retrieval datasets show that PVWGN can outperform all state-of-the-art methods on the 3D model retrieval task with mAPs of 93.2% and 96.2%, respectively.

Sparse array arrangement has been widely used in vector-sensor arrays because of increased degree-of-freedoms for identifying more sources than sensors. For large-size sparse vector-sensor arrays, one-bit measurements can further reduce the receiver system complexity by using low-resolution ADCs. In this paper, we present a sparse cross-dipole array with one-bit measurements to estimate Direction of Arrivals (DOA) of electromagnetic sources. Based on the independence assumption of sources, we establish the relation between the covariance matrix of one-bit measurements and that of unquantized measurements by Bussgang Theorem. Then we develop a Spatial-Smooth MUSIC (SS-MUSIC) based method, One-Bit MUSIC (OB-MUSIC), to estimate the DOAs. By jointly utilizing the covariance matrices of two dipole arrays, we find that OB-MUSIC is robust against polarization states. We also derive the Cramer-Rao bound (CRB) of DOA estimation for the proposed scheme. Furthermore, we theoretically analyze the applicability of the independence assumption of sources, which is the fundamental of the proposed and other typical methods, and verify the assumption in typical communication applications. Numerical results show that, with the same number of sensors, one-bit sparse cross-dipole arrays have comparable performance with unquantized uniform linear arrays and thus provide a compromise between the DOA estimation performance and the system complexity.

Social network stores and disseminates a tremendous amount of user shared images. Deep hashing is an efficient indexing technique to support large-scale social image retrieval, due to its deep representation capability, fast retrieval speed and low storage cost. Particularly, unsupervised deep hashing has well scalability as it does not require any manually labelled data for training. However, owing to the lacking of label guidance, existing methods suffer from severe semantic shortage when optimizing a large amount of deep neural network parameters. Differently, in this paper, we propose a Dual-level Semantic Transfer Deep Hashing (DSTDH) method to alleviate this problem with a unified deep hash learning framework. Our model targets at learning the semantically enhanced deep hash codes by specially exploiting the user-generated tags associated with the social images. Specifically, we design a complementary dual-level semantic transfer mechanism to efficiently discover the potential semantics of tags and seamlessly transfer them into binary hash codes. On the one hand, instance-level semantics are directly preserved into hash codes from the associated tags with adverse noise removing. Besides, an image-concept hypergraph is constructed for indirectly transferring the latent high-order semantic correlations of images and tags into hash codes. Moreover, the hash codes are obtained simultaneously with the deep representation learning by the discrete hash optimization strategy. Extensive experiments on two public social image retrieval datasets validate the superior performance of our method compared with state-of-the-art hashing methods. The source codes of our method can be obtained at https://github.com/research2020-1/DSTDH, Comment: Accepted by IEEE TCSVT

Graph-based multi-view feature extraction has attracted much attention in literature. However, conventional solutions generally rely on a manually defined affinity graph matrix, which is hard to capture the intrinsic sample relations in multiple views. In addition, the graph construction and feature extraction are separated into two independent processes which may result in sub-optimal results. Furthermore, the raw data may contain adverse noises that reduces the reliability of the affinity matrix. In this paper, we propose a novel Unsupervised Multi-view Feature Extraction with Dynamic Graph Learning (UMFE-DGL) to solve these limitations. We devise a unified learning framework which simultaneously performs dynamic graph learning and the feature extraction. Dynamic graph learning adaptively captures the intrinsic multiple view-specific relations of samples. Feature extraction learns the projection matrix that could accordingly preserve the dynamically adjusted sample relations modelled by graph into the low-dimensional features. Experimental results on several public datasets demonstrate the superior performance of the proposed approach, compared with state-of-the-art techniques.

We proposed a music auto-tagging approach based on the latent space modeling both for music context and content. First, we introduce the latent semantic analysis for music tags with Sparse Nonnegative Matrix Factorization. Then the music contents semantics will be learnt by decomposing the music content into a pre-trained dictionary and an adaptive dictionary learning algorithm is proposed. Finally, the two latent spaces will be associated with a certain subspace mapping algorithm. The experimental results show that our proposed approach outperforms the state-of-the-art auto-tagging systems when applied to the CAL500 dataset in the 5-fold cross-validation experiments.

Hashing compresses high-dimensional features into compact binary codes. It is one of the promising techniques to support efficient mobile image retrieval, due to its low data transmission cost and fast retrieval response. However, most of existing hashing strategies simply rely on low-level features. Thus, they may generate hashing codes with limited discriminative capability. Moreover, many of them fail to exploit complex and high-order semantic correlations that inherently exist among images. Motivated by these observations, we propose a novel unsupervised hashing scheme, called topic hypergraph hashing (THH), to address the limitations. THH effectively mitigates the semantic shortage of hashing codes by exploiting auxiliary texts around images. In our method, relations between images and semantic topics are first discovered via robust collective non-negative matrix factorization. Afterwards, a unified topic hypergraph, where images and topics are represented with independent vertices and hyperedges, respectively, is constructed to model inherent high-order semantic correlations of images. Finally, hashing codes and functions are learned by simultaneously enforcing semantic consistence and preserving the discovered semantic relations. Experiments on publicly available datasets demonstrate that THH can achieve superior performance compared with several state-of-the-art methods, and it is more suitable for mobile image retrieval.

As an emerging technology to support scalable content-based image retrieval (CBIR), hashing has recently received great attention and became a very active research domain. In this study, we propose a novel unsupervised visual hashing approach called semantic-assisted visual hashing (SAVH). Distinguished from semi-supervised and supervised visual hashing, its core idea is to effectively extract the rich semantics latently embedded in auxiliary texts of images to boost the effectiveness of visual hashing without any explicit semantic labels. To achieve the target, a unified unsupervised framework is developed to learn hash codes by simultaneously preserving visual similarities of images, integrating the semantic assistance from auxiliary texts on modeling high-order relationships of inter-images, and characterizing the correlations between images and shared topics. Our performance study on three publicly available image collections: Wiki , MIR Flickr , and NUS-WIDE indicates that SAVH can achieve superior performance over several state-of-the-art techniques.

Intelligent travel guide systems have grown increasingly popular in recent years. They also benefit a lot from the development of social media, resulting in a large amount of attractions uploaded by users. To tackle this, attractions should be real time classified by user-generated photos automatically to gain better user experience. However, in practice, the given label of photos and text ratings may be incomplete or missing. Moreover, recently, domain adaptation has been applied to deal with few labeled data. Thus, in this paper, we propose a novel framework for automatically attraction classification in leveraging web-harvesting data from search engine and the photos of attractions uploaded by users. Specifically, we assume that top-k web-harvesting images from search engines have correct labels. The classification problem is formulated as a regularized domain adaptation approach. Experiments conducted on the collected real-world data set demonstrated that the promising performance is gained over state-of-the art classification methods.

Hashtags, a user provides to a micro-video, are the ones which can well describe the semantics of the micro-video's content in his/her mind. At the same time, hashtags have been widely used to facilitate various micro-video retrieval scenarios (e.g., search, browse, and categorization). Despite their importance, numerous micro-videos lack hashtags or contain inaccurate or incomplete hashtags. In light of this, hashtag recommendation, which suggests a list of hashtags to a user when he/she wants to annotate a post, becomes a crucial research problem. However, little attention has been paid to micro-video hashtag recommendation, mainly due to the following three reasons: 1) lack of benchmark dataset; 2) the temporal and multi-modality characteristics of micro-videos; and 3) hashtag sparsity and long-tail distributions. In this paper, we recommend hashtags for micro-videos by presenting a novel multi-view representation interactive embedding model with graph-based information propagation. It is capable of boosting the performance of micro-videos hashtag recommendation by jointly considering the sequential feature learning, the video-user-hashtag interaction, and the hashtag correlations. Extensive experiments on a constructed dataset demonstrate our proposed method outperforms state-of-the-art baselines. As a side research contribution, we have released our dataset and codes to facilitate the research in this community.

Personalized hashtag recommendation methods aim to suggest users hashtags to annotate, categorize, and describe their posts. The hashtags, that a user provides to a post (e.g., a micro-video), are the ones which in her mind can well describe the post content where she is interested in. It means that we should consider both users' preferences on the post contents and their personal understanding on the hashtags. Most existing methods rely on modeling either the interactions between hashtags and posts or the interactions between users and hashtags for hashtag recommendation. These methods have not well explored the complicated interactions among users, hashtags, and micro-videos. In this paper, towards the personalized micro-video hashtag recommendation, we propose a Graph Convolution Network based Personalized Hashtag Recommendation (GCN-PHR) model, which leverages recently advanced GCN techniques to model the complicate interactions among and learn their representations. In our model, the users, hashtags, and micro-videos are three types of nodes in a graph and they are linked based on their direct associations. In particular, the message-passing strategy is used to learn the representation of a node (e.g., user) by aggregating the message passed from the directly linked other types of nodes (e.g., hashtag and micro-video). Because a user is often only interested in certain parts of a micro-video and a hashtag is typically used to describe the part (of a micro-video) that the user is interested in, we leverage the attention mechanism to filter the message passed from micro-videos to users and hashtags, which can significantly improve the representation capability. Extensive experiments have been conducted on two real-world micro-video datasets and demonstrate that our model outperforms the state-of-the-art approaches by a large margin.

Multi-modal hashing fuses multi-modal features at both offline training and online query stage for compact binary hash learning. It has aroused extensive attention in research filed of efficient large-scale multimedia retrieval. However, existing methods adopt batch-based learning scheme or unsupervised learning paradigm. They cannot efficiently handle the very common online streaming multi-modal data (for batch-learning methods), or learn the hash codes suffering from limited discriminative capability and less flexibility for varied streaming data (for existing online multi-modal hashing methods). In this paper, we develop a supervised Flexible Online Multi-modal Hashing (FOMH) method to adaptively fuse heterogeneous modalities and flexibly learn the discriminative hash code for the newly coming data, even if part of the modalities is missing. Specifically, instead of adopting the fixed weights, the modalities weights in FOMH are automatically learned with the proposed flexible multi-modal binary projection to timely capture the variations of streaming samples. Further, we design an efficient asymmetric online supervised hashing strategy to enhance the discriminative capability of the hash codes, while avoiding the challenging symmetric semantic matrix decomposition and storage cost. Moreover, to support fast hash updating and avoid the propagation of binary quantization errors in online learning process, we propose to directly update the hash codes with an efficient discrete online optimization. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method from various aspects.

Human action analysis and understanding in videos is an important and challenging task. Although substantial progress has been made in past years, the explainability of existing methods is still limited. In this work, we propose a novel action reasoning framework that uses prior knowledge to explain semantic-level observations of video state changes. Our method takes advantage of both classical reasoning and modern deep learning approaches. Specifically, prior knowledge is defined as the information of a target video domain, including a set of objects, attributes and relationships in the target video domain, as well as relevant actions defined by the temporal attribute and relationship changes (i.e. state transitions). Given a video sequence, we first generate a scene graph on each frame to represent concerned objects, attributes and relationships. Then those scene graphs are associated by tracking objects across frames to form a spatio-temporal graph (also called video graph), which represents semantic-level video states. Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans. Compared to previous works, the action reasoning results of our method can be explained by both logical rules and semantic-level observations of video content changes. Besides, the proposed method can be used to detect multiple concurrent actions with detailed information, such as who (particular objects), when (time), where (object locations) and how (what kind of changes). Experiments on a re-annotated dataset CAD-120 show the effectiveness of our method.

Partial person re-identification (ReID) is a hot research problem in computer vision. Accurate partial ReID is very challenging due to the common occlusion problem. To address this problem, in this paper, we propose a novel D eep spatial pyramid feature C ollaborative R econstruction approach (DCR ) for partial person ReID, which can effectively and efficiently tackle the occlusion in arbitrary sizes. Specifically, a fully convolutional network (FCN) is first leveraged to extract feature maps of an arbitrary-size image, and then the spatial pyramid pooling (SPP) is adopted to obtain spatial pyramid features. Thereafter, our DCR method is designed to efficiently solve the matching problem between the partial person and the holistic person in the partial person ReID task where the occlusion problem often occurs. Experiments on two partial person ReID datasets demonstrate the efficiency and efficacy of the proposed method by comparing to several state-of-the-art partial person ReID approaches. Our method outperforms all the competitors with a large margin and can achieve an improvement of 9.07% and 5.95% over the DSR method on the Partial REID and Partial-iLIDS Person ReID datasets in terms of the Rank-1 accuracy, respectively.

Multi-modal hashing is an effective technique to support large-scale multimedia retrieval, due to its capability of encoding heterogeneous multi-modal features into compact and similarity-preserving binary codes. Although great progress has been achieved so far, existing methods still suffer from several problems, including: 1) All existing methods simply adopt fixed modality combination weights in online hashing process to generate the query hash codes. This strategy cannot adaptively capture the variations of different queries. 2) They either suffer from insufficient semantics (for unsupervised methods) or require high computation and storage cost (for the supervised methods, which rely on pair-wise semantic matrix). 3) They solve the hash codes with relaxed optimization strategy or bit-by-bit discrete optimization, which results in significant quantization loss or consumes considerable computation time. To address the above limitations, in this paper, we propose an Online Multi-modal Hashing with Dynamic Query-adaption (OMH-DQ) method in a novel fashion. Specifically, a self-weighted fusion strategy is designed to adaptively preserve the multi-modal feature information into hash codes by exploiting their complementarity. The hash codes are learned with the supervision of pair-wise semantic labels to enhance their discriminative capability, while avoiding the challenging symmetric similarity matrix factorization. Under such learning framework, the binary hash codes can be directly obtained with efficient operations and without quantization errors. Accordingly, our method can benefit from the semantic labels, and simultaneously, avoid the high computation complexity. Moreover, to accurately capture the query variations, at the online retrieval stage, we design a parameter-free online hashing module which can adaptively learn the query hash codes according to the dynamic query contents. Extensive experiments demonstrate the state-of-the-art performance of the proposed approach from various aspects.

Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years. Although some progress has been achieved so far, several studies have pointed out that current VQA models are heavily affected by the language prior problem, which means they tend to answer questions based on the co-occurrence patterns of question keywords (e.g., how many) and answers (e.g., 2) instead of understanding images and questions. Existing methods attempt to solve this problem by either balancing the biased datasets or forcing models to better understand images. However, only marginal effects and even performance deterioration are observed for the first and second solution, respectively. In addition, another important issue is the lack of measurement to quantitatively measure the extent of the language prior effect, which severely hinders the advancement of related techniques. In this paper, we make contributions to solve the above problems from two perspectives. Firstly, we design a metric to quantitatively measure the language prior effect of VQA models. The proposed metric has been demonstrated to be effective in our empirical studies. Secondly, we propose a regularization method (i.e., score regularization module) to enhance current VQA models by alleviating the language prior problem as well as boosting the backbone model performance. The proposed score regularization module adopts a pair-wise learning strategy, which makes the VQA models answer the question based on the reasoning of the image (upon this question) instead of basing on question-answer patterns observed in the biased training set. The score regularization module is flexible to be integrated into various VQA models. We conducted extensive experiments over two popular VQA datasets (i.e., VQA 1.0 and VQA 2.0) and integrated the score regularization module into three state-of-the-art VQA models. Experimental results show that the score regularization module can not only effectively reduce the language prior problem of these VQA models but also consistently improve their question answering accuracy.

In this paper, we investigate the research problem of unsupervised multi-view feature selection. Conventional solutions first simply combine multiple pre-constructed view-specific similarity structures into a collaborative similarity structure, and then perform the subsequent feature selection. These two processes are separate and independent. The collaborative similarity structure remains fixed during feature selection. Further, the simple undirected view combination may adversely reduce the reliability of the ultimate similarity structure for feature selection, as the view-specific similarity structures generally involve noises and outlying entries. To alleviate these problems, we propose an adaptive collaborative similarity learning (ACSL) for multi-view feature selection. We propose to dynamically learn the collaborative similarity structure, and further integrate it with the ultimate feature selection into a unified framework. Moreover, a reasonable rank constraint is devised to adaptively learn an ideal collaborative similarity structure with proper similarity combination weights and desirable neighbor assignment, both of which could positively facilitate the feature selection. An effective solution guaranteed with the proved convergence is derived to iteratively tackle the formulated optimization problem. Experiments demonstrate the superiority of the proposed approach.

Efficiency is crucial to the online recommender systems. Representing users and items as binary vectors for Collaborative Filtering (CF) can achieve fast user-item affinity computation in the Hamming space, in recent years, we have witnessed an emerging research effort in exploiting binary hashing techniques for CF methods. However, CF with binary codes naturally suffers from low accuracy due to limited representation capability in each bit, which impedes it from modeling complex structure of the data. In this work, we attempt to improve the efficiency without hurting the model performance by utilizing both the accuracy of real-valued vectors and the efficiency of binary codes to represent users/items. In particular, we propose the Compositional Coding for Collaborative Filtering (CCCF) framework, which not only gains better recommendation efficiency than the state-of-the-art binarized CF approaches but also achieves even higher accuracy than the real-valued CF method. Specifically, CCCF innovatively represents each user/item with a set of binary vectors, which are associated with a sparse real-value weight vector. Each value of the weight vector encodes the importance of the corresponding binary vector to the user/item. The continuous weight vectors greatly enhances the representation capability of binary codes, and its sparsity guarantees the processing speed. Furthermore, an integer weight approximation scheme is proposed to further accelerate the speed. Based on the CCCF framework, we design an efficient discrete optimization algorithm to learn its parameters. Extensive experiments on three real-world datasets show that our method outperforms the state-of-the-art binarized CF methods (even achieves better performance than the real-valued CF method) by a large margin in terms of both recommendation accuracy and efficiency., Comment: SIGIR2019

At present, rural areas in China have problems such as poor living environment, shortage of facilities, and industrial disappointment. They still use the concept and strategy of urban planning and design to manage the rural areas, which leads to the rural planning results become empty shells, which is time-consuming and laborious, but does not bring practical use to the improvement of rural life. The purpose of this paper is to better implement the rural revitalization strategy, more scientifically guide the beautiful rural construction, and more effectively restore the rural vitality. Based on the strategy of beautiful village planning under the background of rural revitalization strategy, this paper takes Shuanglongqiao Village, Fengming Town, Xichong County, Sichuan Province as the empirical research object, and comprehensively applies research methods such as literature inquiry, on-site investigation and comparative analysis. Firstly, the paper puts forward the background of rural revitalization strategy and the concept of beautiful rural planning. Based on the theory and practice of beautiful rural planning at home and abroad, it analyzes the guiding significance and enlightenment of these theories and practices on beautiful rural planning. And further reflect on the concept and strategy of planning and construction of beautiful villages; then, in combination with national policies, profoundly analyze the scientific connotation of rural revitalization strategy, on this basis, comprehensively analyze the requirements of rural revitalization strategy for beautiful rural construction, and then combine with rural planning The characteristics of the compilation system of urban planning, and the basic strategies of beautiful rural planning under the background of rural revitalization strategy. Finally, combined with the characteristics of the village and the construction needs, the paper summarizes the planning strategies suitable for the development needs and village characteristics of Shuanglongqiao Village under the background of rural revitalization strategy. On the basis of previous studies, this paper comprehensively interprets the scientific connotation of rural revitalization strategy, and conducts summative research on beautiful rural planning. It hopes to refine some beautiful rural planning concepts and strategies, and other similar Shuanglongqiao villages under the background of rural revitalization strategy. The beautiful village planning provides a new perspective and also provides some experience in building practices in other villages.

Hashing is an effective technique to improve the efficiency of large-scale recommender system by representing both users and items into binary codes. However, existing hashing-based recommendation methods still suffer from two important problems: 1) Cold-start. They employ the user-item interactions and single auxiliary information to learn the binary hash codes. But the full interaction history is not always available and the single auxiliary information may be missing. 2) Efficient optimization. They learn the hash codes with two-step relaxed optimization or one-step discrete hash optimization based on the cyclic coordinate descent, which results in significant quantization loss or still consumes considerable computation time. In this paper, we propose a Multi-modal Discrete Collaborative Filtering (MDCF) for efficient cold-start recommendation. We map the multi-modal features of users and items to a consensus Hamming space based on the matrix factorization framework. Specifically, a low-rank self-weighted multi-modal fusion module is designed to adaptively fuse the multi-modal features into binary hash codes. Additionally, to support large-scale recommendation, a fast discrete optimization method based on augmented Lagrangian multiplier is developed to directly compute the binary hash codes with simple operations. Experiments show the superior performance of the proposed method over state-of-the-art baselines.

High quality test collections have been becoming more and more important for the technological advancement in geo-referenced image retrieval and analytics. In this paper, we present a large scale test collection to support robust performance evaluation of landmark image search and corresponding construction methodology. Using the approach, we develop a very large scale test collection consisting of three key components: (1) 355,141 images of 128 landmarks in five cities across three continents crawled from Flickr; (2) different kinds of textual features for each image, including surrounding text (e.g. tags), contextual data (e.g. geo-location and upload time), and metadata (e.g. uploader and EXIF); and (3) six types of low-level visual features. In order to support robust and effective performance assessment, a series of baseline experimental studies have been conducted on the search performance over both textual and visual queries. The results demonstrate importance and effectiveness of the test collection. HighlightsA comprehensive methodology and procedure are developed to construct very large landmark search test collection.A large scale benchmarking test collection has been developed for effective, reliable and robust evaluation and comparisons of landmark image search systems.Extensive experiments have been conducted using CBIR and TBIR methods to provide baseline results and insights for advanced system development.