Search

As an extended model of data envelopment analysis (DEA), cross-efficiency is utilized in various fields. It applies the relative evaluation values of homogeneous decision-making units (DMUs) in ranking completely. Due to the non-uniqueness of DEA optimal weights, and the cross-efficiency methods focus on the aggregation of cross-efficiency matrix values. There is a low correction between the final efficiency evaluation value and the value assigned to the criteria weight or the value of the criteria. This paper uses the modified aggressive cross-efficiency model to calculate the weights of the decision-making units (DMUs). Then according to the consistency of peer-evaluation criteria, the entropy weight method is used to aggregate the index weights to generate a set of common weights, and a new Measurement of Alternatives and Ranking according to Compromise Solution (MARCOS) method is used to calculate the utility score of DMUs that can rank them completely. Finally, a numerical case of supplier selection is offered to illustrate the feasibility and effectiveness of the proposed method.

In this article, the gained and lost dominance score (GLDS) method is extended into the 2-tuple linguistic neutrosophic environment, which also combined the power aggregation operator with the evaluation information to deal with the multi-attribute group decision-making problem. Since the power aggregation operator can eliminate the effects of extreme evaluating data from some experts with prejudice, this paper further proposes the 2-tuple linguistic neutrosophic numbers power-weighted average operator and 2-tuple linguistic neutrosophic numbers power-weighted geometric operator to aggregate the decision makers’ evaluation. Moreover, a model based on the score function and distance measure of 2-tuple linguistic neutrosophic numbers (2TLNNs) is developed to get the criteria weights. Combing the GLDS method with 2-tuple linguistic neutrosophic numbers and developing a 2TLNN-GLDS method for multiple attribute group decision making, it can express complex fuzzy information more conveniently in a qualitative environment and also consider the dominance relations between alternatives which can get more effective results in real decision-making problems. Finally, an applicable example of selecting the optimal low-carbon logistics park site is given. The comparing results show that the proposed method outperforms the other existing methods, as it can get more reasonable results than others and it is more convenient and effective to express uncertain information in solving realistic decision-making problems.

The natural concept ontology structure of clothes has enabled easy management of large quantities of fashion images for online retailers and it is meaningful to study how to automatically recognize fashion images for both commercial promotion and academic research. In this paper, a new hierarchical approach is developed for large-scale fashion recognition. We first embed concept ontology into deep convolutional neural network (CNN) by adopting multiple deep CNN branches to learn node-specific features and classifiers explicitly. Then, we introduce a hierarchical knowledge distillation method to further improve the performance of fashion recognition. Finally, we employ the proposed approach for fashion recommendation. To deal with hierarchical deep learning constrains, we leverage back propagation to simultaneously refine the shared deep CNNs and the diverse CNN branches for relevant node features and classifiers by using our joint objective function. The main advantages of this paper lie in (1) providing an effective way for recognizing rich semantic explanations of fashion images without training large or multiple networks, and (2) saving the storage&time costs by learning personalized features and classifiers for each tree node. The experimental results on both our organized fashion dataset and the public DeepFashion dataset have verified the effectiveness and efficiency of the proposed approach on both hierarchical fashion recognition and within category fine-grained fashion recommendation.

To generate an image caption, firstly, the content of the image should be fully understood; and then the semantic information contained in the image should be described using a phrase or statement that conforms to certain grammatical rules. Thus, it requires techniques from both computer vision and natural language processing to connect the two different media forms together, which is highly challenging. To adaptively adjust the effect of visual information and language information on the captioning process, in this paper, the part of speech information is proposed to novelly integrate with image captioning models based on the encoder-decoder framework. First, a part of speech prediction network is proposed to analyze and model the part of speech sequences for the words in natural language sentences; then, different mechanisms are proposed to integrate the part of speech guidance information with merge-based and inject-based image captioning models, respectively; finally, according to the integrated frameworks, a multi-task learning paradigm is proposed to facilitate model training. Experiments are conducted on two widely used image captioning datasets, Flickr30 k and COCO, and the results have validated that the image captions generated by the proposed method contain more accurate visual information and comply with language habits and grammar rules better.

In this paper, a novel fusion method is proposed to deal with large-scale plant species identification by aggregating diverse outputs from multiple deep networks, where each deep network focus on one subset of the whole plant species. Firstly, a fixed plant taxonomy is constructed for organizing large number of fine-grained plant species hierarchically and it is further used as a guideline to help generating diverse but overlapped task groups. Secondly, an attention-based deep hierarchical multi-task learning (AHMTL) algorithm is proposed to recognize fine-grained plant species belonging to the same task group effectively by learning more discriminative deep features and classifiers jointly. Finally, we fuse all outputs from multiple deep networks to obtain the final high-level feature representation and give the prediction probability for each plant species. The experimental results have proved the effectiveness of our proposed method on large-scale plant species identification.

Detecting communities in complex networks has been one of the most popular research areas in recent years. There have been many community detection algorithms proposed to date. However, the local information (cliques) of communities and the search efficiency of algorithm have not been considered both in previous studies. In this paper, we propose a novel local expansion algorithm for detecting overlapping communities based on cliques. The algorithm draws on the assumption that cliques are the core of communities, as the clique takes into account the local characteristics of the community. The proposed algorithm adopts a single node with the maximum density as an initial community to prevent the formation of a large number of near-duplicate community structures, which improves the search efficiency of the algorithm. In many experiments using computer-generated and real-world networks, the proposed algorithm based on this idea verifies that the algorithm is able to detect overlapping communities effectively. The experiment yields better community uncover results, and the time efficiency and the complexity of algorithm are also satisfactory.

The growing application of face images and modern AI technology has raised another important concern in privacy protection. In many real scenarios like scientific research, social sharing and commercial application, lots of images are released without privacy processing to protect people's identity. In this paper, we develop a novel effective de-identification generative adversarial network (DeIdGAN) for face anonymization by seamlessly replacing a given face image with a different synthesized yet realistic one. Our approach consists of two steps. First, we anonymize the input face to obfuscate its original identity. Then, we use our designed de-identification generator to synthesize an anonymized face. During the training process, we leverage a pair of identity-adversarial discriminators to explicitly constrain identity protection by pushing the synthesized face away from the predefined sensitive faces to resist re-identification and identity invasion. Finally, we validate the effectiveness of our approach on public datasets. Compared with existing methods, our approach can not only achieve better identity protection rates but also preserve superior image quality and data reusability, which suggests the state-of-the-art performance.

As the amount of available video data has grown substantially, automatic video classification has become an urgent yet challenging task. Most video classification methods focus on acquiring discriminative spacial visual features and motion patterns for video representation, especially deep learning methods, which have achieved very good results on action recognition problems. However, the performance of most of these methods drastically degenerates for more generic video classification tasks where the video contents are much more complex. Thus, in this paper, the mid-level semantics of videos are exploited to bridge the semantic gap between low-level features and high-level video semantics. Inspired by the term ``frequency-inverse document frequency'', a word weighting method for the problem of text classification is introduced to the video domain. The visual objects in videos are regarded as the words in texts, and two new weighting methods are proposed to encode videos by weighting visual objects according to the characteristics of videos. In addition, the semantic similarities between video categories and visual objects are introduced from the text domain as privileged information to facilitate classifier training on the obtained semantic representations of videos. The proposed semantic encoding method (semantic stream) is then fused with the popular two-stream CNN model for the final classification results. Experiments are conducted on two large-scale complex video datasets, CCV and ActivityNet. The experimental results validate the effectiveness of the proposed methods.

Chinese calligraphy is a valuable cultural heritage belonging to the world. It is liked by many people, and our mission is to endeavor to pursue calligraphy and make contributions to the business with technical means. The automatic recognition of the styles of calligraphy by image processing techniques has important meaning in arts collection and auction, etc. Traditional feature operators have some drawbacks that leave room for modern methods like convolutional neural network (CNN). However, most of the studies focus on the classification of five basic fonts that is somewhat different from styles. In this paper, four kinds of styles belonging to standard font are classified with a novel CNN structure where two squeeze-and-excitation modules that emphasize informative feature maps and suppress useless features are embedded after convolution layers, and a Haar transform layer that fuses the features is imposed before softmax layer. Experiment result shows the significance of the proposed structure over other networks in both font and style classification.

Safety accidents caused by Lithium-ion (Li-ion) batteries are numerous in recent years. Therefore, more and more attention has been drawn to the Remaining Useful Life (RUL) prediction and health status monitoring for Li-ion batteries. This paper proposes a deep learning method that combines the Forgetting Online Sequential Extreme Learning Machine (FOS-ELM) with the Hybrid Grey Wolf Optimizer (HGWO) algorithm and attention mechanism for the Prognostic and Health Management (PHM) of Li-ion battery. First, we use the Variational Mode Decomposition (VMD) to denoise the raw data before the training. Then the key parameters optimization of the FOS-ELM model based on the HGWO algorithm is introduced. Finally, we apply the attention mechanism to further improve the accuracy of the algorithm. Compared with traditional neural network methods, the method proposed in this paper has higher efficiency and accuracy.

Accurate segmentation of brain tumors from magnetic resonance imaging (MRI) is clinically relevant in diagnoses, prognoses and surgery treatment, which requires multiple modalities to provide complementary morphological and physiopathologic information. However, missing modality commonly occurs due to image corruption, artifacts, different acquisition protocols or allergies to certain contrast agents in clinical practice. Though existing efforts demonstrate the possibility of a unified model for all missing situations, most of them perform poorly when more than one modality is missing. In this paper, we propose a novel Adversarial Co-training Network (ACN) to solve this issue, in which a series of independent yet related models are trained dedicated to each missing situation with significantly better results. Specifically, ACN adopts a novel co-training network, which enables a coupled learning process for both full modality and missing modality to supplement each other’s domain and feature representations, and more importantly, to recover the ‘missing’ information of absent modalities. Then, two unsupervised modules, i.e., entropy and knowledge adversarial learning modules are proposed to minimize the domain gap while enhancing prediction reliability and encouraging the alignment of latent representations, respectively. We also adapt modality-mutual information knowledge transfer learning to ACN to retain the rich mutual information among modalities. Extensive experiments on BraTS2018 dataset show that our proposed method significantly outperforms all state-of-the-art methods under any missing situation.

Background Predictions of cardiovascular disease risks based on health records have long attracted broad research interests. Despite extensive efforts, the prediction accuracy has remained unsatisfactory. This raises the question as to whether the data insufficiency, statistical and machine-learning methods, or intrinsic noise have hindered the performance of previous approaches, and how these issues can be alleviated. Objective Based on a large population of patients with hypertension in Shenzhen, China, we aimed to establish a high-precision coronary heart disease (CHD) prediction model through big data and machine-learning Methods Data from a large cohort of 42,676 patients with hypertension, including 20,156 patients with CHD onset, were investigated from electronic health records (EHRs) 1-3 years prior to CHD onset (for CHD-positive cases) or during a disease-free follow-up period of more than 3 years (for CHD-negative cases). The population was divided evenly into independent training and test datasets. Various machine-learning methods were adopted on the training set to achieve high-accuracy prediction models and the results were compared with traditional statistical methods and well-known risk scales. Comparison analyses were performed to investigate the effects of training sample size, factor sets, and modeling approaches on the prediction performance. Results An ensemble method, XGBoost, achieved high accuracy in predicting 3-year CHD onset for the independent test dataset with an area under the receiver operating characteristic curve (AUC) value of 0.943. Comparison analysis showed that nonlinear models (K-nearest neighbor AUC 0.908, random forest AUC 0.938) outperform linear models (logistic regression AUC 0.865) on the same datasets, and machine-learning methods significantly surpassed traditional risk scales or fixed models (eg, Framingham cardiovascular disease risk models). Further analyses revealed that using time-dependent features obtained from multiple records, including both statistical variables and changing-trend variables, helped to improve the performance compared to using only static features. Subpopulation analysis showed that the impact of feature design had a more significant effect on model accuracy than the population size. Marginal effect analysis showed that both traditional and EHR factors exhibited highly nonlinear characteristics with respect to the risk scores. Conclusions We demonstrated that accurate risk prediction of CHD from EHRs is possible given a sufficiently large population of training data. Sophisticated machine-learning methods played an important role in tackling the heterogeneity and nonlinear nature of disease prediction. Moreover, accumulated EHR data over multiple time points provided additional features that were valuable for risk prediction. Our study highlights the importance of accumulating big data from EHRs for accurate disease predictions.

Due to the ease of use, face authentication could be a promising way to replace hard-to-remember passwords to access web services. However, to make face authentication suitable for web services, there are still several critical security and privacy challenges unaddressed. First, the existing authentication servers typically collect the plaintext of users' facial images in order to conduct authentication. If the servers are compromised, the attackers would obtain the users' facial images and can easily impersonate the users in any other applications that use face authentication. Second, it is also hard to prevent attackers from using facial images published in social network sites to impersonate the true user. In this paper, we conquered these two issues by proposing a novel secure face authentication system, called UFace. UFace uses special close-up facial images (which cannot be found online) for authentication. To further ensure the confidentiality of these close-up images, UFace guarantees that these images are only stored at user side and the servers have not any plaintext of these images. The face authentication is conducted securely with two collaborative authentication servers. UFace was implemented through both an Android application and multiple server side programs which were then evaluated in a real setting. The experimental results demonstrate that the UFace system can accurately authenticate users within a few seconds.

To support large-scale visual recognition (i.e., recognizing thousands or even tens of thousands of object classes), a multi-level deep learning algorithm is developed to learn multiple deep networks and a tree classifier jointly, where a concept ontology is constructed to organize large numbers of object classes hierarchically in a coarse-to-fine fashion and determine the inter-related learning tasks automatically. Our multi-level deep learning algorithm can: (a) train multiple deep networks simultaneously to achieve more discriminative representations of both coarse-grained groups and fine-grained object classes at different levels of the concept ontology (i.e., learning multiple sets of deep features simultaneously for different tasks); (b) leverage multi-task learning to train more discriminative classifiers for the fine-grained object classes in the same group to enhance their separability significantly and enable inter-class knowledge transferring; and (c) learn multiple deep networks and the tree classifier jointly in an end-to-end fashion. Our experimental results on three image sets have demonstrated that our multi-level deep learning algorithm can achieve very competitive results on both the accuracy rates and the computational efficiency for large-scale visual recognition.

Ranking influential nodes of networks is very meaningful for many applications, such as disease propagation inhibition and information dissemination control. Taking multiple attributes into consideration is a hopeful strategy. However, traditional multi-attribute ranking methods have some defects. Firstly, the computational complexity of these methods is usually higher than O(n), inapplicable to large scale social networks. Secondly, contributions of different attributes are viewed as equally important, leading to the limited improvement in performance. This paper proposes a multi-attribute ranking method based on node position and neighborhood, with low computational complexity O(n). The proposed method utilizes iteration information in the K-shell decomposition to further distinguish the node position and also fully considers the neighborhood’s effect upon the influence capability of a node. Furthermore, the entropy method is used to weight the node position and neighborhood attributes. Experiment results in terms of monotonicity, correctness and efficiency have demonstrated the good performance of the proposed method on both artificial networks and real world ones. It can efficiently and accurately provide a more reasonable ranking list than previous approaches.

An enhanced hierarchical visual tree is constructed to organize large numbers of image categories and automatically identify the inter-related tasks for multi-task sparse metric learning.A new objective function is define for multi-task sparse metric learning.A top-down approach is developed for supporting hierarchical learning of a tree of multi-task sparse metrics over the enhanced visual tree. In this paper, a novel approach is developed to learn a tree of multi-task sparse metrics hierarchically over a visual tree to achieve a fast solution to large-scale image classification, where an enhanced visual tree is first learned to organize large numbers of image categories hierarchically in a coarse-to-fine fashion. Over the visual tree, a tree of multi-task sparse metrics is learned hierarchically by: (a) performing multi-task sparse metric learning over the sibling child nodes under the same parent node to explicitly separate their commonly-shared metric from their node-specific metrics; and (b) propagating the node-specific metric for the parent node to its sibling child nodes (at the next level of the visual tree), so that more discriminative metrics can be learned for controlling inter-level error propagation effectively. We have evaluated our hierarchical multi-task sparse metric learning algorithm over three different image sets and the experimental results demonstrated that our hierarchical multi-task sparse metric learning algorithm can obtain better performance than the state-of-the-art algorithms on large-scale image classification.

Object proposal generation is an important step in object detection, obtaining high-quality proposals can effectively improve the performance of detection. In this paper, we propose a semantic, class-specific approach to re-rank object proposals, which can consistently improve the recall performance even with fewer proposals. Specifically, we first extract features for each proposal including semantic segmentation, stereo information, contextual information, CNN-based objectness and low-level cue, and then score them using class-specific weights learned by Structured SVM. The advantages of the proposed model are two-fold: 1) it can be easily merged to existing generators with few computational costs, and 2) it can achieve high recall rate under strict critical even using fewer proposals. Experimental evaluation on the KITTI benchmark demonstrates that our approach significantly improves existing popular generators on recall performance. Moreover, in the experiment conducted for object detection, even with 1500 proposals, our approach can still have higher average precision (AP) than baselines with 5000 proposals.

To achieve automatic recommendation of privacy settings for image sharing, a new tool called iPrivacy (image privacy) is developed for releasing the burden from users on setting the privacy preferences when they share their images for special moments. Specifically, this paper consists of the following contributions: 1) massive social images and their privacy settings are leveraged to learn the object-privacy relatedness effectively and identify a set of privacy-sensitive object classes automatically; 2) a deep multi-task learning algorithm is developed to jointly learn more representative deep convolutional neural networks and more discriminative tree classifier, so that we can achieve fast and accurate detection of large numbers of privacy-sensitive object classes; 3) automatic recommendation of privacy settings for image sharing can be achieved by detecting the underlying privacy-sensitive objects from the images being shared, recognizing their classes, and identifying their privacy settings according to the object-privacy relatedness; and 4) one simple solution for image privacy protection is provided by blurring the privacy-sensitive objects automatically. We have conducted extensive experimental studies on real-world images and the results have demonstrated both the efficiency and effectiveness of our proposed approach.

In the age of big data, information overload is getting worse. Most of the existing recommender systems which apply data analysis and behavioral analysis to make personal recommendation have suffered from the problem of low prediction accuracy. To address this problem, a new preference elicitation algorithm is developed by using bipartite graphical correlation and implicit trust. More precisely, to compute the bipartite graphical correlation, an improved single-source shortest path is firstly presented to get the shortest behavior path based on the user-item bipartite graph. While the Markov separation algorithm is employed to separate uncorrelated vertexes and obtain the values of bipartite graphical correlation. Secondly, the implicit trust is used to represent trust relationship between users and items which have no historical behaviors. Independent trust group is applied to represent the closed-loop path on the user-item bipartite graph, and a compute method is developed to get the values of implicit trust based on the trusty values of independent trust group and explicit trust. Finally, a weighted prediction method based on bipartite graphical correlation and implicit trust is employed to compute the final ratings. Our empirical experiments are performed on a sparse data set, the results of which have demonstrated that our method can achieve lower P@R and efficiently improve recommendation quality. HighlightsWe extract the most relevant items for users on user-item bipartite graph.A trust measure method is proposed by using the explicit confidence, implicit confidence and independent confidence group.Combine bipartite graphical correlation and implicit trust can improve recommendation accuracy efficiently.

In this paper, a hierarchical deep multi-task learning (HD-MTL) algorithm is developed to support large-scale visual recognition (e.g., recognizing thousands or even tens of thousands of atomic object classes automatically). First, multiple sets of multi-level deep features are extracted from different layers of deep convolutional neural networks (deep CNNs), and they are used to achieve more effective accomplishment of the coarseto- fine tasks for hierarchical visual recognition. A visual tree is then learned by assigning the visually-similar atomic object classes with similar learning complexities into the same group, which can provide a good environment for determining the interrelated learning tasks automatically. By leveraging the inter-task relatedness (inter-class similarities) to learn more discriminative group-specific deep representations, our deep multi-task learning algorithm can train more discriminative node classifiers for distinguishing the visually-similar atomic object classes effectively. Our hierarchical deep multi-task learning (HD-MTL) algorithm can integrate two discriminative regularization terms to control the inter-level error propagation effectively, and it can provide an end-to-end approach for jointly learning more representative deep CNNs (for image representation) and more discriminative tree classifier (for large-scale visual recognition) and updating them simultaneously. Our incremental deep learning algorithms can effectively adapt both the deep CNNs and the tree classifier to the new training images and the new object classes. Our experimental results have demonstrated that our HD-MTL algorithm can achieve very competitive results on improving the accuracy rates for large-scale visual recognition.

In multi-task learning, multiple interrelated tasks are jointly learned to achieve better performance. In many cases, if we can identify which tasks are related, we can also clearly identify which tasks are unrelated. In the past, most researchers emphasized exploiting correlations among interrelated tasks while completely ignoring the unrelated tasks that may provide valuable prior knowledge for multi-task learning. In this paper, a new approach is developed to hierarchically learn a tree of multi-task metrics by leveraging prior knowledge about both the related tasks and unrelated tasks. First, a visual tree is constructed to hierarchically organize large numbers of image categories in a coarse-to-fine fashion. Over the visual tree, a multi-task metric classifier is learned for each node by exploiting both the related and unrelated tasks, where the learning tasks for training the classifiers for the sibling child nodes under the same parent node are treated as the interrelated tasks, and the others are treated as the unrelated tasks. In addition, the node-specific metric for the parent node is propagated to its sibling child nodes to control inter-level error propagation. Our experimental results demonstrate that our hierarchical metric learning algorithm achieves better results than other state-of-the-art algorithms.

Making use of the general physical model of the Mach-Zehnder interferometer with photon loss which is a fundamental physical issue, we investigate the continuous-variable quantum phase estimation based on machine learning approach, and an efficient recursive Bayesian estimation algorithm for Gaussian states phase estimation has been proposed. With the proposed algorithm, the performance of the phase estimation may be improved distinguishably. For example, the physical limits (i.e., the standard quantum limit and Heisenberg limit) for the phase estimation precision may be reached in more efficient ways especially in the situation of the prior information being employed, the range for the estimated phase parameter can be extended from [0, π/2] to [0, 2π] compared with the conventional approach, and influences of the photon losses on the output parameter estimation precision may be suppressed dramatically in terms of saturating the lossy bound. In addition, the proposed algorithm can be extended to the time-variable or multi-parameter estimation framework.

This paper introduces a fast and efficient network architecture, NeXtVLAD, to aggregate frame-level features into a compact feature vector for large-scale video classification. Briefly speaking, the basic idea is to decompose a high-dimensional feature into a group of relatively low-dimensional vectors with attention before applying NetVLAD aggregation over time. This NeXtVLAD approach turns out to be both effective and parameter efficient in aggregating temporal information. In the 2nd Youtube-8M video understanding challenge, a single NeXtVLAD model with less than 80M parameters achieves a GAP score of 0.87846 in private leaderboard. A mixture of 3 NeXtVLAD models results in 0.88722, which is ranked 3rd over 394 teams. The code is publicly available at https://github.com/linrongc/youtube-8m.

In this paper, an effective algorithm is developed for tackling the problem of near-duplicate image identification from large-scale image sets, where the LLC (locality-constrained linear coding) method is seamlessly integrated with the maxIDF cut model to achieve more discriminative representations of images. By incorporating MapReduce framework for image clustering and pairwise merging, the near duplicates of images can be identified effectively from large-scale image sets. An intuitive strategy is also introduced to guide the process for parameter selection. Our experimental results on large-scale image sets have revealed that our algorithm can achieve significant improvement on both the accuracy rates and the computation efficiency as compared with other baseline methods.

Semantic entities carry the most important semantics of text data. Therefore, the identification and the relationship integration of semantic entities are very important for applications requiring semantics of text data. However, current strategies are still facing many problems such as semantic entity identification, new word identification and relationship integration among semantic entities. To address these problems, a two-phase framework for semantic entity identification with relationship integration in large scale text data is proposed in this paper. In the first semantic entities identification phase, we propose a novel strategy to extract unknown text semantic entities by integrating statistical features, Decision Tree (DT), and Support Vector Machine (SVM) algorithms. Compared with traditional approaches, our strategy is more effective in detecting semantic entities and more sensitive to new entities that just appear in the fresh data. After extracting the semantic entities, the second phase of our framework is for the integration of Semantic Entities Relationships (SER) which can help to cluster the semantic entities. A novel classification method using features such as similarity measures and co-occurrence probabilities is applied to tackle the clustering problem and discover the relationships among semantic entities. Comprehensive experimental results have shown that our framework can beat state-of-the-art strategies in semantic entity identification and discover over 80% relationship pairs among related semantic entities in large scale text data.

Large-scale image classification is a challenging task and has recently attracted active research interests. In this paper, a new algorithm is developed to achieve more effective implementation of large-scale image classification by hierarchical learning of large-margin metrics (HLMMs). A hierarchical visual tree is seamlessly integrated with metric learning to learn a set of node-specific/category-specific large-margin metrics. First, a hierarchical visual tree is learned to characterize the inter-category visual correlations effectively and organize large numbers of image categories in a coarse-to-fine fashion. Second, a new algorithm is developed to support hierarchical learning of large-margin metrics by training nearest class mean (NCM) classifiers over our hierarchical visual tree. In addition, we also consider dimensionality reduction as a regularizer for high-dimensional data in our large-margin metric learning. Two top-down approaches are developed for supporting hierarchical learning of large-margin metrics. We focus on learning more discriminative metrics for NCM node classifiers to identify the visually similar sub-nodes (visually similar image categories) under the same parent node over our hierarchical visual tree. A mini-batch stochastic gradient descend method is used to optimize our HLMMs learning algorithm. The experimental results on ImageNet Large Scale Visual Recognition Challenge 2010 dataset (ILSVRC2010) have demonstrated that our HLMMs learning algorithm is very promising for supporting large-scale image classification. HighlightsA hierarchical visual tree is built to characterize the inter-category correlations.A new objective function is developed for large-margin metric learning.A hierarchical visual tree is seamlessly integrated with metric learning.Top-down methods are developed for hierarchical learning of large-margin metrics.Our proposed algorithm outperforms recent tree-based classification approaches.

Automatically classifying large scale of video data is an urgent yet challenging task. To bridge the semantic gap between low-level features and high-level video semantics, we propose a method to represent videos with their mid-level semantics. Inspired by the problem of text classification, we regard the visual objects in videos as the words in documents, and adapt the TF-IDF word weighting method to encode videos by visual objects. Some extensions upon the proposed method are also made according to the characteristics of videos. We integrate the proposed semantic encoding method with the popular two-stream CNN model for video classification. Experiments are conducted on two large-scale video datasets, CCV and ActivityNet. The experimanetal results validates the effectiveness of our method.

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a `co-attention' mechanism is developed by using a deep neural network architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multi-modal feature fusion, a generalized Multi-modal Factorized High-order pooling approach (MFH) is developed to achieve more effective fusion of multi-modal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the KL (Kullback-Leibler) divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A deep neural network architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA datasets and win the runner-up in VQA Challenge 2017., Comment: 13 pages, 9 figures. arXiv admin note: substantial text overlap with arXiv:1708.01471

We present an automatic approach for large-scale fashion recognition, given an image without any kind of annotation. We formulate the problem as a hierarchical deep learning (HDL) algorithm which can: (i) integrate the deep CNNs to learn more discriminative high-level features for fashion image representations of both coarse-grained and fine-grained classes at different levels of the fashion ontology tree; (ii) leverage multi-task learning and inter-task relationship constraint to train more discriminative classifiers for the nodes on the fashion ontology; (iii) use back propagation to simultaneously refine both the relevant node classifiers and the deep CNNs according to a joint objective function; and (iv) accelerate the fashion retrieval process via path-based classification. The experimental results have verified the effectiveness and efficiency of our proposed algorithm on both classification and retrieval performance.

Real-world datasets often involve multiple views of data items, e.g., a Web page can be described by both its content and anchor texts of hyperlinks leading to it; photos in Flickr could be characterized by visual features, as well as user contributed tags. Different views provide information complementary to each other. Synthesizing multi-view features can lead to a comprehensive description of the data items, which could benefit many data analytic applications. Unfortunately, the simple idea of concatenating different feature vectors ignores statistical properties of each view and usually incurs the “curse of dimensionality” problem. We propose Multi-view Concept Learning (MCL), a novel nonnegative latent representation learning algorithm for capturing conceptual factors from multi-view data. MCL exploits both multi-view information and label information. The key idea is to learn a common latent space across different views which (1) captures the semantic relationships between data items through graph embedding regularization on labeled items, and (2) allows each latent factor to be associated with a subset of views via sparseness constraints. In this way, MCL could capture flexible conceptual patterns hidden in multi-view features. Experiments on a toy problem and three real-world datasets show that MCL performs well and outperforms baseline methods.

In this paper, a hierarchical learning algorithm is developed for classifying large-scale patient records, e.g., categorizing large-scale patient records into large numbers of known patient categories (i.e., thousands of known patient categories) for automatic treatment stratification. Our hierarchical learning algorithm can leverage tree structure to train more discriminative max-margin classifiers for high-level nodes and control interlevel error propagation effectively. By ruling out unlikely groups of patient categories (i.e., irrelevant high-level nodes) at an early stage, our hierarchical approach can achieve log-linear computational complexity, which is very attractive for big data applications. Our experiments on one specific medical domain have demonstrated that our hierarchical approach can achieve very competitive results on both classification accuracy and computational efficiency as compared with other state-of-the-art techniques.

In this paper, a cost-sensitive learning algorithm is developed to train hierarchical tree classifiers for large-scale image classification application (i.e., categorizing large-scale images into thousands of object classes). A visual tree is first constructed for organizing large numbers of object classes hierarchically and identifying inter-related learning tasks automatically. For the fine-grained object classes at the sibling leaf nodes, they share significant common visual properties but still contain subtle visual differences, thus a multi-task structural learning algorithm is developed to train their inter-related classifiers jointly to enhance their discrimination power. For the coarse-grained categories (i.e., groups of visually similar object classes) at the sibling non-leaf nodes, a hierarchical learning algorithm is developed to leverage tree structure (by adding two inter-level constraints) to train their inter-related classifiers jointly and control inter-level error propagation effectively. To achieve more robust detection of large numbers of object classes, a visual forest is learned by combining multiple visual trees (for different configurations) and their hierarchical tree classifiers. By penalizing various types of misclassification errors differently, a cost-sensitive learning approach is further developed to detect the appearances of new object classes accurately, and an incremental learning algorithm is developed to achieve more effective training of the discriminative classifiers for new object classes. Our experimental results have demonstrated that our cost-sensitive hierarchical learning algorithm can achieve very competitive results on both classification accuracy and computational efficiency as compared with other state-of-the-art techniques. HighlightsVisual tree to organize large-scale object classes hierarchically and determine inter-related learning tasks automatically.Multi-task structural learning for joint classifier training to enhance their discrimination power significantly.Hierarchical learning to leverage inter-level constraints for classifier training and limiting inter-level error propagation.Task and tree parallelism to scale up our hierarchical learning algorithm for large-scale image classification.Cost-sensitive learning and incremental learning for training and detecting for new object classes more effectively.

Learning based super-resolution (SR) methods, which predict the high-resolution pixel values but not directly provide an estimation of uncertainty, are typically non-probabilistic and have limited generalization ability. Gaussian processes can provide a framework for deriving regression techniques with explicit uncertainty models, but Gaussian Process Regression (GPR) has a significant drawback in being time consuming. The computational complexity of GPR is cubic in the number of training examples, which is prohibitively expensive for a large-scale training set. In this article, we have proposed learning local GPR for image SR. Two algorithms are developed to support local GPR for super resolution. A data-driven GPR based super-resolution algorithm is first developed to learn a local GPR model for every LR patch on an input oriented training dataset with moderate size. In order to further improve the running speed, a prototype based GPR algorithm is developed for super resolution. The proposed algorithm is about one-order faster than the data-driven GPR solution because it makes models for the prototypes of image patches rather than for each image patch. Thus, the local regression efforts are greatly reduced to just finding the nearest prototype for each LR image patch and applying its corresponding pre-computed projective matrix for super-resolution prediction. Our algorithms have greater robustness and usability as they provide a formularized way to automatically learn the hyper-parameters introduced for optimizing the covariance function, while most of the state-of-the-art super-resolution methods could only utilize these parameters in a cross-verification way. Moreover, our algorithms offer confidence values at the test points which benefit the pixels' post-processing. Our algorithms are evaluated on popular datasets that are widely used in the super-resolution literature, and the experimental results have demonstrated that the efficiency and effectiveness of our proposed algorithms are comparative with several state-of-the-arts super-resolution methods.

In this paper, an effective algorithm is developed to learn more discriminative multi-class classifiers for achieving more accurate hand detection. At each round of boosting, a set of shared stump classifiers with relatively low discrimination power are selected by using a "slowest error growth" discriminant, and they are further combined to generate a multi-class classifier with high discrimination power. For the learned multi-class classifier, all of its shared stump classifiers can jointly cover all the potential situations (i.e., various classes of hand postures) sufficiently and discriminate each class of hand postures more effectively. In addition, multiple thresholds are set for each stump classifier to enhance its discrimination power. Finally, the optional mask images are further used to reduce both the feature dimensions and the computational cost for searching the appropriate features. The experimental results on both our hand dataset and NUS hand posture dataset-II have demonstrated the effectiveness and efficiency of our algorithm. HighlightsA set of shared stumps are combined to strengthen the discrimination power of weak classifiers.A "slowest error growth" discriminant to determine the optimal combination of stumps.Multiple thresholds are leveraged in shared stumps to fit different classes.We associate effective features with different classes of hands, and employ mix-type features.As compared with JointBoost, our classifier can obtain a better classification performance with less runtime cost.

To achieve more effective solution for large-scale image classification (i.e., classifying millions of images into thousands or even tens of thousands of object classes or categories), a deep multi-task learning algorithm is developed by seamlessly integrating deep CNNs with multi-task learning over the concept ontology, where the concept ontology is used to organize large numbers of object classes or categories hierarchically and determine the inter-related learning tasks automatically. Our deep multi-task learning algorithm can integrate the deep CNNs to learn more discriminative high-level features for image representation, and it can also leverage multi-task learning and inter-level relationship constraint to train more discriminative tree classifier over the concept ontology and control the inter-level error propagation effectively. In our deep multi-task learning algorithm, we can use back propagation to simultaneously refine both the relevant node classifiers (at different levels of the concept ontology) and the deep CNNs according to a joint objective function. The experimental results have demonstrated that our deep multi-task learning algorithm can achieve very competitive results on both the accuracy and the cost of feature extraction for large-scale image classification.

The manual process for privacy setting could be very time-consuming and challenging for common users. By assuming that there are hidden correlations between the visual properties of images (i.e., visual features) or object classes and the privacy settings for image sharing, an effective algorithm is developed in this paper to achieve automatic prediction of image privacy, so that the best-matching privacy setting can be recommended automatically for each single image being shared. Our algorithm for automatic image privacy prediction contains two approaches: (a) feature-based approach by learning more representative deep features and discriminative classifier for assigning each single image being shared into one of two categories: private vs. public, (b) object-based approach by detecting large numbers of privacy-sensitive object classes and events automatically and leveraging them to achieve more discriminative characterization of image privacy, so that we can support more explainable solution for automatic image privacy prediction. We have also conducted extensive experimental studies on large-scale social images, which have demonstrated both efficiency and effectiveness of our proposed algorithm.

In this paper, a distributed approach is developed for achieving large-scale classifier training and image classification. First, a visual concept network is constructed for determining the inter-related learning tasks automatically, e.g., the inter-related classifiers for the visually similar object classes in the same group should be trained in parallel by using multiple machines to enhance their discrimination power. Second, an MPI-based distributed computing approach is constructed by using a master-slave mode to address two critical issues of huge computational cost and huge storage/memory cost for large-scale classifier training and image classification. In addition, an indexing-based storage method is developed for reducing the sizes of intermediate SVM models and avoiding the repeated computations of SVs (support vectors) in the test stage for image classification. Our experiments have also provided very positive results on 2010 ImageNet database for Large Scale Visual Recognition Challenge.

Several clustering analysis techniques for categorical data exist to divide similar objects into groups. Some are able to handle uncertainty in the clustering process, whereas others have stability issues. In this paper, we propose a new technique called TMDP (Total Mean Distribution Precision) for selecting the partitioning attribute based on probabilistic rough set theory. On the basis of this technique, with the concept of granularity, we derive a new clustering algorithm, MTMDP (Maximum Total Mean Distribution Precision), for categorical data. The MTMDP algorithm is a robust clustering algorithm that handles uncertainty in the process of clustering categorical data. We compare the MTMDP algorithm with the MMR (Min–Min–Roughness) algorithm which is the most relevant clustering algorithm, and also compared it with other unstable clustering algorithms, such as k-modes, fuzzy k-modes and fuzzy centroids. The experimental results indicate that the MTMDP algorithm can be successfully used to analyze grouped categorical data because it produces better clustering results.

User-provided tags for social images have facilitated many fields, such as social image organization, summarization and retrieval. Since the users utilize their own knowledge and personalized language to describe the visual content of social images, these social tags are too imprecise and ambiguous to exploit the social image tagging. In this paper, we discover the content-similar images (peers) and leverage the relationships among these images (peer cooperation) to handle the problem of content-irrelevant tags. A bi-layer clustering framework for discovering content-similar images is proposed to divide image collection into different groups, and the tags of peers in these groups are cleaned jointly based on tag statistics and relevance. The relevance of tags measured by Google Distance is used to generate the first-layer clustering and then the bi-modality similarity of images is used to perform the second-layer clustering. Based on the bi-layer clustering, we utilize peers in a group to identify their content-irrelevant tags. Finally, an extended Fisher's criterion is proposed to decide the proper number of content-irrelevant tags. To verify the effectiveness of our proposed technique, we conduct the experiments on the social images of Flickr and the standard benchmark. The comparison experiments show that our proposed algorithm achieves positive results for tag cleansing and image retrieval.

Semantic categorization for the complex videos is an ambiguous task. The semi-supervised learning method based on hypergraph model can achieve multi-semantics labels, but a hypergraph model is sensitive to the radius parameter when it is constructed and the number of vertices belonging to a hyperedge is fixed. In this paper, a semi-supervised incremental learning method based on adaptive probabilistic hypergraph for video semantic detection is presented. In the probabilistic hypergraph modeling, a formula is presented as a measurement to adaptively decide whether a vertex is belonged to a hyperedge or not. The model has high robustness and can overcome the defect of fixed number of vertices belonging to the same hyperedge in the traditional probabilistic hypergraph model. In the semi-supervised incremental learning process, a threshold is defined, which is used to judge whether unlabeled sample can be added into the modeling, in order that the model learning result for unlabeled samples has high certainty. The experimental results show that our method can improve the model generalization ability and utilize the unlabeled samples effectively. In the aspects of recall rate and precision rate for semantic video concept detection from complex videos, our proposed method outperforms the compared methods.

This paper focuses on three types of attribute reducts in inconsistent decision tables: assignment reduct, distribution reduct, and maximum distribution reduct. It is quite inconvenient to judge these three types of reduct directly according to their definitions. This paper proposes judgment theorems for the assignment reduct, the distribution reduct and the maximum distribution reduct, which are expected to greatly simplify the judging of these three types of reducts. On this basis, we derive three new types of attribute significance measures and construct the Q-ARA (Quick Assignment Reduction Algorithm), the Q-DRA (Quick Distribution Reduction Algorithm), and the Q-MDRA (Quick Maximum Distribution Reduction Algorithm). These three algorithms correspond to the three types of reducts. We conduct a series of comparative experiments with twelve UCI (machine learning data repository, University of California at Irvine) data sets (including consistent and inconsistent decision tables) to evaluate the performance of the three reduction algorithms proposed with the relevant algorithm QuickReduct [9,34]. The experimental results show that QuickReduct possesses weak robustness because it cannot find the reduct even for consistent data sets, whereas our proposed three algorithms show strong robustness because they can find the reduct for each data set. In addition, we compare the Q-DRA (Quick Distribution Reduction Algorithm) with the CEBARKNC (conditional entropy-based algorithm for reduction of knowledge without a computing core) [43] because both find the distribution reduct by using a heuristic search. The experimental results demonstrate that Q-DRA runs faster than CEBARKNC does because the distribution function of Q-DRA has a lower calculation cost. Instructive conclusions for these reduction algorithms are drawn from the perspective of classification performance for the C4.5 and RBF-SVM classifiers. Last, we make a comparison between discernibility matrix-based methods and our algorithms. The experimental results indicate that our algorithms are efficient and feasible.

DBSCAN (density-based spatial clustering of applications with noise) is an important spatial clustering technique that is widely adopted in numerous applications. As the size of datasets is extremely large nowadays, parallel processing of complex data analysis such as DBSCAN becomes indispensable. However, there are three major drawbacks in the existing parallel DBSCAN algorithms. First, they fail to properly balance the load among parallel tasks, especially when data are heavily skewed. Second, the scalability of these algorithms is limited because not all the critical sub-procedures are parallelized. Third, most of them are not primarily designed for shared-nothing environments, which makes them less portable to emerging parallel processing paradigms. In this paper, we present MR-DBSCAN, a scalable DBSCAN algorithm using MapReduce. In our algorithm, all the critical sub-procedures are fully parallelized. As such, there is no performance bottleneck caused by sequential processing. Most importantly, we propose a novel data partitioning method based on computation cost estimation. The objective is to achieve desirable load balancing even in the context of heavily skewed data. Besides, We conduct our evaluation using real large datasets with up to 1.2 billion points. The experiment results well confirm the efficiency and scalability of MR-DBSCAN.

Translating image tags at the image level to regions (i.e., tag-to-region assignment), which could play an important role in leveraging loosely-labeled training images for object classifier training, has become a popular research topic in the multimedia research community. In this paper, a novel two-stage multiple instance learning algorithm is presented for automatic tag-to-region assignment. The regions are generated by performing multiple-scale image segmentation and the instances with unique semantics are selected out from those regions by a random walk process. The affinity propagation (AP) clustering technique and Hausdorff distance are performed on the instances to identify the most positive instance and utilize it to initialize the maximum searching of Diverse Density likelihood in the first stage. In the second stage, the most contributive instance, which is chosen from each bag, is treated as the key instance for simplifying the computing procedure of Diverse Density likelihood. At last, an automatic method is proposed to discriminate the boundary between positive instances and negative instances. Our experiments on three well-known image sets have provided positive results.

Crowdedness spot is a crowded area with an abnormal number of objects. Detecting the crowdedness spots of moving vehicles in an urban area is essential to many applications. An intuitive method is to cluster the objects in areas to get the density information. Unfortunately, the data capturing vehicle mobility possesses some new features, such as highly mobile environments, supremely limited size of sample objects, and nonuniform biased samples, and all these features have raised new challenges that make traditional density-based clustering algorithms fail to retrieve the real clustering property of objects, making the results less meaningful. In this paper, we propose a novel nondensity-based approach called mobility-based clustering. The key idea is that sample objects are employed as “sensors” to perceive the vehicle crowdedness in nearby areas using their instant mobility rather than the “object representatives.” As such, the mobility of samples is naturally incorporated. Several key factors beyond the vehicle crowdedness have been identified, and techniques to compensate these effects are accordingly proposed. Furthermore, taking the detected crowdedness spots as a label of the taxi, we can identify one particular taxi to be a crowdedness taxi that crosses a number of different crowdedness spots. We evaluate the performance of our methods and baseline approaches based on real traffic situations (to retrieve the real traffic crowdedness) and real-life data sets. Finally, the interesting findings are provided for further discussions.

A structural learning algorithm is developed in this paper to achieve more effective training of large numbers of inter-related classifiers for supporting large-scale image classification and annotation. A visual concept network is constructed for characterizing the inter-concept visual correlations intuitively and determining the inter-related learning tasks automatically in the visual feature space rather than in the label space. By partitioning large numbers of object classes and image concepts into a set of groups according to their inter-concept visual correlations, the object classes and image concepts in the same group will share similar visual properties and their classifiers are strongly inter-related while the object classes and image concepts in different groups will contain various visual properties and their classifiers can be trained independently. By leveraging the inter-concept visual correlations for inter-related classifier training, our structural learning algorithm can train the inter-related classifiers jointly rather than independently, which can enhance their discrimination power significantly. Our experiments have also provided very positive results on large-scale image classification and annotation.

Many existing 3D model retrieval use KNN (k-nearest neighbor) method for similarity search, but it is inefficient in high-dimension space search. In this paper, the classification tools are integrated for supporting more effective 3D model search in the high-dimensional feature space. Our proposed algorithm used multiple SVM classifiers to predict 3D models for a given query and D-S Evidence theory is used to fuse all the prediction results. Experimental results show that our proposed 3D model retrieval algorithm can improve the accuracy significantly compared with the traditional kNN method.

In this paper, a novel approach is developed to achieve automatic image collection summarization. The effectiveness of the summary is reflected by its ability to reconstruct the original set or each individual image in the set. We have leveraged the dictionary learning for sparse representation model to construct the summary and to represent the image. Specifically we reformulate the summarization problem into a dictionary learning problem by selecting bases which can be sparsely combined to represent the original image and achieve a minimum global reconstruction error, such as MSE (Mean Square Error). The resulting ''Sparse Least Square'' problem is NP-hard, thus a simulated annealing algorithm is adopted to learn such dictionary, or image summary, by minimizing the proposed optimization function. A quantitative measurement is defined for assessing the quality of the image summary by investigating both its reconstruction ability and its representativeness of the original image set in large size. We have also compared the performance of our image summarization approach with that of six other baseline summarization tools on multiple image sets (ImageNet, NUS-WIDE-SCENE and Event image set). Our experimental results have shown that the proposed dictionary learning approach can obtain more accurate results as compared with other six baseline summarization algorithms.

In this paper, a novel data-driven algorithm is developed for achieving quantitative characterization of the semantic gaps directly in the visual feature space, where the visual feature space is the common space for concept classifier training and automatic concept detection. By supporting quantitative characterization of the semantic gaps, more effective inference models can automatically be selected for concept classifier training by: (1) identifying the image concepts with small semantic gaps (i.e., the isolated image concepts with high inner-concept visual consistency) and training their one-against-all SVM concept classifiers independently; (2) determining the image concepts with large semantic gaps (i.e., the visually-related image concepts with low inner-concept visual consistency) and training their inter-related SVM concept classifiers jointly; and (3) using more image instances to achieve more reliable training of the concept classifiers for the image concepts with large semantic gaps. Our experimental results on NUS-WIDE and ImageNet image sets have obtained very promising results.

In the peer-to-peer video-on-demand (P2P VoD) streaming applications, providing VCR-like operations is important but much challenging. In this paper, we propose a novel P2P scheme–Fibonacci ring overlay networks with distributed chunk storage, called FiRiNet, to reduce jump latency caused by VCR-like operations and avoid an adverse impact caused by VCR-like operations. In FiRiNet, video data is divided into chunks and stored at peers’ local storage in a distributed manner. A peer can achieve fast neighbor discovery to reduce jump latency by maintaining some neighbors in a set of concentric rings with Fibonacci sequence radii. Moreover, FiRiNet constructs the overlay networks and distributes the video data based on the stored chunks, which can make FiRiNet avoid an adverse impact caused by VCR-like operations and make FiRiNet resilient to the peer churn caused by VCR-like operations. Through simulations, we demonstrate that FiRiNet is an effcient and resilient scheme with low control overhead, short jump latency, and high streaming quality.