Search

Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher's ability in unfolding the knowledge to be distilled, but also improve the student's capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods. [ABSTRACT FROM AUTHOR]

The feature representation learning is the basic task that plays an important role in artificial intelligence, data mining and robotics. In [[17]], due to the ambiguity of the category boundary, the authors proposed Dempster-Shafer evidence theory (DST) to reduce category boundary ambiguity and output reasonable decisions by combining adaptation outputs based on uncertainty. Preface to the Special Issue "Advancement of Mathematical Methods in Feature Representation Learning for Artificial Intelligence, Data Mining and Robotics" - Special Issue Book Inspired by generative adversarial networks (GANs), the authors of [[18]] proposed a novel adversarial domain adaptation method with an initial state fusion strategy followed by a domain similarity strategy based on information entropy. [Extracted from the article]

Knowledge distillation is a promising model compression technique that generally distills the knowledge from a complex teacher model to a lightweight student model. However, the performance gain of a student model is usually limited by the capacity gap between the large teacher model and the small student model. In this paper, we propose a new collaborative knowledge distillation method that makes use of a new strategy, named Filter Knowledge Transfer (FKT) to detect and learn the valuable filter information following from the teacher to the student. To be specific, the useful knowledge of filters measured by using information entropy is transferred between different peer networks and unimportant filters are reactivated according to the ratio based on the characterized filter information entropy during the online distillation process. Experimental results derived on four popular datasets, CIFAR-10/100, Market-1501, and Tiny-ImageNet, demonstrate the superiority of our proposed method over the others we considered. • Design filter knowledge transfer. • Propose a new collaborative knowledge distillation method. • Design filter knowledge entropy to reflect the importance of filter knowledge. [ABSTRACT FROM AUTHOR]

With recent success of deep neural networks, stacked autoencoder networks have received a lot of attention for robust unsupervised representation learning. However, recent autoencoder methods cannot make full use of multi-view information and thus fail to further improve many real-world applications by exploring the geometric structures of multi-view data. To address the above-mentioned issue, we introduce hierarchical graph augmented stacked autoencoders (HGSAE) for unsupervised multi-view representation learning. Specifically, a hierarchical graph structure is first adapted to stacked autoencoders to learn view-specific representations, aiming to preserve the geometric information of multi-view data through local and non-local graph regularizations. A general or common representation can then be learned by reconstructing each single view using fully connected neural networks. By doing this, the proposed method not only preserves the geometric information in multi-view data but also automatically balances the complementarity/consistency among different views. Extensive experiments on six popular unsupervised representation learning datasets demonstrate the effectiveness of our method when compared with recent state-of-the-art autoencoder methods. • Design local and non-local graph constructions for view-specific representations. • Adapt hierarchical graph structure to stacked autoencoders. • Propose hierarchical graph augmented stacked autoencoders. [ABSTRACT FROM AUTHOR]

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher–student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded. [ABSTRACT FROM AUTHOR]

• Introduce the coarse to fine probabilistic collaborative representation. • Propose two-phase PCRC method with l1-norm and l2-norm fidelities. • Propose two-phase weighted PCRC method with l1-norm and l2-norm fidelities. The probabilistic collaborative representation-based classification (PCRC), as a novel extension of collaborative representation-based classification (CRC), is a promising method in pattern recognition. In this article, we adopt the coarse to fine representation to propose two-phase probabilistic collaborative representation based-classification (TPCRC) to enhance the power of pattern discrimination in PCRC. In TPCRC, the first phase is to utilize probabilistic collaborative representation to coarsely choose the nearest representative samples, and the second phase is to use the chosen nearest samples to finely represent and classify each testing sample. In order to employ the locality of data to further improve classification performance of PCRC, we also propose two-phase weighted probabilistic collaborative representation based-classification (TWPCRC). In the fine representation of TWPCRC, the probabilistic collaborative coefficients are weighted by the local distance similarities between each testing sample and all the training samples. The proposed methods are verified by the comparative experiments on three public image data sets. Experimental results show that the proposed methods outperform the state-of-the-art collaborative representation-based classification methods. [ABSTRACT FROM AUTHOR]

Highlights • Propose GMDKNN method. • Design categorical multi-generalized mean distances. • Design categorical nested generalized mean distance for classification. Abstract K -nearest neighbor (KNN) rule is a well-known non-parametric classifier that is widely used in pattern recognition. However, the sensitivity of the neighborhood size k always seriously degrades the KNN-based classification performance, especially in the case of the small sample size with the existing outliers. To overcome this issue, in this article we propose a generalized mean distance-based k -nearest neighbor classifier (GMDKNN) by introducing multi-generalized mean distances and the nested generalized mean distance that are based on the characteristic of the generalized mean. In the proposed method, multi-local mean vectors of the given query sample in each class are calculated by adopting its class-specific k nearest neighbors. Using the achieved k local mean vectors per class, the corresponding k generalized mean distances are calculated and then used to design the categorical nested generalized mean distance. In the classification phase, the categorical nested generalized mean distance is used as the classification decision rule and the query sample is classified into the class with the minimum nested generalized mean distance among all the classes. Extensive experiments on the UCI and KEEL data sets, synthetic data sets, the KEEL noise data sets and the UCR time series data sets are conducted by comparing the proposed method to the state-of-art KNN-based methods. The experimental results demonstrate that the proposed GMDKNN performs better and has the less sensitiveness to k. Thus, our proposed GMDKNN with the robust and effective classification performance could be a promising method for pattern recognition in some expert and intelligence systems. [ABSTRACT FROM AUTHOR]

Sparse representation methods based on l 1 and/or l 2 regularization have shown promising performance in different applications. Previous studies show that the l 1 regularization based representation has more sparse property, while the l 2 regularization based representation is much simpler and faster. However, when dealing with noisy data, both naive l 1 and l 2 regularization suffer from the issue of unsatisfactory robustness. In this paper, we explore the method to implement an antinoise sparse representation method for robust face recognition based on a joint version of l 1 and l 2 regularization. The contributions of this paper are mainly shown in the following aspects. First, a novel objective function combining both l 1 and l 2 regularization is proposed to implement an antinoise sparse representation. An iterative fitting operation via l 1 regularization is integrated with l 2 norm minimization, to obtain an antinoise classification. Second, the rationale how the proposed method produces promising discriminative and antinoise performance for face recognition is analyzed. The l 2 regularization enhances robustness and runs fast, and l 1 regularization helps cope with the noisy data. Third, the classification robustness of the proposed method is demonstrated by extensive experiments on several benchmark facial datasets. The method can be considered as an option for the expert systems for biometrics and other recognition problems facing unstable and noisy data. [ABSTRACT FROM AUTHOR]

K -nearest neighbor rule (KNN) has been regarded as one of the top 10 methods in the field of data mining. Due to its simplicity and effectiveness, it has been widely studied and applied to various classification tasks. In this article, we develop a novel representation coefficient-based k -nearest centroid neighbor method (RCKNCN), which aims to further improve the classification performance and reduce the method's sensitivity to the neighborhood size k , especially in the cases of small sample size. Different from existing KNN-based methods, RCKNCN is able to capture both the proximity and the geometry of k -nearest neighbors, and learn to differentiate the contribution of each neighbor to the classification of a testing sample through a linear representation method. Moreover, under the RCKNCN framework, we also propose a novel weighted majority voting algorithm using the representation coefficients associated with individual nearest centroid neighbors, which are deemed to hold more discriminative information of the neighbors. To fully study the classification performance of RCKNCN, we compare it with the state-of-the-art KNN-based methods on many data sets that are widely used in the literature. The extensive experiments demonstrate the effectiveness and robustness of our method in various classification tasks. • Propose RCKNCN based on both NCN and representation of neighbors. • Differentiate the contribution of each centroid neighbor via representation. • Design a new representation coefficient-based majority voting decision. [ABSTRACT FROM AUTHOR]

Representation-based classification (RBC) has attracted much attention in pattern recognition. As a linear representative RBC method, collaborative representation-based classification (CRC) is very promising for classification. Although many extensions of CRC have been developed recently, the discriminative and competitive representations of different classes for favorable classification has not yet been fully explored. To design the discriminative and competitive collaborative representations for enhancing the power of pattern discrimination, we propose a novel double competitive constraints-based collaborative representation for classification (DCCRC). In the proposed DCCRC, one competitive constraint is the l 2 -norm regularization of residuals between each query sample and the class-specific representations, the other one is the l 2 -norm regularization of the representations of all the classes excluding any one class. In two competitive constraints, the class discrimination information is employed to generate competitive representations. Moreover, the proposed method integrates both the representation learning and classification into the unified model. We study the effectiveness and robustness of the proposed method by comparing it with the state-of-the-art CRC methods on six face databases and twelve UCI data sets. The experimental results demonstrate the promising classification performance of the proposed method. [ABSTRACT FROM AUTHOR]

• Propose the DGLPGE method. • Design new local and global graph constructions. • Characterize locality and globality preserving scatters. • It can capture the geometry of data and enhance pattern discrimination. Graph embedding in dimensionality reduction has attracted much attention in the high-dimensional data analysis. Graph construction in graph embedding plays an important role in the quality of dimensionality reduction. However, the discrimination information and the geometrical distributions of data samples are not fully exploited for discovering the essential geometrical and discriminant structures of data and strengthening the pattern discrimination in graph constructions of graph embedding. To overcome the limitations of graph constructions, in this article we propose a novel graph-based dimensionality reduction method entitled discriminative globality and locality preserving graph embedding (DGLPGE) by designing the informative globality and locality preserving graph constructions. In the constructed graphs, bidirectional weights of edges are newly defined by considering both the geometrical distributions of each point of edges and the class discrimination. Using the adjacent weights of graphs, we characterize the intra-class globality preserving scatter, the inter-class globality preserving scatter and the locality preserving scatter to formulate the objective function of DGLPGE in order to optimize the projection of dimensionality reduction. Extensive experiments demonstrate that the proposed DGLPGE often outperforms the state-of-the-art dimensionality reduction methods. [ABSTRACT FROM AUTHOR]

• Propose the WELRMC method. • Propose the TPWELRMC method. • Design weighted sparsity augmented collaborative representation. • Design coarse to fine augmented representation. Representation-based classification (RBC) methods have recently been the promising pattern recognition technologies for object recognition. The representation coefficients of RBC as the linear reconstruction measure (LRM) can be well used for classifying objects. In this article, we propose two enhanced linear reconstruction measure-based classification methods based on the sparsity-augmented collaborative representation-based classification method (SA-CRC). One is the weighted enhancement linear reconstruction measure-based classification method (WELRMC) that introduces data localities into SA-CRC. Another is the two-phase weighted enhancement linear reconstruction measure-based classification method (TPWELRMC) that integrates both the coarse and fine representations into SA-CRC. To demonstrate the effectiveness of the proposed methods, experiments are conducted on several public face databases in comparison with the state-of-the-art representation-based classification methods. The experimental results show that the proposed methods significantly outperform the competing RBC methods. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher's knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods. [ABSTRACT FROM AUTHOR]

Point cloud completion is prevalent due to the insufficient results from current point cloud acquisition equipments, where a large number of point data failed to represent a relatively complete shape. Existing point cloud completion algorithms, mostly encoder‐decoder structures with grids transform (also presented as folding operation), can hardly obtain a persuasive representation of input clouds due to the issue that their bottleneck‐shape result cannot tell a precise relationship between the global and local structures. For this reason, this article proposes a novel point cloud completion model based on a Stack‐Style Folding Network (SSFN). Firstly, to enhance the deep latent feature extraction, SSFN enhances the exploitation of shape feature extractor by integrating both low‐level point feature and high‐level graphical feature. Next, a precise presentation is obtained from a high dimensional semantic space to improve the reconstruction ability. Finally, a refining module is designed to make a more evenly distributed result. Experimental results shows that our SSFN produces the most promising results of multiple representative metrics with a smaller scale parameters than current models. [ABSTRACT FROM AUTHOR]

Graph neural networks (GNNs) have garnered significant attention for their ability to effectively process graph-related data. Most existing methods assume that the input graph is noise-free; however, this assumption is frequently violated in real-world scenarios, resulting in impaired graph representations. To address this issue, we start from the essence of graph structure learning, considering edge discovery and removal, reweighting of existing edges, and differentiability of the graph structure. We introduce virtual nodes and consider connections with virtual nodes to generate optimized graph structures, and subsequently utilize Gumbel-Softmax to reweight edges and achieve differentiability of the Graph Structure Learning (VN-GSL for abbreviation). We conducted a thorough evaluation of our method on a range of benchmark datasets under both clean and adversarial circumstances. The results of our experiments demonstrate that our approach exhibits superiority in terms of both performance and efficiency. Our implementation will be made available to the public. [ABSTRACT FROM AUTHOR]

Most of the previous regression models for predicting the remaining useful life (RUL) of hard disks primarily focused on failure data obtained through SMART monitoring. However, models that only considered failure data were not accurate enough. In order to address the issue of imbalanced sample quantities in different categories within the dataset, this paper proposes a method for predicting the RUL of hard disks under imbalanced conditions. It takes into account the imbalance issues in both SMART failure data and healthy data, as well as the imbalance in monitoring data attributes between hard disk drives and solid-state drives. The paper improves upon the traditional Transformer time series model and constructs a comprehensive and unified model for the RUL of hard disks across multiple categories. The proposed model achieves an RMSE score of 3.325, and has 66 % in MAE improved to the conventional models. The proposed model has promising prospects for future applications in predicting the remaining life of hard disks. [ABSTRACT FROM AUTHOR]

Platform merchants mine user clusters and their characteristics to assist in precision marketing. In view of the information overload in e-commerce reviews, machine methods are needed to efficiently obtain clustering information from text. This study innovatively integrated the emotional correlation analysis model and Self-organizing Map (SOM) in application, to construct fine-grained user emotion vector based on review text and perform visual cluster analysis, which helped quickly mine user clustering and characteristics from review text. The result of empirical analysis based on real reviews of Amazon books showed that the proposed method had the average precision as 0.71, confirming that the clustering method integrating the emotional correlation analysis model and SOM could efficiently mine user groups and match appropriate marketing strategies, which will help platform merchants carry out precision marketing. The study makes contributions to the application and innovation of researches in the field of user clustering and e-commerce precision marketing. [ABSTRACT FROM AUTHOR]

In multi-label classification, challenges arise from missing labels due to subjective analysis or label ambiguity. This makes it difficult to accurately capture label correlations and enhance classifier performance. Previous research has primarily focused on obtaining label correlations with sparsity assumptions from incomplete label matrices to fill in these missing labels. However, this approach leads to inaccurate label correlations since it relies on incomplete label matrix information. To overcome this issue and improve accuracy, this paper proposes a new method called LCFM, which effectively learns label correlations using a combination of global and local label-specific features to recover missing labels. Firstly, we rely on the sparsity assumption of Lasso regression coefficients to select label-specific features from the training samples. Secondly, we introduce l 2 -norm regularization to learn global label correlations through global label-specific features. simultaneously, we utilize similar label-specific features to describe local label correlations. Finally, we minimize the difference between predicted results and true labels using the binary cross-entropy loss function and employ the Accelerated Proximal Gradient (APG) algorithm for model optimization. Experimental results comparing LCFM with five benchmark indicators on nine datasets demonstrate that our proposed method outperforms other advanced methods, showing strong competitiveness. [ABSTRACT FROM AUTHOR]

Non-contact facial video-based physiological signal measurement using remote photoplethysmography (rPPG) has shown great potential in many applications (e.g. telemedicine, driver fatigue detection, sleep detection) and has achieved reliable results in the controlled environment. With the development of deep learning, more and more methods have been proposed. However, these methods were usually implemented based on model variations that have achieved better results in computer vision tasks, and finally chose average pooling layer or one-dimensional convolution layer directly reduce dimension. Such these operations ignored the fact that the physiological signals of one individual are consistent in different facial regions. In this paper, we propose a novel remote photoplethysmography framework that utilizes graph neural network to extract physiological signal, named GraphPhys. And we design the Average Relative GraphConv based on the fact that the physiological signals of one individual are always consistent in different facial regions for the remote physiological signal measurement task. We apply GraphPhys to two different architectures for rPPG methods, and the methods based on GraphPhys significantly outperform the original methods on the VIPL-HR (Niu et al., 2019) dataset. To the best of our knowledge, this is the first work to utilize graph neural network to aggregate physiological signals. The code is available at https://github.com/Xiong-JiaHao/GraphPhys. [ABSTRACT FROM AUTHOR]

Subspace clustering has attracted much attention on account of its excellent performance in various tasks, such as computer vision and pattern recognition. Since the high-dimensional data may comprise redundant information and noises, low-dimensional subspace structure is difficult to be recovered exactly. Existing methods adopt the non-convex Schatten- p norm to approximate the rank function closer. However, these methods are still unable to handle each singular value flexibly, which may yield unsatisfactory solutions. To address this deficiency, in this paper, we propose a iterative reweighted Frobenius norm regularized latent low rank representation (IRFLLRR) model, which can restrain the small and large rank components simultaneously with rational weights. Specifically, the reweighting strategy can preserve the desirable structure information, while removing the sparse noise and redundant information. Hence the constructed coefficient matrix can capture the global structure of data more adequately. To solve the proposed model, an efficient algorithm is developed via the alternating direction method of multipliers (ADMM), which ensures each subproblem can be optimized in closed-form. In addition, we provide detailed mathematical proofs to guarantee that the sequence generated by our algorithm convergences to a Karush–Kuhn–Tucker (KKT) point. Extensive experimental results reveal the superiority of the proposed method over several state-of-the-art methods. MATLAB code is available at https://github.com/wangzhi-swu/IRFLLRR. • Propose a new subspace clustering model. • Develop an efficient algorithm to solve the proposed model. • Provide detailed mathematical proofs to analyze the proposed algorithm. [ABSTRACT FROM AUTHOR]

With the development of medical informatization, a large number of patients' electronic medical records (EMRs) have been accumulated in the hospital information system, which is characterized by multi-structured data form, diversified professional vocabulary categories and fuzzy vocabulary demarcation. Natural language processing (NLP) provides a silver lining for parsing electronic medical records, and the mainstream methods include dictionary-based methods, rule-based and statistical methods, and machine learning methods. Due to the semantic richness and structural diversity of Chinese text, Chinese EMRs analysis methods are still scarce compared with English EMRs. In order to overcome the problems of unstructured, multiple meanings of words and unclear word boundaries in text of Chinese electronic medical records, this paper proposes a medical entity recognition method based on RoFormerV 2 -BiLSTM-CRF fusion model, using BIO annotation method to annotate the recognized medical entities, and using knowledge graph to analyze the medical entity relationships identified in single patient medical record, multiple patient medical records respectively. The relationships between the medical entities identified in a single patient record and multiple patient records are analyzed using knowledge graphs. The experimental analysis was conducted on the expert-annotated dataset CCKS2019, and the results showed that the proposed method was effective for recognizing "Diseases and Diagnoses", "Laboratory Tests", "Imaging Examinations", "Anatomical Sites", "Drugs" and "Surgery" in the dataset. The average accuracy, recall and F 1 -Score of the proposed method for the identification of the six medical entities, are 84.8%, 83.5% and 83.9%, respectively, which were 5.3%, 8.6% and 6.8% higher than the traditional Word2Vec-BiLSTM-CRF model, and the existing Word2Vec-BiLSTM-CRF and BERT-BiLSTM-CRF models were iteratively trained with the same evaluation. The experimental results show that the proposed model performs better in medical entity recognition. • Connect an optimized Transformer with the BiLSTM-CRF to form a medical entity recognition model. • Make Medical knowledge graphs based on a single medical record and multiple medical records. [ABSTRACT FROM AUTHOR]

Cross-modal hashing methods have attracted considerable attention due to their low memory usage and high query speed in large-scale cross-modal retrieval. During the encoding process, there still remains two crucial bottlenecks: how to equip hash codes with cross-modal semantic information, and how to rapidly obtain hash codes. In this paper, we propose Semantic Constraints Matrix Factorization Hashing (SCMFH) which simultaneously considers modality-specific and cross-modal semantic information for hash codes with closed solution. Specifically, the original representation of each modality is factorized into modality-specific semantic representations, and then the cross-modal semantic similarity matrices is used to constrain correlation of modality-specific representation. Finally, all the modality-specific representations are regressed to the common hash codes, and the fast hash codes with a closed solution is obtained. Extensive experimental have been carried out on public datasets. The results show that the method outperforms many current cross-modal hashing methods in terms of mean average precision (mAP), up to 2.7%. [ABSTRACT FROM AUTHOR]