Search

Variable selection is one of the most important contents of machine learning research and a large number of variable selection methods have been proposed at present. From deep learning perspective, we propose a new variable selection method based on multi-layer artificial neural networks in this paper. The method propagates errors gained form a trained network model through the network from output layer to input layer and the values gained by input layer give information about the relative contribution. The variables with big contribution value are selected, which we call neural inverse propagation (NIP). Specifically, after training a neural network, the gained errors between the actual network outputs and the desired outputs go through the network from top to down and arrive at the input layer. The absolute values gained by input units provide information about the relative importance (contribution) of input units, which is that the larger the value is, the more important the corresponding input variable is. We remove k variables corresponding to the first k minimum absolute value gained by input units to the predictive variables, to perform variable selection. The efficiency of the proposed method is illustrated through its application to classification and regression tasks on a number of synthetic datasets and a real-world image classification dataset. [ABSTRACT FROM AUTHOR]

Product quality is the lifeline of enterprise survival and development. With the rapid development of information technology, the semiconductor manufacturing process produces multitude of quality features. Due to the increasing quality features, the requirement on the training time and classification accuracy of quality prediction methods becomes increasingly higher. Aiming at realizing the quality prediction for semiconductor manufacturing process, this paper proposes a modified support vector machine (SVM) model based on feature selection, considering the high dimensional and nonlinear characteristics of data. The model first improves the Radial Basis Function (RBF) in SVM, and then combines the Duelist algorithm (DA) and variable neighborhood search algorithm (VNS) for feature selection and parameters optimization. Compared with some other SVM models that are based on DA, genetic algorithm (GA), and Information Gain algorithm (IG), the experiment results show that our DA-VNS-SVM can obtain higher classification accuracy rate with a smaller feature subset. In addition, we compare the DA-VNS-SVM with some common machine learning algorithms such as logistic regression, naive Bayes, decision tree, random forest, and artificial neural network. The results indicate that our model outperform these machine learning algorithms for the quality prediction of semiconductor. [ABSTRACT FROM AUTHOR]

Artificial neural networks, consisting of many levels of nonlinearities, have been widely used to deal with various supervised learning tasks. At present, the most popular and effective training method is back-propagation algorithm (BP). Inspired by manifold regularization framework, we introduce a novel regularization framework, which aims at preserving the inter-object-distance of the data. Then a refined BP algorithm (IOD-BP) is proposed by imposing the proposed regularization framework into the objective function of BP algorithm. Comparative experiments on various benchmark classification tasks show that the new regularization BP method significantly improves the performance of BP algorithm in terms of classification accuracy. [ABSTRACT FROM AUTHOR]

Auto-encoders have been proved to be powerful unsupervised learning methods that able to extract useful features from input data or construct deep artificial neural networks by recent studies. In such settings, the extracted features or the initialized networks only learn the data structure while contain no class information which is a disadvantage in classification tasks. In this paper, we aim to leverage the class information of input to learn a reconstructive and discriminative auto-encoder. More specifically, we introduce a supervised auto-encoder that combines the reconstruction error and the classification error to form a unified objective function while taking the noisy concatenate data and label as input. The noisy concatenate input is constructed in such a method that one third has only original data and zero labels, one third has only label and zero data, the last one third has both original data and label. We show that the representations learned by the proposed supervised auto-encoder are more discriminative and more suitable for classification tasks. Experimental results demonstrate that our model outperforms many existing learning algorithms. [ABSTRACT FROM AUTHOR]

The objective of the relation classification task is to extract relations between entities. Recent studies have found that R-BERT (Wu and He, 2019) based on pre-trained BERT (Devlin et al., 2019) acquires extremely good results in the relation classification task. However, this method does not take into account the semantic differences between different kinds of entities and global semantic information either. In this paper, we set two different fully connected layers to take into account the semantic difference between subject and object entities. Besides, we build a new module named Concat Module to fully fuse the semantic information among the subject entity vector, object entity vector, and the whole sample sentence representation vector. In addition, we apply the average pooling to acquire a better entity representation of each entity and add the activation operation with a new fully connected layer after our Concat Module. Modifying R-BERT, we propose a new model named BERT with Global Semantic Information (GSR-BERT) for relation classification tasks. We use our approach on two datasets: the SemEval-2010 Task 8 dataset and the Chinese character relationship classification dataset. Our approach achieves a significant improvement over the two datasets. It means that our approach enjoys transferability across different datasets. Furthermore, we prove that these policies we used in our approach also enjoy applicability to named entity recognition task. • We take into account the semantic difference between subject and object entities. • We fully fuse the semantic information between entities and the whole sentence. • These policies we used enjoy high applicability to different NLP tasks. [ABSTRACT FROM AUTHOR]

Deep learning has become one of very important machine learning methods in image classification, but most of them require a long training time to solve a non-convex optimization problem. In comparison, the training of extreme learning machine (ELM) is very simple, fast and effective. In order to combine the advantages of both methods, many researchers have tried to introduce ELM to deep architectures (Kasun et al. in IEEE Intell Syst 28:31-34, 2013; Yu et al. in Neurocomputing 149:308-315, 2015; Tissera and McDonnell in Neurocomputing 174:42-49, 2016 and in: Proceedings of ELM-2014, vol 1, Proceedings in adaptation, learning and optimization, vol 3, 2016; Junying et al. in Neurocomputing 171:63-72, 2016; Uzair et al. in Neural Comput Appl, 2015) to solve unsupervised learning and supervised learning problems. In this paper, we propose a new deep neural network based on ELM called discriminative deep ELM (DDELM) to address the semi-supervised learning problems in image classification. The proposed deep architecture consists of several stacked unsupervised ELMs and an additional label layer on the top layer of the stacked model. Experiments on three standard image data show that DDELM outperforms both representative semi-supervised learning algorithms and existing deep architectures such as DCNN in terms of accuracy and training time. [ABSTRACT FROM AUTHOR]

The Helmholtz machine is an unsupervised deep neural network with different bottom-up recognition weights and top-down generative weights, which attempts to build probability density models of sensory inputs. The recognition weights are used to determine the recognition probability of each unit from bottom layer to top layer and the generative weights are used to determine the generative probability of each unit from top layer to bottom layer. The model parameters can be gained by minimizing the sum of the Kullback–Leibler divergence between generative and recognition distributions of all units. In this paper, we proposed a modified Helmholtz machine by adding an additional hidden layer on the top layer of the Helmholtz machine, which is used to model the generative probability of the top layer. The additional added hidden layer provides ‘complementary prior’ to the original top layer and can eliminate the ‘explaining away effects’ to make the Helmholtz machine fitting sensory inputs much better. The experimental results of new algorithm on various data sets show that the modified Helmholtz machine learns better generative models. [ABSTRACT FROM AUTHOR]

• Biological studies have shown that one neuron simultaneously releases different types of neurotransmitters to another neuron for information transfer, of which different types of neurotransmitters play different roles. Motivated by this biological discovery, a novel neural networks with Multiple Connection Weights (MNN) is proposed. • The greedy layer-by-layer pre-training method is proposed to pre-train MNN model. • The universal approximation of MNN has been proved. • The experimental results on MNIST, NORB and seven UCI datasets have been done and show that the performance of MNN outperformed the traditional neural network. • The dimension expansion of weights provide new insights into structure design of the neural network model. Biological studies have shown that the interaction between neurons are based on neurotransmitters, which transmit signals between neurons, and that one neuron sends information to another neuron by releasing a number of different neurotransmitters, which play different roles. Motivated by this biological discovery, a novel neural networks model is proposed by extending the dimension of connection weights from one to multiple, i.e. there are multiple not only one connections between each two units. The number of dimensions of connection weight represents the number of categories of neurotransmitters and different components of the weight correspond to different neurotransmitters. In order to make these neurotransmitters collaborate and compete appropriately, the input and output for each unit in our proposed model have been heuristically defined. From the biological perspective, the proposed neural network is much closer to biological neural network. From the viewpoint of new model structure, the characteristic that the activation of each hidden unit is based on several filters, can improve the interpretability of features learned by the proposed neural network. Experimental results on MNIST, NORB and several other data sets have demonstrated that the performances of traditional neural networks can be improved by extending the dimension of connection weight between units, and the idea of multiple connection weights provides a new paradigm for the design of neural networks. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) is an efficient learning algorithm of training single layer feed-forward neural networks (SLFNs). With the development of unsupervised learning in recent years, integrating ELM with autoencoder has become a new perspective for extracting feature using unlabeled data. In this paper, we propose a new variant of extreme learning machine autoencoder (ELM-AE) called generalized extreme learning machine autoencoder (GELM-AE) which adds the manifold regularization to the objective of ELM-AE. Some experiments carried out on real-world data sets show that GELM-AE outperforms some state-of-the-art unsupervised learning algorithms, including k -means, laplacian embedding (LE), spectral clustering (SC) and ELM-AE. Furthermore, we also propose a new deep neural network called multilayer generalized extreme learning machine autoencoder (ML-GELM) by stacking several GELM-AE to detect more abstract representations. The experiments results show that ML-GELM outperforms ELM and many other deep models, such as multilayer ELM autoencoder (ML-ELM), deep belief network (DBN) and stacked autoencoder (SAE). Due to the utilization of ELM, ML-GELM is also faster than DBN and SAE. [ABSTRACT FROM AUTHOR]