Search

Catastrophic forgetting is a chronic problem during the online training process of deep neural networks. That is, once a new data set is used to train an existing neural network, the network will lose the ability to recognize the original data set. In literature, online contrastive divergence (CD) with generative replay (GR) exploits the generative capacity of the neural network to facilitate online training. It greatly alleviates catastrophic forgetting but cannot totally eliminate it. To overcome this shortcoming and further solve the challenging issue, in this article, we propose a novel approach named asynchronous dual-channel online restricted Boltzmann machine, where online CD with dual-channel GR plays an important role in further eliminating catastrophic forgetting. The asynchronous gradient estimation, by which the Markov chain sampling and the network calculation are conducted asynchronously on separate computing nodes, is designed to speed up training. The experimental results show that the proposed method outperforms several algorithms in increasing training speed and minimizing catastrophic forgetting. Besides, online learning with dual-channel can be effectively extended to other online learning neural networks with GR and has achieved excellent results in our verification experiments. [ABSTRACT FROM AUTHOR]

In the context of human-robot and robot-robot interactions, the better cooperation can be achieved by predicting the other party's subsequent actions based on the current action of the other party. The time duration for adjustment is not sufficient provided by short term forecasting models to robots. A longer duration can by achieved by mid-term forecasting. But the mid-term forecasting models introduce the previous errors into the follow-up forecasting and amplified gradually, eventually invalidating the forecasting. A new mid-term forecasting with error suppression based on restricted Boltzmann machine(RBM) is proposed in this paper. The proposed model can suppress the error amplification by replacing the previous inputs with their features, which are retrieved by a deep belief network(DBN). Furthermore, a new mechanism is proposed to decide whether the forecasting result is accepted or not. The model is evaluated with several datasets. The reported experiments demonstrate the superior performance of the proposed model compared to the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

During data acquisition, samples in a time series may contain noise, such as inconsistent data ranges, inconsistent data, and incomplete data. Therefore, the classification model requires improved robustness to correctly classify the sequence of human motion. This paper presents a classification model with improved robustness performance based on the factored gated restricted Boltzmann machine to effectively overcome the various aforementioned data problems. The proposed model acquires the deformation code of each action first and integrates the deformation codes together to be an integrated deformation code of the entire sequence. Then, the model determines the classification from the integrated deformation code. This approach mainly focuses on the deformation relations among action samples in the extraction sequence, and it ignores the data expression in the sequence samples. Experiments show that the proposed model performs better than state-of-the-art approaches in terms of the robustness of time series classification with noise. [ABSTRACT FROM AUTHOR]

Although existing sparse restricted Boltzmann machine (SRBM) can make some hidden units activated, the major disadvantage is that the sparseness of data distribution is usually overlooked and the reconstruction error becomes very large after the hidden unit variables become sparse. Different from the SRBMs which only incorporate a sparse constraint term in the energy function formula from the original restricted Boltzmann machine (RBM), an energy function constraint SRBM (ESRBM) is proposed in this paper. The proposed ESRBM takes into account the sparseness of the data distribution so that the learned features can better reflect the intrinsic features of data. Simulations show that compared with SRBM, ESRBM has smaller reconstruction error and lower computational complexity, and that for supervised learning classification, ESRBM obtains higher accuracy rates than SRBM, classification RBM, and Softmax classifier. [ABSTRACT FROM AUTHOR]

Abstract Temporal dependency plays a fundamental role in nonlinear generative models for capturing temporal features. One such model, the conditional restricted Boltzmann machine (CRBM), learns these temporal dependencies by considering the visible variables in the previous time slice as additional fixed inputs so that static and temporal features can be captured simultaneously to generate new human motions. However, the temporal dependencies in the CRBM fail to describe various common equilibrium postures in human motion. In this paper, we present cross-correlation as a new representation for modeling temporal dependencies by introducing the Pearson correlation coefficient. We also propose an approach to enhance the discrimination of the CRBM for various human motions by incorporating cross-correlation and temporal dependency features. The experimental results on benchmark databases demonstrate that the proposed method not only retains all the merits of the CRBM, such as exact inference and efficient learning, but also greatly improves the model’s ability to blend motion styles and achieve smooth transitions between various motion segments. [ABSTRACT FROM AUTHOR]

Many models are used to solve classification problems in machine learning. The classification restricted Boltzmann machine (ClassRBM) is a type of self-contained network model that is widely used in various classification applications. To implement classification, the ClassRBM updates the model parameters constantly during the training phase in terms of their class labels so that the model parameters learned from the ClassRBM are different from those learned from the conventional restricted Boltzmann machine (RBM), which is trained by unsupervised learning. In this paper, we demonstrate that the reconstruction errors of the ClassRBM are larger than those of the conventional RBM because of the label information. We then propose a classification model of the restricted Boltzmann machine based on these reconstruction errors. The reconstruction errors are used to train the proposed model to improve the classification performance of the ClassRBM. Extensive experiments are carried out to verify the proposed model. The experimental results demonstrate that the proposed model can improve the classification performance of the ClassRBM. [ABSTRACT FROM AUTHOR]

Restricted Boltzmann machines (RBMs) have been successfully applied in unsupervised learning and image density-based modeling. The aim of the pre-training step for RBMs is to discover an unknown stationary distribution based on the sample data that has the lowest energy. However, conventional RBM pre-training is sensitive to the initial weights and bias. The selection of initial values in RBM pre-training will directly affect the capabilities and efficiency of the learning process. This paper uses principal component analysis to capture the principal component directions of the training data. A set of initial parameter values for the RBM can be obtained by computing the same reconstruction of the data. Experiments on the Yale and MNIST datasets show that the proposed method not only retains a strong learning ability, but also significantly accelerates the learning speed. [ABSTRACT FROM AUTHOR]