Search

A general deep learning (DL) mechanism for a multiple hidden layer feed-forward neural network contains two parts, i.e., 1) an unsupervised greedy layer-wise training and 2) a supervised fine-tuning which is usually an iterative process. Although this mechanism has been demonstrated in many fields to be able to significantly improve the generalization of neural network, there is no clear evidence to show which one of the two parts plays the essential role for the generalization improvement, resulting in an argument within the DL community. Focusing on this argument, this paper proposes a new DL approach to train multilayer feed-forward neural networks. This approach uses restricted Boltzmann machine (RBM) as the layer-wise training and uses the generalized inverse of a matrix as the supervised fine-tuning. Different from the general deep training mechanism like back-propagation (BP), the proposed approach does not need to iteratively tune the weights, and therefore, has many advantages such as quick training, better generalization, and high understandability, etc. Experimentally, the proposed approach demonstrates an excellent performance in comparison with BP-based DL and the traditional training method for multilayer random weight neural networks. To a great extent, this paper demonstrates that the supervised part plays a more important role than the unsupervised part in DL, which provides some new viewpoints for exploring the essence of DL. [ABSTRACT FROM AUTHOR]

Deep computation models achieve super performance for big data feature learning. However, training a deep computation model poses a significant challenge since a deep computation model typically involves a large number of parameters. Specially, it needs a high-performance computing server with a large-scale memory and a powerful computing unit to train a deep computation model, making it difficult to increase the size of a deep computation model further for big data feature learning on low-end devices such as conventional desktops and portable CPUs. In this paper, we propose an improved deep computation model based on the canonical polyadic decomposition scheme to compress the parameters and to improve the training efficiency. Furthermore, we devise a learning algorithm based on the back-propagation strategy to train the parameters of the proposed model. The learning algorithm can be directly performed on the compressed parameters to improve the training efficiency. Finally, we carry on the experiments on three representative datasets, i.e., CUAVE, SNAE2, and STL-10, to evaluate the performance of the proposed model by comparing with the conventional deep computation model and other two improved deep computation models based on the Tucker decomposition and the tensor-train network. Results demonstrate that the proposed model can compress parameters greatly and improve the training efficiency significantly with a low classification accuracy drop. [ABSTRACT FROM AUTHOR]

In lifelong machine learning (LML) systems, consecutive new tasks from changing circumstances are learned and added to the system. However, sufficiently labeled data are indispensable for extracting intertask relationships before transferring knowledge in classical supervised LML systems. Inadequate labels may deteriorate the performance due to the poor initial approximation. In order to extend the typical LML system, we propose a novel hierarchical lifelong learning algorithm (HLLA) consisting of two following layers: 1) the knowledge layer consisted of shared representations and integrated knowledge basis at the bottom and 2) parameterized hypothesis functions with features at the top. Unlabeled data is leveraged in HLLA for pretraining of the shared representations. We also have considered a selective inherited updating method to deal with intertask distribution shifting. Experiments show that our HLLA method outperforms many other recent LML algorithms, especially when dealing with higher dimensional, lower correlation, and fewer labeled data problems. [ABSTRACT FROM AUTHOR]

The ever-increasing complexity of robot applications induces the need for methods to approach problems with no (viable) analytical solution. Deep learning (DL) provides a set of tools to address this kind of problems. This survey presents a categorization of the major challenges in robotics that leverage DL technologies and introduces representative examples of successful solutions for the described problems. We also consider the question when and whether to use modular, monolithic models or end-to-end DL, in order to provide a guideline for the selection of the correct model structure and training strategy. By doing so, the current role and adaptability of different techniques at different hierarchical levels of a robot-application can be highlighted, thus providing a well-structured basis to assist future approaches. [ABSTRACT FROM AUTHOR]