Search

In this paper, we propose a novel annealed gradient descent (AGD) algorithm for deep learning. AGD optimizes a sequence of gradually improving smoother mosaic functions that approximate the original non-convex objective function according to an annealing schedule during optimization process. We present a theoretical analysis on AGD’s convergence properties and learning speed, as well as use some visualization methods to show its advantages. The proposed AGD algorithm is applied to learn both deep neural networks (DNNs) and Convolutional Neural Networks (CNNs) for variety of tasks includes image recognition and speech recognition. Experimental results on several widely-used databases, such as Switchboard, CIFAR-10 and Pascal VOC 2012, show that AGD yields better classification accuracy than SGD, and obviously accelerates the training speed of DNNs and CNNs.

Extreme learning machine (ELM) has been intensively studied during the last decade due to its high efficiency, effectiveness and easy to implement. Recently, a variant of ELM named local receptive fields based ELM (ELM-LRF) has been proposed to reduce the global connections and introduce local receptive fields to the input layer. However, an ELM-LRF model with large number of hidden neurons spend plenty of time on solving large scale Moore-Penrose Matrix Inversion (MPMI) problem which has heavy computational cost and needs much more runtime memory. Moreover, this procedure can not be directly accelerated by GPU platforms due to the limited memory of GPU devices. In this paper, we propose three efficient approaches to perform ELM-LRF on GPU platform. First we propose a novel blocked LU decomposition algorithm, which overcomes the limitation of global memory size so that any size of ELM-LRF models can be trained. Furthermore, an efficient blocked Cholesky decomposition algorithm is presented to accelerate blocked LU decomposition algorithm according to matrix characteristics in the ELM-LRF model. Finally we present a heterogeneous blocked CPU-GPU parallel algorithm to fully exploit resources on a GPU node such as to accelerate blocked Cholesky decomposition algorithm furthermore in the ELM-LRF model.

Target detection is a hard real-time task for video and image processing. This task has recently been accomplished through the feedforward process of convolutional neural networks (CNN), which is usually accelerated by general-purpose graphic units (GPUs). However, there are two challenges for this task. One is that the running speed remains to be improved. The other is that we probably use a deeper and larger CNN model, but a more sophisticated model may not be trained well due to the shortage of GPU memory. In this paper, we present two scheduling algorithms to solve the aforementioned challenges for improving the system performance holistically. The first one is an efficient image combination algorithm used to accelerate the feedforward process of CNN. The other is a light-memory-cost algorithm used to train an arbitrarily large CNN model for a GPU device with a limited memory. We run our experiments on a GTX980 card and use a CNN model with 8GB of model parameters, which is larger than the size of the global memory of a GPU. Compared with that of cuDNNv3, a high speedup of 6.97x is obtained in the detection task.

A wide variety of kernel-based methods have been developed with great successes in many fields, but very little research has focused on the reproducing kernel function in Reproducing Kernel Hilbert Space (RKHS). In this paper, we propose a novel method which we call a local-global mixed kernel with reproducing property (LGMKRP) to successfully perform a range of classification tasks in the RKHS rather than the more conventionally used Hilbert space. The LGMKRP proposed in this paper consists of two major components. First, we find the basic solution of a generalized differential operator by the delta function, and prove that this basic solution is a new specific reproducing kernel called a local H-reproducing kernel (LHRK) in RKHS. This reproducing kernel has good local properties, including odd order vanishing moment, and fast dilation attenuation. Second, in the RKHS, we prove that the LHRK satisfies the condition of Mercer's theorem, and prove that it is a typical polynomial kernel with global property, which also possesses the reproducing property. Furthermore, the novel specific mixed kernel (i.e., LGMKRP) proposed in this paper is based on these two different properties. Experimental results demonstrate that the LGMKRP possesses the approximation and regularization performance of a reproducing kernel, and can enhance the generalization ability of kernel methods. We find the basic solution of a generalized differential operator, and prove that this basic solution is a new specific reproducing kernel.We prove that the local H-reproducing kernel satisfies the condition of Mercer kernel.We prove that the typical polynomial kernel with global property possesses reproducing property.We define a novel method named local-global mixed kernel with reproducing property.We evaluate the performance of our mixed kernel on standard UCI datasets.We demonstrate the effectiveness of the proposed mixed kernel.