1. Toward Near-Real-Time Training With Semi-Random Deep Neural Networks and Tensor-Train Decomposition
- Author
-
Uttam Majumder, Ryan Bryla, Dhireesha Kudithipudi, and Humza Syed
- Subjects
Atmospheric Science ,neural networks with random weights ,QC801-809 ,Computer science ,Random projection ,Geophysics. Cosmic physics ,Training (meteorology) ,synthetic aperture radar (SAR) ,Computational resource ,Convolutional neural networks (CNNs) ,Matrix decomposition ,Ocean engineering ,extreme learning machine (ELM) ,Memory management ,Compression (functional analysis) ,Decomposition (computer science) ,Network performance ,Computers in Earth Sciences ,TC1501-1800 ,Algorithm ,image classification ,random vector functional link (RVFL) - Abstract
In recent years, deep neural networks have shown to achieve state-of-the-art performance on several classification and prediction tasks. However, these networks demand undesirable lengthy training times coupled with high computational resources (memory, I/O, processing time). In this work, we explore semi-random deep neural networks to achieve near real-time training and less computational resource usage. Although many works enhance the underlying hardware for real-time training, this work focuses on algorithmic optimization. It is shown that random projection networks with additional skipped connectivity and randomly weighted layers can boost the overall network performance while enabling for real-time training. Additionally, a tensor-train decomposition technique is leveraged to further reduce the model complexity of these networks. Our investigation accomplishes the following: 1) Tensor-train decomposition decreases the complexity of random projection networks, 2) compression of the fully connected hidden layer leads to a minimum $\sim40\times$ decrease in memory size, and 3) training under random projection networks can be achieved in near-real time.
- Published
- 2021