Back to Search
Start Over
Learnability of the Boolean Innerproduct in Deep Neural Networks.
- Source :
- Entropy; Aug2022, Vol. 24 Issue 8, p1117-N.PAG, 14p
- Publication Year :
- 2022
-
Abstract
- In this paper, we study the learnability of the Boolean inner product by a systematic simulation study. The family of the Boolean inner product function is known to be representable by neural networks of threshold neurons of depth 3 with only 2 n + 1 units (n the input dimension)—whereas an exact representation by a depth 2 network cannot possibly be of polynomial size. This result can be seen as a strong argument for deep neural network architectures. In our study, we found that this depth 3 architecture of the Boolean inner product is difficult to train, much harder than the depth 2 network, at least for the small input size scenarios n ≤ 16 . Nonetheless, the accuracy of the deep architecture increased with the dimension of the input space to 94% on average, which means that multiple restarts are needed to find the compact depth 3 architecture. Replacing the fully connected first layer by a partially connected layer (a kind of convolutional layer sparsely connected with weight sharing) can significantly improve the learning performance up to 99% accuracy in simulations. Another way to improve the learnability of the compact depth 3 representation of the inner product could be achieved by adding just a few additional units into the first hidden layer. [ABSTRACT FROM AUTHOR]
- Subjects :
- ARTIFICIAL neural networks
CIRCUIT complexity
DEEP learning
Subjects
Details
- Language :
- English
- ISSN :
- 10994300
- Volume :
- 24
- Issue :
- 8
- Database :
- Complementary Index
- Journal :
- Entropy
- Publication Type :
- Academic Journal
- Accession number :
- 158806884
- Full Text :
- https://doi.org/10.3390/e24081117