Search

• Empirical observations on convolutional neural networks are presented. • The size of the receptive field did not guarantee improved accuracy. • We observed that dead pixels exist in modern convolutional neural networks. • The effect of dead pixels is discussed with a solution to resolve them. Deep neural networks have been used in various fields, but their internal behavior in how they understand images is not well known. In this study, we discuss two counterintuitive properties of convolutional neural networks (CNNs). First, we evaluated the size of the receptive field of CNNs with their classification accuracy. Previous studies have attempted to increase the size of the receptive field for performance gain. However, we observed that some CNNs with a smaller receptive field can achieve higher classification accuracy. In this regard, we claim that a larger receptive field does not guarantee improved classification accuracy. Second, using the effective receptive field, we examined the contribution of each pixel to the output of CNN. Intuitively, each pixel is expected to equally contribute to the final output, but we found that there exist pixels in a partially dead state with little contribution to the output. We reveal that the reason for dead pixels lies in even stride operations with odd-sized kernels in CNN and propose a kernel padding method to remove the dead pixels. We demonstrated the vulnerability of CNNs with dead pixels when we detect a noise or small box that is on dead pixels. Our findings on dead pixels should be understood and considered in practical applications of CNN. [ABSTRACT FROM AUTHOR]

Recently, various normalization layers have been proposed to stabilize the training of deep neural networks. Among them, group normalization is a generalization of layer normalization and instance normalization by allowing a degree of freedom in the number of groups it uses. However, to determine the optimal number of groups, trial-and-error-based hyperparameter tuning is required, and such experiments are time-consuming. In this study, we discuss a reasonable method for setting the number of groups. First, we find that the number of groups influences the gradient behavior of the group normalization layer. Based on this observation, we derive the ideal number of groups, which calibrates the gradient scale to facilitate gradient descent optimization. This paper is the first to propose an optimal number of groups that is theoretically grounded, architecture-aware, and can provide a proper value in a layer-wise manner for all layers. The proposed method exhibited improved performance over existing methods in numerous neural network architectures, tasks, and datasets. • We propose a method to determine the number of groups of group normalization. • A theoretical analysis of group normalization with activation function is provided. • The proposed method is validated against various practical problems. [ABSTRACT FROM AUTHOR]

• We provide empirical tests on various image corruption using vision transformers. • Vision transformers showed performance degradation on contrast-enhanced images. • We proposed PreLayerNorm for the consistent behavior of positional embedding. • We observed that PreLayerNorm improved performance on contrast-enhanced images. • We provide theoretical analyses on the inconsistent behavior of vision transformers. [Display omitted] Vision Transformers (ViTs) have recently demonstrated state-of-the-art performance in various vision tasks, replacing convolutional neural networks (CNNs). However, because ViT has a different architectural design than CNN, it may behave differently. To investigate whether ViT has a different performance or robustness, we tested ViT and CNN under various imaging conditions in practical vision tasks. We confirmed that for most image transformations, ViT's robustness was comparable or even better than that of CNN. However, for contrast enhancement, ViT performed particularly poorly. We show that this is because positional embedding in ViT's patch embedding can work improperly when the color scale changes. We demonstrate that the use of PreLayerNorm, a modified patch embedding structure, ensures the consistent behavior of ViT. Results demonstrate that ViT with PreLayerNorm exhibited improved robustness in the contrast-varying environments. [ABSTRACT FROM AUTHOR]

Predicting electrical power consumption is essential in many industries. This paper presents a method to predict total electrical power consumption in a certain period of Hot Rolling Mill (HRM) in steel mills. We adequately pre-processed operation data, including work schedule, work result, and power measurement. In addition, based on domain knowledge, we derived several useful variables relevant to power consumption. To predict power consumption, we developed a Multi-Level Stacked Regression model that combines the strengths of multiple prediction models. Real-world applications have dataset shift characteristics in which the covariance between features or distribution of the features changes over time due to frequent changes in the working environment. To verify the generalization accuracy of our model, we trained and tested our model for various training/test splits and compared it with several comparative models. Our model showed the best accuracy in all training/test splits. We verified whether the predictive model reflects the real-world environment well by analyzing the actual physical meaning of the essential features used in the predictive model. • A series of pre-processing methods are applied to real data from Hot Rolling Mill. • A Multi-Level Stacked Regression algorithm is proposed. • Performance on various training/test splits that contains dataset shifts is verified. [ABSTRACT FROM AUTHOR]