Search

Neural network compression is a widely used technique when deploying the neural network in energy-constrained and computation-constrained devices. To guarantee the compressed neural network is still usable without too much accuracy loss, the equivalence between the compressed networks and the original ones must be verified. However, the current verification approach can only verify the equivalence of the two neural networks in the same structure which is limited and unseen in real-world scenarios. In this paper, we proposed an equivalence verification method named VeriPrune, which can verify the equivalence of the deep neural networks without the structure limitation. In detail, we proposed an innovative virtual node completing method to solve the problem that node pruning causes structure change and invalidates the existing equivalence verification approaches. To demonstrate the feasibility and efficiency of the proposed approach, we conducted experiments on the public dataset with 49 DNNs and 1272 properties. The results show that the 83.9% properties, the 1067 of the total 1272 properties can be verified. The proposed VeriPrune can be further developed as a CASE tool for the industry settings. [Display omitted] [ABSTRACT FROM AUTHOR]

Recently, various normalization layers have been proposed to stabilize the training of deep neural networks. Among them, group normalization is a generalization of layer normalization and instance normalization by allowing a degree of freedom in the number of groups it uses. However, to determine the optimal number of groups, trial-and-error-based hyperparameter tuning is required, and such experiments are time-consuming. In this study, we discuss a reasonable method for setting the number of groups. First, we find that the number of groups influences the gradient behavior of the group normalization layer. Based on this observation, we derive the ideal number of groups, which calibrates the gradient scale to facilitate gradient descent optimization. This paper is the first to propose an optimal number of groups that is theoretically grounded, architecture-aware, and can provide a proper value in a layer-wise manner for all layers. The proposed method exhibited improved performance over existing methods in numerous neural network architectures, tasks, and datasets. • We propose a method to determine the number of groups of group normalization. • A theoretical analysis of group normalization with activation function is provided. • The proposed method is validated against various practical problems. [ABSTRACT FROM AUTHOR]