Search

Neural networks have been extensively applied to a variety of tasks, achieving astounding results. Applying neural networks in the scientific field is an important research direction that is gaining increasing attention. In scientific applications, the scale of neural networks is generally moderate size, mainly to ensure the speed of inference during application. Additionally, comparing neural networks to traditional algorithms in scientific applications is inevitable. These applications often require rapid computations, making the reduction in neural network sizes increasingly important. Existing work has found that the powerful capabilities of neural networks are primarily due to their nonlinearity. Theoretical work has discovered that under strong nonlinearity, neurons in the same layer tend to behave similarly, a phenomenon known as condensation. Condensation offers an opportunity to reduce the scale of neural networks to a smaller subnetwork with a similar performance. In this article, we propose a condensation reduction method to verify the feasibility of this idea in practical problems, thereby validating existing theories. Our reduction method can currently be applied to both fully connected networks and convolutional networks, achieving positive results. In complex combustion acceleration tasks, we reduced the size of the neural network to 41.7% of its original scale while maintaining prediction accuracy. In the CIFAR10 image classification task, we reduced the network size to 11.5% of the original scale, still maintaining a satisfactory validation accuracy. Our method can be applied to most trained neural networks, reducing computational pressure and improving inference speed.

Abstract It has been an important approach of using matrix completion to perform image restoration. Most previous works on matrix completion focus on the low‐rank property by imposing explicit constraints on the recovered matrix, such as the constraint of the nuclear norm or limiting the dimension of the matrix factorisation component. Recently, theoretical works suggest that deep linear neural network has an implicit bias towards low rank on matrix completion. In this work, a regularised deep matrix factorised (RDMF) model for image restoration is proposed, which utilises the implicit bias of the low rank of deep neural networks and the explicit bias of total variation. RDMF is a powerful and flexible framework for inverse problems in image processing while the combination of implicit and explicit regularisation represents the intrinsic characteristics of a natural image. The effectiveness of the RDMF model with extensive experiments are demonstrated, in which the method surpasses the recently proposed models in common examples, especially for the restoration from very few observations. This work sheds light on a more general framework for solving other inverse problems by combining the implicit bias of deep learning with explicit regularisation.

Motivated by the impressive success of deep learning in a wide range of scientific and industrial applications, we explore in this work the application of deep learning into a specific class of optimization problems lacking explicit formulas for both objective function and constraints. Such optimization problems exist in many design problems, e.g., rotor profile design, in which objective and constraint values are available only through experiment or simulation. They are especially challenging when design parameters are high-dimensional due to the curse of dimensionality. In this work, we propose a data-informed deep optimization (DiDo) approach emphasizing on the adaptive fitting of the the feasible region as follows. First, we propose a deep neural network (DNN) based adaptive fitting approach to learn an accurate DNN classifier of the feasible region. Second, we use the DNN classifier to efficiently sample feasible points and train a DNN surrogate of the objective function. Finally, we find optimal points of the DNN surrogate optimization problem by gradient descent. To demonstrate the effectiveness of our DiDo approach, we consider a practical design case in industry, in which our approach yields good solutions using limited size of training data. We further use a 100-dimension toy example to show the effectiveness of our approach for higher dimensional problems. Our results indicate that, by properly dealing with the difficulty in fitting the feasible region, a DNN-based method like our DiDo approach is flexible and promising for solving high-dimensional design problems with implicit objective and constraints.

Maximum entropy principle (MEP) analysis with few non-zero effective interactions successfully characterizes the distribution of dynamical states of pulse-coupled networks in many fields, e.g., in neuroscience. To better understand the underlying mechanism, we found a relation between the dynamical structure, i.e., effective interactions in MEP analysis, and the anatomical coupling structure of pulse-coupled networks and it helps to understand how a sparse coupling structure could lead to a sparse coding by effective interactions. This relation quantitatively displays how the dynamical structure is closely related to the anatomical coupling structure.