Search

Pruning generates sparse networks by setting parameters to zero. In this work we improve one-shot pruning methods, applied before training, without adding any additional storage costs while preserving the sparse gradient computations. The main difference to pruning is that we do not sparsify the network’s weights but learn just a few key parameters and keep the other ones fixed at their random initialized value. This mechanism is called freezing the parameters. Those frozen weights can be stored efficiently with a single 32bit random seed number. The parameters to be frozen are determined one-shot by a single for- and backward pass applied before training starts. We call the introduced method FreezeNet. In our experiments we show that FreezeNets achieve good results, especially for extreme freezing rates. Freezing weights preserves the gradient flow throughout the network and consequently, FreezeNets train better and have an increased capacity compared to their pruned counterparts. On the classification tasks MNIST and CIFAR-10/100 we outperform SNIP, in this setting the best reported one-shot pruning method, applied before training. On MNIST, FreezeNet achieves \(99.2\%\) performance of the baseline LeNet-5-Caffe architecture, while compressing the number of trained and stored parameters by a factor of \(\times 157\).

We discuss the statistical bundle of the manifold of two-variate stricly positive probability functions with given marginals. The fiber associated to each coupling turns out to be the vector space of interacions in the ANOVA decomposition with respect to the given weight. In this setting, we derive the form of the gradient flow equation for the Kantorovich optimal transport problem.

We propose an innovative algorithm that iteratively evolves a particle system to approximate the sample-wised Optimal Transport plan for given continuous probability densities. Our algorithm is proposed via the gradient flow of certain functional derived from the Entropy Transport Problem constrained on probability space, which can be understood as a relaxed Optimal Transport problem. We realize our computation by designing and evolving the corresponding interacting particle system. We present theoretical analysis as well as numerical verifications to our method.

We show that the gradient flows associated with a recently found family of Morse functions converge exponentially to the roots of the symmetric continuous Hahn polynomials. By symmetry reduction the rate of the exponential convergence can be improved, which is clarified by comparing with corresponding gradient flows for the roots of the Wilson polynomials.

In a previous work by the authors a second order gradient flow of the p-elastic energy for a planar theta-network of three curves with fixed lengths was considered and a weak solution of the flow was constructed by means of an implicit variational scheme. Long-time existence of the evolution and convergence to a critical point of the energy were shown. The purpose of this note is to prove uniqueness of the weak solution when p = 2.

Batch normalization was introduced as a novel solution to help with training fully-connected feed-forward deep neural networks. It proposes to normalize each training-batch in order to alleviate the problem caused by internal covariate shift. The original method claimed that Batch Normalization must be performed before the ReLu activation in the training process for optimal results. However, a second method has since gained ground which stresses the importance of performing BN after the ReLu activation in order to maximize performance. In fact, in the source code of PyTorch, common architectures such as VGG16, ResNet and DenseNet have Batch Normalization layer after the ReLU activation layer. Our work is the first to demystify the aforementioned debate and offer a comprehensive answer as to the proper order for Batch Normalization in the neural network training process. We demonstrate that for convolutional neural networks (CNNs) without skip connections, it is optimal to do ReLu activation before Batch Normalization as a result of higher gradient flow. In Residual Networks with skip connections, the order does not affect the performance or the gradient flow between the layers.

Sobolev loss is used when training a network to approximate the values and derivatives of a target function at a prescribed set of input points. Recent works have demonstrated its successful applications in various tasks such as distillation or synthetic gradient prediction. In this work we prove that an overparameterized two-layer relu neural network trained on the Sobolev loss with gradient flow from random initialization can fit any given function values and any given directional derivatives, under a separation condition on the input data.

The image labeling problem can be described as assigning to each pixel a single element from a finite set of predefined labels. Recently, a smooth geometric approach for inferring such label assignments was proposed by following the Riemannian gradient flow of a given objective function on the so-called assignment manifold. Due to the specific Riemannian structure, this results in a coupled replicator dynamic incorporating local spatial geometric averages of lifted data-dependent distances. However, in this framework an approximation of the flow is necessary in order to arrive at explicit formulas. We propose an alternative variational model, where lifting and averaging are decoupled in the objective function so as to stay closer to established approaches and at the same time preserve the main ingredients of the original approach: the overall smooth geometric setting and regularization through geometric local averages. As a consequence the resulting flow is explicitly given, without the need for any approximation. Furthermore, there exists an interesting connection to graphical models.

In this work we investigate a two-phase model for concentrated suspensions. We construct a PDE formulation using a gradient flow structure featuring dissipative coupling between fluid and solid phase as well as different driving forces. Our construction is based on the concept of flow maps that also allows it to account for flows in moving domains with free boundaries. The major difference compared to similar existing approaches is the incorporation of a non-smooth two-homogeneous term to the dissipation potential, which creates a normal pressure even for pure shear flows.

Let X1, …, Xn, n ≥ 1, be independent identically distributed (i.i.d.) \(\mathbb {R}^{d}\) valued random variables with a smooth density function f. We discuss how to use these X′s to estimate the gradient flow line of f connecting a point x0 to a local maxima point (mode) based on an empirical version of the gradient ascent algorithm using a kernel estimator based on a bandwidth h of the gradient ∇f of f. Such gradient flow lines have been proposed to cluster data. We shall establish a uniform in bandwidth h result for our estimator and describe its use in combination with plug in estimators for h.

This paper introduces a gradient flow in infinite dimension, whose long-time dynamics is expected to be an approximation of the quantization problem for probability densities, in the sense of Graf and Luschgy (Lecture Notes in Mathematics, vol 1730. Springer, Berlin, 2000). Quantization of probability distributions is a problem which one encounters in a great variety of contexts, such as signal processing, pattern or speech recognition, economics... The present work describes a dynamical approach of the optimal quantization problem in space dimensions one and two, involving (systems of) parabolic equations. This is an account of recent work in collaboration with Caglioti et al. (Math Models Methods Appl Sci 25:1845–1885, 2015 and arXiv:1607.01198 (math.AP), to appear in Ann. Inst. H. Poincare, Anal. Non Lin. https://doi.org/10.1016/j.anihpc.2017.12.003).

As we have mentioned in the introduction, a crucial step towards defining an \(A_\infty \)-algebra structure on the Morse cochain complex of a single Morse function is to consider perturbed Morse trajectories. More precisely, we want to dicuss curves which do not satisfy a negative gradient flow equation, but a perturbed negative gradient flow equation of the form \(\dot{\gamma }(s) + \nabla ^g f\circ \gamma (s) + Z(s,\gamma (s)) = 0 , \)where we pick up the notation from Chap. 1 and where Z is a suitable time-dependent vector field.

Recent progress in continuum dislocation dynamics (CDD) has been achieved through the construction of a local density approximation for the dislocation energy and the derivation of constitutive laws for the average dislocation velocity by means of variational methods from irreversible thermodynamics. Individual dislocations are driven by the Peach–Koehler-force which is likewise derived from a variational principle. This poses the question if we may expect that the averaged dislocation state expressed through the CDD density variables is driven by a variational gradient of the average energy, as is assumed in irreversible thermodynamics. In the current contribution we do not answer this questions, but rather present the mathematical framework within which the evolution of discrete dislocations is literally understood as a gradient descent. The suggested framework is that of de Rham currents and differential forms. We briefly sketch why we believe the results to be useful for formulating CDD theory as a gradient flow.

This paper describes a unified design methodology of distributed controllers of multi-agent systems for general tasks based on the authors’ recent work. First, a complete characterization is given to the distributed controllers via the gradient-flow approach. It is stressed that not edges but cliques (i.e., complete subgraphs) of network topologies are the crucial components of this characterization. Next, an optimal distributed controller is introduced, which achieves a given task as long as the network satisfies a certain condition. Even if the network does not satisfy the condition, the best approximate result to the task is achieved. Then, it is shown that the connection structure between the cliques plays an important role in achieving the task. While the conventional distributed controller design can handle specific tasks based on the edges of networks, the introduced approach provides us a systematic design methodology applicable to general tasks by using cliques.

In the remainder of of this book, we will discuss perturbed Morse ribbon trees, which can be interpreted as continuous maps from a tree to the manifold M which edgewise fulfill perturbed negative gradient flow equations. In this chapter, we will make this notion precise in terms of the constructions of Chaps. 2 and 3. Moreover, we will apply the nonlocal transversality result Theorem 3.10 to equip moduli spaces of perturbed Morse ribbon trees with the structures of finite-dimensional manifolds of class \(C^{n+1}\).

The current chapter discusses an application arising in the process of linear friction welding applied in metallurgy. In the first place a one-dimensional non-local model defined in the half-line is constructed in order to describe the evolution of the temperature within the welding region. In this study we mainly consider two cases: the soft-material which is modeled by an exponential nonlinearity and the hard-material case when a power-law nonlinearity is regarded. In the former case the non-local problem has variational structure, and so can be treated as a gradient flow, which is used to derive appropriate a priori estimates for the solution. Thus parabolic regularity theory can be used to prove global-in-time existence and finally prove the convergence of its solution towards the unique steady state. On the other hand, the power-law case lacks such a variational structure and thus we have to appeal to a numerical scheme of Crank–Nicolson type in order to presume the long-tume behavior in this case as well as to confirm the analytical results derived in the exponential case.

Recently, a smooth geometric approach to the image labeling problem was proposed [1] by following the Riemannian gradient flow of a given objective function on the so-called assignment manifold. The approach evaluates user-defined data term and additionally performs Riemannian averaging of the assignment vectors for spatial regularization. In this paper, we consider more elaborate graphical models, given by both data and pairwise regularization terms, and we show how they can be evaluated using the geometric approach. This leads to a novel inference algorithm on the assignment manifold, driven by local Wasserstein flows that are generated by pairwise model parameters. The algorithm is massively edge-parallel and converges to an integral labeling solution.

We propose a two-point flux approximation Finite Volume scheme for a model of incompressible and immiscible two-phase flow of Cahn–Hilliard type with degenerate mobility. This model was derived from a variational principle and can be interpreted as the Wasserstein gradient flow of the free energy. The fundamental properties of the continuous model, namely the positivity of the concentrations, the decay of the free energy, and the boundedness of the Boltzmann entropy, are preserved by the numerical scheme. Numerical simulations are provided to illustrate the behavior of the model and of the numerical scheme.

The image labeling problem can be described as assigning to each pixel a single element from a finite set of predefined labels. Recently, a smooth geometric approach was proposed [2] by following the Riemannian gradient flow of a given objective function on the so-called assignment manifold. In this paper, we adopt an approach from the literature on uncoupled replicator dynamics and extend it to the geometric labeling flow, that couples the dynamics through Riemannian averaging over spatial neighborhoods. As a result, the gradient flow on the assignment manifold transforms to a flow on a vector space of matrices, such that parallel numerical update schemes can be derived by established numerical integration. A quantitative comparison of various schemes reveals a superior performance of the adaptive scheme originally proposed, regarding both the number of iterations and labeling accuracy.

We discuss possible extensions of the recently established theory of evolutionary \(\varGamma \)-convergence for gradient systems to nonlinear dynamical systems obtained by perturbation of a gradient systems. Thus, it is possible to derive effective equations for pattern forming systems with multiple scales. Our applications include homogenization of reaction-diffusion systems, the justification of amplitude equations for Turing instabilities, and the limit from pure diffusion to reaction-diffusion. This is achieved by generalizing the \(\varGamma \)-limit approaches based on the energy-dissipation principle or the evolutionary variational estimate.

Image registration aims at establishing pointwise correspondences between given images. However, in many practical applications, no correspondences can be established in certain parts of the images. A typical example is the tumor resection area in pre- and post-operative medical images. In this paper, we introduce a novel variational framework that combines registration with an automatic detection of non-correspondence regions. The formulation of the proposed approach is simple but efficient, and compatible with a large class of image registration similarity measures and regularizers. The resulting minimization problem is solved numerically with a non-alternating gradient flow scheme. Furthermore, the method is validated on synthetic data as well as axial slices of pre-, post- and intra-operative MR T1 head scans.

In this paper, we present an algorithm to simplify low-contributing atoms and generate quality T-meshes for multi-resolution biomolecular surfaces. The structure of biomolecules is first simplified using an error-bounded atom elimination method. An extended cross field-based parameterization method is then developed to adapt the parametric line spacings to different surface resolutions. Moreover, an anisotropy defined from an input scalar field can also be achieved. From the parameterization results, we extract adaptive and anisotropic T-meshes for the further T-spline surface construction. Finally, a gradient flow-based method is developed to improve the T-mesh quality, with the anisotropy preserved in the quadrilateral elements. The effectiveness of the presented algorithm has been verified using several large biomolecular complexes.

The functionalized Cahn-Hilliard (FCH) free energy models interfacial energy in amphiphilic phase-separated mixtures. Its minimizers and quasi-minimizers encompass rich classes of network morphologies with detailed inner layers incorporating bilayers, pore, pearled pore, and micelle type structures. We present an overview of the stability of the network morphologies as well as the competitive evolution of bilayer and pore morphologies under a gradient flow in three space-dimensions.