Search

Parameter estimation of a nonlinear model based on maximizing the likelihood using gradient-based numerical optimization methods can often fail due to premature termination of the optimization algorithm. One reason for such failure is that these numerical optimization methods cannot distinguish between the minimum, maximum, and a saddle point; hence, the parameters found by these optimization algorithms can possibly be in any of these three stationary points on the likelihood surface. We have found that for maximization of the likelihood for nonlinear mixed effects models used in pharmaceutical development, the optimization algorithm Broyden-Fletcher-Goldfarb-Shanno (BFGS) often terminates in saddle points, and we propose an algorithm, saddle-reset, to avoid the termination at saddle points, based on the second partial derivative test. In this algorithm, we use the approximated Hessian matrix at the point where BFGS terminates, perturb the point in the direction of the eigenvector associated with the lowest eigenvalue, and restart the BFGS algorithm. We have implemented this algorithm in industry standard software for nonlinear mixed effects modeling (NONMEM, version 7.4 and up) and showed that it can be used to avoid termination of parameter estimation at saddle points, as well as unveil practical parameter non-identifiability. We demonstrate this using four published pharmacometric models and two models specifically designed to be practically non-identifiable.

Efficiency maximization based on load optimization has been thoroughly investigated for the conventional single-input single-output (SISO) inductive power transfer (IPT) system. This paper further extends the problem for multiple-input multiple-output (MIMO) IPT system operating at a single frequency. For the MIMO IPT system, efficiency maximization requires a joint optimization of loads—at the receivers—and input currents—in the transmitters. The optimal complex values of input currents and loads are derived by applying the first-order necessary condition, followed by the second partial derivative test. The derived maximum efficiency expression enables performance evaluation for any coil configuration, under any coil arrangement and at any frequency. Effectiveness and feasibility of the derived formulas are evaluated by electromagnetic simulations. Our work generalizes a theory on efficiency maximization for SISO, single-input multiple-output, multiple-input single-output, and MIMO systems.

This article presents a new method for classification of critical points. A vector field is usually classified using only a Jacobian matrix of the approximated vector field. This work shows how an approximation using a second order derivative can be used for more detailed classification. An algorithm for calculation of the curvature of main axes is also presented.

For a standard tool in econometrics, matrix calculus, an approach is illustrated in this note that is unusual in that context, a coordinate-free approach. It can help to eliminate the persistent use of non-standard conventions. The Kronecker product and its use can be better understood. The complications and pitfalls of defining twice differentiability by partial derivatives are avoided. Its use is demonstrated, for example by giving a coordinate-free determination of the derivative of the determinant. A vector-space calculation of the derivative of the determinant function.All conventions of matrix calculus arise naturally in the vector-space approach.The vector-space approach clarifies the role of the Kronecker product in matrix calculus.The vector-space approach gives a quick access to the v -space in interior point methods.

A framework for calculating the shape Hessian for the domain optimization problem, with a partial differential equation as the constraint, is presented. First and second order approximations of the cost with respect to geometry perturbations are arranged in an efficient manner that allows the computation of the shape derivative and Hessian of the cost without the necessity to involve the shape derivative of the state variable. In doing so, the state and adjoint variables are only required to be Holder continuous with respect to geometry perturbations.

We prove a relationship between the bordered Hessian in an equality constrained extremum problem and the Hessian of the equivalent lower-dimension unconstrained problem. This relationship can be used to derive principal minor conditions for the former from the relatively simple and accessible conditions for the latter.

In generalized pattern search algorithms for linearly constrained optimization, certain limit points have been shown to satisfy a pseudo-second-order necessary condition, in which positive semidefiniteness of the Hessian is achieved only with respect to an orthonormal basis. Satisfaction of the full second-order condition can only be ensured if the finite set of positive spanning directions includes the eigenvectors of the Hessian at the limit point. Of course, this is generally not possible in practice, due to the unavailability of derivatives. The present work takes advantage of a chief feature of the more general parent class of generating set search algorithms – that of allowing the use of an infinite (non-dense) set of directions in the limit. If approximate second derivative information is gathered during the iteration process, then the normalized eigenvectors of the approximate Hessian can be used to construct the set of 2n orthonormal directions used by the algorithm. If the Hessian approximation ...

We give a detailed derivation of a generalization of the second derivative test of single-variable calculus which can classify critical points as local minima or local maxima (or neither), whenever the traditional second derivative test fails, by considering the values of higher-order derivatives evaluated at the critical points. The enhanced test is local, in the sense that it is only necessary to evaluate all relevant derivatives at the critical point itself, and it is reasonably robust. We illustrate an application of the generalized test on a trigonometric function where the second derivative test fails to classify some of the critical points.

In this paper, we investigate a symmetric rank-one (SR1) quasi-Newton (QN) formula in which the Hessian of the objective function has some special structure. Instead of approximating the whole Hessian via the SR1 formula, we consider an approach which only approximates part of the Hessian matrix that is not easily acquired. Although the SR1 update possesses desirable features, it is unstable in the sense that, it may not retain positive definiteness and may become undefined. Therefore, we describe some safeguards to overcome these difficulties. Since the structured SR1 method provides a more accurate Hessian approximation, therefore the proposed method reduces significantly the computational efforts needed in solving a problem. The results of a series of experiments on a typical set of standard unconstrained optimization problems are reported, which show that the structured SR1 method exhibits a clear improvement in numerical performance over some existing QN algorithms.

Using the saddle point method, we obtain from the generating function of the q-Catalan numbers and Cauchy’s integral formula asymptotic results in central and non-central regions.

In this paper, we investigate how the Gauss-Newton Hessian matrix affects the basin of convergence in Newton-type methods. Although the Newton algorithm is theoretically superior to the Gauss-Newton algorithm and the Levenberg-Marquardt (LM) method as far as their asymptotic convergence rate is concerned, the LM method is often preferred in nonlinear least squares problems in practice. This paper presents a theoretical analysis of the advantage of the Gauss-Newton Hessian matrix. It is proved that the Gauss-Newton approximation function is the only nonnegative convex quadratic approximation that retains a critical property of the original objective function: taking the minimal value of zero on an $(n-1)$-dimensional manifold (or affine subspace). Due to this property, the Gauss-Newton approximation does not change the zero-on-$(n-1)$-D “structure” of the original problem, explaining the reason why the Gauss-Newton Hessian matrix is preferred for nonlinear least squares problems, especially when the initial point is far from the solution.

This paper presents determination of equilibrium paths for Mises trusses with different ratio of height to span. Unsymmetrical deformation modes are considered and the structure is treated as a two DOF system. First, a few special equilibrium configurations are resolved from considerations of free body diagrams. Complete equilibrium paths are determined by solving numerically the governing non-linear equilibrium equations. The stability of possible equilibrium configurations is checked using the second partial derivative test for the total potential energy. The positive definiteness of the appropriate Hessian matrices is checked numerically using the Sylvester criterion.

We construct a uniform approximation for generalized Hessian matrix of an SC 1 function. Using the discrete gradient and the extended second order derivative, we define the discrete Hessian matrix. We construct a sequence of sets, where each set is composed of discrete Hessian matrices. We first show some new properties of SC 1 functions. Then, we prove that for SC 1 functions the sequence of the set of discrete Hessian matrices is uniformly convergent to the generalized Hessian matrix.

This article deals with the evaluation of the impact of some errors on the prediction when an inverse heat conduction problem (IHCP) is solved. This evaluation is performed through an analysis of the cost function Hessian to be minimized. More precisely, this article presents a comparison between the computations of a cost function Hessian and of the related Fisher matrix. The Fisher matrix is an approximation of the Hessian matrix. It needs only the solution of the sensitivity problems. On the other hand, the computation of the exact Hessian needs the solution of the Euler equation along with the second-order adjoint problems.

In this paper, we study the Dirichlet problem for a class of fully nonlinear degenerate elliptic equations which depend only on the eigenvalues of the Hessian matrix. We provide a new and simpler a...

We prove an estimate on the L2(Ω) -norm of the Hessian of a function u ∈ W2,q(Ω) , satisfying an oblique derivative type condition on the boundary, allowing the oblique axis to be tangential at a finite number of points of ∂Ω . Using this inequality, the solvability in Sobolev spaces W2,q(Ω) , with q close to 2 , follows for a class of nonlinear differential equations in the plane with quadratic growth. Mathematics subject classification (2000): 35J65, 35R05.

Optimizing a function with respect to a set of variables using the quasi-Newton–Raphson method implies updating the Hessian matrix at each iteration. The Broyden–Fletcher–Goldfarb–Shanno update formula is used for minimization and the Murtagh–Sargent–Powell update formula for optimization of first-order saddle points. Two new formulae are proposed to update the Hessian matrix. One of these formulae is derived using exponential weights and should be used to locate first-order saddle points. The second formula is a modification of the TS–Broyden–Fletcher–Goldfarb–Shanno update and could used for both minimum and first-order saddle point optimizations. These two update Hessian matrix formulae present a performance that is the same and in many cases better that the Broyden–Fletcher–Goldfarb–Shanno and Murtagh–Sargent–Powell formulae. © 2003 Wiley Periodicals, Inc. Int J Quantum Chem 94: 324–332, 2003

A way to update the Hessian matrix according to the Powell formula is given. With this formula one does not need to store the full Hessian matrix at any iteration. A method to find transition structures, which is a combination of the quasi‐Newton–Raphson augmented Hessian algorithm with the proposed Powell update scheme, is also given. The diagonalization of the augmented Hessian matrix is carried out by Lanczos‐like methods. In this way, during all the optimization process, one avoids to store full matrices.

In this note, a modified Second Derivative Test is introduced for the relative extrema of a single variable function. This improved test overcomes the difficulty of the second derivative vanishing at the critical point, while in contrast the traditional test fails for this case. A proof for this improved Second Derivative Test is presented, followed by an example.

We present new integral formulas for the Hessian determinant. We use them for new definitions of Hessian under minimal regularity assumptions. The Hessian becomes a continuous linear functional on a Sobolev space.

In this paper, we consider the Dirichlet problem for a new class of augmented Hessian equations. Under sharp assumptions that the matrix function in the augmented Hessian is regular and there exists a smooth subsolution, we establish global second order derivative estimates for the solutions to the Dirichlet problem in bounded domains. The results extend the corresponding results in the previous paper [11] from the Monge-Ampere type equations to the more general Hessian type equations., Comment: Some minor revision and corrections made to previous version

The subject of this paper is the systematic study of second order notions concerning differentiable functions with Lipschitz derivative. The results and notions are motivated by recent papers of Cominetti, Correa and Hiriart-Urruty. The first goal of this paper is the comparison of several known second order directional derivatives. The second goal is the introduction of a generalized Hessian which is a set of certain symmetric bilinear forms. The relation of this generalized Hessian to other existing second order derivatives is also described.

We study nonlinear oblique boundary value problems for nonuniformly elliptic Hessian equations in two dimensions. These are equations whose principal part is given by a suitable symmetric function of the eigenvalues of the Hessian matrix D 2 u of the solution u . An interesting feature of our second derivative estimates is the need for certain strong structural hypotheses on the boundary condition, which are not needed in the uniformly elliptic case. Restrictions of this type are natural in our context; we present examples showing that second derivative bounds may fail if we do not assume such conditions.

The problem considered is that of minimizing a nonlinear function subject to a set of equality constraints. Recent research has focused on methods that solve such problems by recurring an approximation to second derivative information projected onto the tangent space of the constraints. The motivation for these methods is that only the projected matrix enters into the optimality conditions for the nonlinear problem and that the reduced matrix can be recurred by means of updates that maintain hereditary positive definiteness. However, such methods can achieve at best local two-step Q-superlinear convergence. The algorithm presented here performs a two-piece update of a “one-sided projected Hessian matrix.” This method maintains one piece of the projected Hessian as a positive definite matrix and uses Broyden’s method for updating the other piece. It is shown that if at least one of the updates is performed at each iteration, the method is locally one-step Q-superlinearly convergent.

The phase-equilibrium stability criterion is formulated in terms of the eigenvalues of the Hessian matrix of the Gibbs potential. The usefulness of this formulation is demonstrated by an analysis of the relation between the zero-determinant lines of the Hessian matrix and the spinodal manifolds.

We study some realization problems related to the Hessian polynomials. In particular, we solve the Hessian curve realization problem for degrees zero, one, two, and three and the Hessian polynomial realization problem for degrees zero, one, and two.

The quadratic variation of Brownian motion is used to give a new definition of a generalized Hessian for nonsmooth functions, totally independently of any gradients. A calculus is developed for the generalized Hessian of functions whose gradient is Lipschitz. It is also shown that for convex functions the Hessian is nonempty.

The tangent stiffness matrix (i.e., the Hessian) for nonlinear structural models in computational solid mechanics is typically computed by linearization of the weak form of the equilibrium equations via the directional derivative formula. Depending on the specific mechanical model, away from equilibrium this procedure will in general yield a nonsymmetric tangent stiffness matrix. By contrast, if the directional (Gateaux) derivative is replaced by the covariant derivative (relative to a certain Riemannian metric) an intrinsic definition of the Hessian is obtained which is always symmetric away from equilibrium. It is shown that by endowing the rotation group with the standard bi-invariant metric, the resulting (symmetric) Hessian is simply the symmetrization of the usual expression obtained via the Gateaux derivative. In a finite element context, this property provides a rigorous justification for the common practice of symmetrizing the ‘seemingly nonsymmetric tangent’ away from equilibrium. This apparently ad hoc symmetrization procedure yields, in fact, the actual Hessian.

In this paper, a new generalized second-order directional derivative and a set-valued generalized Hessian are introudced for C1,1 functions in real Banach spaces. It is shown that this set-valued generalized Hessian is single-valued at a point if and only if the function is twice weakly Gateaux differentiable at the point and that the generalized second-order directional derivative is upper semi-continuous under a regularity condition. Various generalized calculus rules are also given for C1,1 functions. The generalized second-order directional derivative is applied to derive second-order necessary optirnality conditions for mathematical programming problems.

The Hessian for the quasi-one-dimensional Euler equations is derived. A pressure minimization problem and a pressure matching inverse problem are considered. The flow sensitivity, adjoint sensitivity, gradient and Hessian are calculated analytically using a direct approach that is specific to the model problems. For the pressure minimization problem we find that the Hessian exists and it contains elements with significantly larger values around the shock location. For the pressure matching inverse problem we find at least one case for which the gradient as well as the Hessian do not exist. In addition, two formulations for calculating the Hessian are proposed and implemented for the given problems. Both methods can be implemented in industrial applications such as large scale aerodynamic optimization.

The computation of statistical properties in nonlinear parameter estimation is generally carried out using the hessian matrix of the likelihood function according to approximated expressions of the type V ϑ ≌ H -1 , where Vϑ is the estimated variance-covariance matrix of the parameters, and H is the hessian matrix of the likelihood function L at a maximum. It is shown in this paper that this approximation, based on local expansion of H, can lead to large deviations from the exact values whenever the truncation of L after the second term of the expansion is not sufficient. Furthermore, a method for continuously monitoring the validity of this approximation is proposed, and an alternative procedure is examined for the computation of Vϑ, whenever the usual estimation based on the hessian matrix cannot be used.

In this study we use surface-fitting equations to generate the energy and its first derivative in terms of two torsional angles in the methanediol model, using GAUSSIAN 88 at the 4-31G level of approximation. The Hessian matrix is further used to locate the stationary points, and a Gaussian fit of the absolute values of the sum of the eigenvalues of the Hessian is used in order to generate a surface in which all the stationary points are identified.

This paper analyzes methods of the type proposed by Coleman and Conn for nonlinearly constrained optimization. It is shown that if the reduced Hessian approximation is sufficiently accurate, then the method generates a sequence of iterates that converges one-step superlinearly. This result applies to a quasi-Newton implementation. If the reduced exact Hessian is used, the method has an R-order equal to that of the secant method. A similar result for a modified version of successive quadratic programming is also proved. Finally, some parallels between convergence results for methods that approximate the reduced Hessian method, and multiplier methods that use the reduced Hessian inverse, are pointed out.

International audience; We study the positivity of the second shape derivative around an equilibrium for a 2-dimensional functional involving the perimeter of the shape and its the Dirichlet energy under volume constraint. We prove that, generally, convex equilibria lead to strictly positive second derivatives. We also exhibit some examples where strict positivity of the second order derivative holds at an equilibrium while existence of a minimum does not.

The cost of calculating nuclear hessians, either analytically or by finite difference methods, during the course of quantum chemical analyses can be prohibitive for systems containing hundreds of atoms. In many applications, though, only a few eigenvalues and eigenvectors, and not the full hessian, are required. For instance, the lowest one or two eigenvalues of the full hessian are sufficient to characterize a stationary point as a minimum or a transition state (TS), respectively. We describe here a method that can eliminate the need for hessian calculations for both the characterization of stationary points as well as searches for saddle points. A finite differences implementation of the Davidson method that uses only first derivatives of the energy to calculate the lowest eigenvalues and eigenvectors of the hessian is discussed. This method can be implemented in conjunction with geometry optimization methods such as partitioned-rational function optimization (P-RFO) to characterize stationary points on the potential energy surface. With equal ease, it can be combined with interpolation methods that determine TS guess structures, such as the freezing string method, to generate approximate hessian matrices in lieu of full hessians as input to P-RFO for TS optimization. This approach is shown to achieve significant cost savings relative to exact hessian calculation when applied to both stationary point characterization as well as TS optimization. The basic reason is that the present approach scales one power of system size lower since the rate of convergence is approximately independent of the size of the system. Therefore, the finite-difference Davidson method is a viable alternative to full hessian calculation for stationary point characterization and TS search particularly when analytical hessians are not available or require substantial computational effort. © 2014 AIP Publishing LLC.

When dimension[codomain] = 1, the Jacobian of a single function is just the gradient of the function, and the zeros of the gradient determine the critical points of the function; in this case, the hessian classifies the critical points of a function. When dimension[domain] = 2, the Jacobian determines the first fundamental form of a parametric surface; in this case, the hessian, in vector form, determines the second fundamental form of a surface. The two fundamental forms together determine the Gaussian and mean curvatures of a surface. The analysis of the critical points involves looking at the values of the principal minors of this matrix at the critical points, namely, the determinants of the square submatrices running down the main diagonal in the upper left-hand side of the hessian. This chapter defines the critical points of a function as the points where the gradient is zero. If the hessian is positive definite at a critical point, then the critical point is a local minimum, and if it is negative definite, then the critical point is a local maximum. If a critical point is neither positive nor negative definite, then it is considered to be a saddle point. A minimal surface is one whose mean curvature is zero. The boundary of a two-dimensional piece of a surface is a closed curve. A surface is minimal if it has the least area of any surface with the same boundary.

We present a multilevel algorithm for the solution of a source identification problem in which the forward problem is an elliptic partial differential equation on the 2D unit box. The Hessian corresponds to a Tikhonov-regularized first-kind Fredholm equation. Our method uses an approximate Hessian operator for which, first, the spectral decomposition is known, and second, there exists a fast algorithm that can perform the spectral transform. Based on this decomposition we propose a conjugate gradients solver which we precondition with a multilevel subspace projection scheme. The coarse-level preconditioner is an exact solve and the finer-levels preconditioner is one step of the scaled Richardson iteration. As a model problem, we consider the 2D-Neumann Poisson problem with variable coefficients and partial observations. The approximate Hessian for this case is the Hessian related to a problem with constant coefficients and full observations. We can use a fast cosine transform to compute the spectral transforms. We examine the effect of using Galerkin or level-discretized Hessian operators and we provide results from numerical experiments that indicate the effectiveness of the method for full and partial observations.

This paper is intended to give a characterization of the minimum point of a function in terms of the gradient of the function at some other point using some concepts from differential geometry. The function is assumed to have continuous partial derivatives up to and including order four. It is also assumed to have a positive-definite Hessian matrix onR n and a unique minimum point.

In this paper, we present a generalization of the Hessian matrix toC 1,1 functions, i.e., to functions whose gradient mapping is locally Lipschitz. This type of function arises quite naturally in nonlinear analysis and optimization. First the properties of the generalized Hessian matrix are investigated and then some calculus rules are given. In particular, a second-order Taylor expansion of aC 1,1 function is derived. This allows us to get second-order optimality conditions for nonlinearly constrained mathematical programming problems withC 1,1 data.

We present a general active set algorithm for the solution of a convex quadratic programming problem having a parametrized Hessian matrix. The parametric Hessian matrix is a positive semidefinite Hessian matrix plus a real parameter multiplying a symmetric matrix of rank one or two. The algorithm solves the problem for all parameter values in the open interval upon which the parametric Hessian is positive semidefinite. The algorithm is general in that any of several existing quadratic programming algorithms can be extended in a straightforward manner for the solution of the parametric Hessian problem.

For a constrained minimization problem, the restriction of a Hessian of Lagrangian to a tangent space of the feasible set can be used to detect whether a Karush–Kuhn–Tucker point is a local minimum, maximum or saddle point of the problem. It is shown in this paper that the restriction of the Hessian to a normal space with respect to the indefinite inner product induced by the Hessian can be used to characterize a Karush–Kuhn–Tucker point for the Wolfe dual. From this result and by an inertia theorem in [S.-P. Han and O. Fujiwara, An inertia theorem and its application to nonlinear programs, manuscript March 1984], we deduce that, under a regularity condition, the Hessian is positive semidefinite if and only if the considered Karush–Kuhn–Tucker point satisfies the second order necessary condition for a local minimum point of the primal problem and, at the same time, satisfies the second order necessary condition for a local maximum point of thee dual. Similar results on the positive definiteness of the Hes...

The derivative of a polynomial p ( z ) p(z) with respect to a point ς \varsigma is defined by the formula A ς p ( z ) = ( ς − z ) p ′ ( z ) + n p ( z ) {A_\varsigma }p(z) = (\varsigma - z)p’(z) + np(z) , where n n is the degree of the polynomial. Let p ( z ) p(z) have all its zeros in the unit disk and one zero at z = 1 z = 1 . We determine a minimal region that must contain at least one zero of A ς p ( z ) {A_\varsigma }p(z) .

The matrix of second derivatives of a function f(x), the Hessian H, has to be positive definite at a minimum of fIt is shown that Gauss elimination together with an appropriate pivoting prescription can be used (a) to check the definiteness of H and (b) to find directions of negative second derivation, if they exist. Line search along such directions improves the Hessian and accelerates the convergence of minimization.

A first derivative test is formulated for determining the nature of extrema of a function f(x) of n real variables, subject to the constraints gi(x) = 0, i = 1,2.....r, where r

In this paper the convergence of variablemetric methods is considered. The minimum of a quadratic function is attained with a number of iterations equal to the rank of the Hessian matrix of the function. Under certain conditions the sequence of variable-metric matrices tends to the pseudoinverse of the Hessian matrix.