Search

This paper presents two methods for approximating a proper subset of the entries of a Hessian using only function evaluations. These approximations are obtained using the techniques called \emph{generalized simplex Hessian} and \emph{generalized centered simplex Hessian}. We show how to choose the matrices of directions involved in the computation of these two techniques depending on the entries of the Hessian of interest. We discuss the number of function evaluations required in each case and develop a general formula to approximate all order-$P$ partial derivatives. Since only function evaluations are required to compute the methods discussed in this paper, they are suitable for use in derivative-free optimization methods., Comment: arXiv admin note: text overlap with arXiv:2304.03222

Positive spanning sets span a given vector space by nonnegative linear combinations of their elements. These have attracted significant attention in recent years, owing to their extensive use in derivative-free optimization. In this setting, the quality of a positive spanning set is assessed through its cosine measure, a geometric quantity that expresses how well such a set covers the space of interest. In this paper, we investigate the construction of positive $k$-spanning sets with geometrical guarantees.Our results build on recently identified positive spanning sets, called orthogonally structured positive bases. We first describe how to identify such sets and compute their cosine measures efficiently. We then focus our study on positive $k$-spanning sets, for which we provide a complete description, as well as a new notion of cosine measure that accounts for the resilient nature of such sets. By combining our results, we are able to use orthogonally structured positive bases to create positive $k$-spanning sets with guarantees on the value of their cosine measures.

The properties of positive bases make them a useful tool in derivative-free optimization (DFO) and an interesting concept in mathematics. The notion of the \emph{cosine measure} helps to quantify the quality of a positive basis. It provides information on how well the vectors in the positive basis uniformly cover the space considered. The number of vectors in a positive basis is known to be between $n+1$ and $2n$ inclusively. When the number of vectors is strictly between $n+1$ and $2n$, we say that it is an intermediate positive basis. In this paper, the structure of intermediate positive bases with maximal cosine measure is investigated. The structure of an intermediate positive basis with maximal cosine measure over a certain subset of positive bases is provided. This type of positive bases has a simple structure that makes them easy to generate with a computer software.

Model-based methods are popular in derivative-free optimization (DFO). In most of them, a single model function is built to approximate the objective function. This is generally based on the assumption that the objective function is one blackbox. However, some real-life and theoretical problems show that the objective function may consist of several blackboxes. In those problems, the information provided by each blackbox may not be equal. In this situation, one could build multiple sub-models that are then combined to become a final model. In this paper, we analyze the relation between the accuracy of those sub-models and the model constructed through their operations. We develop a broad framework that can be used as a theoretical tool in model error analysis and future research in DFO algorithms design.

An explicit formula to approximate the diagonal entries of the Hessian is introduced. When the derivative-free technique called \emph{generalized centered simplex gradient} is used to approximate the gradient, then the formula can be computed for only one additional function evaluation. An error bound is introduced and provides information on the form of the sample set of points that should be used to approximate the diagonal of a Hessian. If the sample set of points is built in a specific manner, it is shown that the technique is $\mathcal{O}(\Delta_S^2)$ accurate approximation of the diagonal entries of the Hessian where $\Delta_S$ is the radius of the sample set.

This work investigates the asymptotic behaviour of the gradient approximation method called the generalized simplex gradient (GSG). This method has an error bound that at first glance seems to tend to infinity as the number of sample points increases, but with some careful construction, we show that this is not the case. For functions in finite dimensions, we present two new error bounds ad infinitum depending on the position of the reference point. The error bounds are not a function of the number of sample points and thus remain finite.

This work introduces the nested-set Hessian approximation, a second-order approximation method that can be used in any derivative-free optimization routine that requires such information. It is built on the foundation of the generalized simplex gradient and proved to have an error bound that is on the order of the maximal radius of the two sets used in its construction. We show that when the points used in the computation of the nested-set Hessian have a favourable structure, (n+1)(n+2)/2 function evaluations are sufficient to approximate the Hessian. However, the nested-set Hessian also allows for evaluation sets with more points without negating the error analysis. Two calculus-based approximation techniques of the Hessian are developed and some advantages of the same are demonstrated., Comment: 23 pages excluding references, 2 figures

Using the Moore--Penrose pseudoinverse, this work generalizes the gradient approximation technique called centred simplex gradient to allow sample sets containing any number of points. This approximation technique is called the \emph{generalized centred simplex gradient}. We develop error bounds and, under a full-rank condition, show that the error bounds have order $O(\Delta^2)$, where $\Delta$ is the radius of the sample set of points used. We establish calculus rules for generalized centred simplex gradients, introduce a calculus-based generalized centred simplex gradient and confirm that error bounds for this new approach are also order $O(\Delta^2)$. We provide several examples to illustrate the results and some benefits of these new methods., Comment: 25 pages, no figures

Originally developed in 1954, positive bases and positive spanning sets have been found to be a valuable concept in derivative-free optimization (DFO). The quality of a positive basis (or positive spanning set) can be quantified via the {\em cosine measure} and convergence properties of certain DFO algorithms are intimately linked to the value of this measure. However, it is unclear how to compute the cosine measure for a positive basis from the definition. In this paper, a deterministic algorithm to compute the cosine measure of any positive basis or finite positive spanning set is provided. The algorithm is proven to return the exact value of the cosine measure in finite time.