Search

The constrained minimization (respectively maximization) of directed distances and of related generalized entropies is a fundamental task in information theory as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition. In our previous paper "A precise bare simulation approach to the minimization of some distances. I. Foundations", we obtained such kind of constrained optima by a new dimension-free precise bare (pure) simulation method, provided basically that (i) the underlying directed distance is of f-divergence type, and that (ii) this can be connected to a light-tailed probability distribution in a certain manner. In the present paper, we extend this approach such that constrained optimization problems of a very huge amount of directed distances and generalized entropies -- and beyond -- can be tackled by a newly developed dimension-free extended bare simulation method, for obtaining both optima as well as optimizers. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). For instance, we cover constrained optimizations of arbitrary f-divergences, Bregman distances, scaled Bregman distances and weighted Euclidean distances. The potential for wide-spread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences in various different research fields (which may also serve as an interdisciplinary interface)., Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Density-based directed distances -- particularly known as divergences -- between probability distributions are widely used in statistics as well as in the adjacent research fields of information theory, artificial intelligence and machine learning. Prominent examples are the Kullback-Leibler information distance (relative entropy) which e.g. is closely connected to the omnipresent maximum likelihood estimation method, and Pearson's chisquare-distance which e.g. is used for the celebrated chisquare goodness-of-fit test. Another line of statistical inference is built upon distribution-function-based divergences such as e.g. the prominent (weighted versions of) Cramer-von Mises test statistics respectively Anderson-Darling test statistics which are frequently applied for goodness-of-fit investigations; some more recent methods deal with (other kinds of) cumulative paired divergences and closely related concepts. In this paper, we provide a general framework which covers in particular both the above-mentioned density-based and distribution-function-based divergence approaches; the dissimilarity of quantiles respectively of other statistical functionals will be included as well. From this framework, we structurally extract numerous classical and also state-of-the-art (including new) procedures. Furthermore, we deduce new concepts of dependence between random variables, as alternatives to the celebrated mutual information. Some variational representations are discussed, too.

In information theory -- as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -- many flexibilizations of the omnipresent Kullback-Leibler information distance (relative entropy) and of the closely related Shannon entropy have become frequently used tools. To tackle corresponding constrained minimization (respectively maximization) problems by a newly developed dimension-free bare (pure) simulation method, is the main goal of this paper. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). As a side effect, we also derive an innovative way of constructing new useful distances/divergences. To illustrate the core of our approach, we present numerous solved cases. The potential for widespread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences and entropies in various different research fields (which may also serve as an interdisciplinary interface)., Comment: v3: considerably shortened and restructured version of v1/v2; 64 pages + 7 pages supplement. This work is accepted by the journal "IEEE Transactions on Information Theory", and is available in early-access form at https://ieeexplore.ieee.org/document/9925151

The authors transpose a discrete notion of indetermination coupling in the case of continuous probabilities. They show that this coupling, expressed on densities, cannot be captured by a specific copula which acts on cumulative distribution functions without a high dependence on the margins. Furthermore, they define a notion of average likelihood which extends the discrete notion of couple matchings and demonstrate it is minimal under indetermination. Eventually, they leverage this property to build up a statistical test to distinguish indetermination and estimate its efficiency using the Bahadur's slope.

This paper provides constructive procedures for the indeterminacy coupling between two marginal distributions, an alternative to independence coupling. It also introduces a drawing under indeterminacy into a mixture of three independent couplings. Leveraging on this decomposition it states that indeterminacy optimally reduces couple matchings, minimizing the expected number of equal couples drawn in a row. Besides it is seen that the Janson Vegelius coefficient is nothing but a deviation to indeterminacy and it is shown that it tends to 0 when the number of modalities increases.

This paper is an attempt to set a justification for making use of some dicrepancy indexes, starting from the classical Maximum Likelihood definition, and adapting the corresponding basic principle of inference to situations where minimization of those indexes between a model and some extension of the empirical measure of the data appears as its natural extension. This leads to the so called generalized bootstrap setting for which minimum divergence inference seems to replace Maximum Likelihood one. 1 Motivation and context Divergences between probability measures are widely used in Statistics and Data Science in order to perform inference under models of various kinds, paramet-ric or semi parametric, or even in non parametric settings. The corresponding methods extend the likelihood paradigm and insert inference in some minimum "distance" framing, which provides a convenient description for the properties of the resulting estimators and tests, under the model or under misspecifica-tion. Furthermore they pave the way to a large number of competitive methods , which allows for trade-off between efficiency and robustness, among others. Many families of such divergences have been proposed, some of them stemming from classical statistics (such as the Chi-square), while others have their origin in other fields such as Information theory. Some measures of discrepancy involve regularity of the corresponding probability measures while others seem to be restricted to measures on finite or countable spaces, at least when using them as inferential tools, henceforth in situations when the elements of a model have to be confronted with a dataset. The choice of a specific discrepancy measure in specific context is somehow arbitrary in many cases, although the resulting conclusion of the inference might differ accordingly, above all under misspecification; however the need for such approaches is clear when aiming at robustness.

This paper aims at comparing two coupling approaches as basic layers for building clustering criteria, suited for modularizing and clustering very large networks. We briefly use "optimal transport theory" as a starting point, and a way as well, to derive two canonical couplings: "statistical independence" and "logical indetermination". A symmetric list of properties is provided and notably the so called "Monge's properties", applied to contingency matrices, and justifying the $\otimes$ versus $\oplus$ notation. A study is proposed, highlighting "logical indetermination", because it is, by far, lesser known. Eventually we estimate the average difference between both couplings as the key explanation of their usually close results in network clustering., Comment: arXiv admin note: text overlap with arXiv:2012.14674

We propose a sequential design method aiming at the estimation of an extreme quantile based on a sample of dichotomic data corresponding to peaks over a given threshold. This study is motivated by an industrial challenge in material reliability and consists in estimating a failure quantile from trials whose outcomes are reduced to indicators of whether the specimen have failed at the tested stress levels. The solution proposed is a sequential design making use of a splitting approach, decomposing the target probability level into a product of probabilities of conditional events of higher order. The method consists in gradually targeting the tail of the distribution and sampling under truncated distributions. The model is GEV or Weibull, and sequential estimation of its parameters involves an improved maximum likelihood procedure for binary data, due to the large uncertainty associated with such a restricted information.

We consider semiparametric moment condition models invariant to transformation groups. The parameter of interest is estimated by minimum empirical divergence approach, introduced by Broniatowski and Keziou (2012). It is shown that the minimum empirical divergence estimates, including the empirical likelihood one, are equivariants. The minimum risk equivariant estimate is then identied to be any one of the minimum empirical divergence estimates minus its expectation conditionally to maximal invariant statistic of the considered group of transformations. An asymptotic approximation to the conditional expectation, is obtained, using the result of Jureckov{\'a} and Picek (2009)., Comment: arXiv admin note: text overlap with arXiv:1002.0730

We explore some properties of the conditional distribution of an i.i.d. sample under large exceedances of its sum. Thresholds for the asymptotic independance of the summands are observed, in contrast with the classical case when the conditioning event is in the range of a large deviation. This paper is an extension to [7]. Tools include a new Edgeworth expansion adapted to specific triangular arrays where the rows are generated by tilted distribution with diverging parameters, together with some Abelian type results., Comment: arXiv admin note: text overlap with arXiv:1206.6951, arXiv:1305.3482

This paper deals with the problem of preventive maintenance (PM) scheduling of pipelines subject to external corrosion defects. The preventive maintenance strategy involves an inspection step at some epoch, together with a repair schedule. This paper proposes to determine the repair schedule as well as an inspection time minimizing the maintenance cost. This problem is formulated as a binary integer non-linear programming model and we approach it under a decision support framework. We derive a polynomial-time algorithm that computes the optimum PM schedule and suggests different PM strategies in order to assist practitioners in making decision.

This paper presents a new algorithm which aims at the resolution of inverse problems of the form f(x) = 0, for x a vector of dimension d and f an arbitrary function with mild regularity condition. The set of solutions S may be infinite. This algorithm produces a good coverage of S, with a limited number of evaluations of the function f. It is therefore appropriate for complex problems where those evaluations are costly. Various examples are presented, with d varying from 2 to 10. Proofs of convergence and of coverage of S are presented.

Estimators derived from an EM algorithm are not robust since they are based on the maximization of the likelihood function. We propose a proximal-point algorithm based on the EM algorithm which aim to minimize a divergence criterion. Resulting estimators are generally robust against outliers and misspecification. An EM-type proximal-point algorithm is also introduced in order to produce robust estimators for mixture models. Convergence properties of the two algorithms are treated. We relax an identifiability condition imposed on the proximal term in the literature; a condition which is generally not fulfilled by mixture models. The convergence of the introduced algorithms is discussed on a two-component Weibull mixture and a two-component Gaussian mixture entailing a condition on the initialization of the EM algorithm in order for the later to converge. Simulations on mixture models using different statistical divergences are provided to confirm the validity of our work and the robustness of the resulting estimators against outliers in comparison to the EM algorithm., Comment: Article submitted to IEEE Transactions on Information Theory so that copy writes may be transfered if accepted. arXiv admin note: substantial text overlap with arXiv:1603.07117

Estimators derived from a divergence criterion such as $\varphi-$divergences are generally more robust than the maximum likelihood ones. We are interested in particular in the so-called MD$\varphi$DE, an estimator built using a dual representation of $\varphi$--divergences. We present in this paper an iterative proximal point algorithm which permits to calculate such estimator. This algorithm contains by its construction the well-known EM algorithm. Our work is based on the paper of \citep{Tseng} on the likelihood function. We provide several convergence properties of the sequence generated by the algorithm, and improve the existing results by relaxing the identifiability condition on the proximal term, a condition which is not verified for most mixture models and hard to be verified for non mixture ones. Since convergence analysis uses regularity conditions (continuity and differentiability) of the objective function, which has a supremal form, we find it useful to present some analytical approaches for studying such functions. Convergence of the EM algorithm is discussed here again in a Gaussian and Weibull mixtures in the spirit of our approach. Simulations are provided to confirm the validity of our work and the robustness of the resulting estimators against outliers., Comment: 19 pages. Article submitted to Journal Entropy, special issue : Diffierential Geometrical Theory of Statistics

This paper extends the empirical minimum divergence approach for models which satisfy linear constraints with respect to the probability measure of the underlying variable (moment constraints) to the case where such constraints pertain to its quantile measure (called here semi parametric quantile models). The case when these constraints describe shape conditions as handled by the L-moments is considered and both the description of these models as well as the resulting non classical minimum divergence procedures are presented. These models describe neighborhoods of classical models used mainly for their tail behavior, for example neighborhoods of Pareto or Weibull distributions, with which they may share the same first L-moments. A parallel is drawn with similar problems held in elasticity theory and in optimal transport problems. The properties of the resulting estimators are illustrated by simulated examples comparing Maximum Likelihood estimators on Pareto and Weibull models to the minimum Chi-square empirical divergence approach on semi parametric quantile models, and others., Comment: 35 pages

This paper considers the construction of optimal designs due to Hoel and Levine and Guest. It focuses on the relation between the theory of the uniform approximation of functions and the optimality of the designs. Some application to accelerated tests is also presented. The multivariate case is also handled in some special situations., Comment: 43 pages

This paper states asymptotic equivalents for the three first moments of the Eescher transform of a distribution on R with smooth density in the upper tail. As a by product if provides a tail approximation for its moment generating function, and shows that the Esscher transforms have a Gaussian behavior for large values of the parameter., Comment: To appear in M. Hallin, D. Mason, D. Pfeifer, and J. Steinebach Eds, Mathematical Statistics and Limit Theorems: Festschrift in Honor of Paul Deheuvels. Springer, 20 pages

Let $X_{1},..,X_{n}$ denote an i.i.d. sample with light tail distribution and $S_{1}^{n}$ denote the sum of its terms; let $a_{n}$ be a real sequence\ going to infinity with $n.$\ In a previous paper (\cite{BoniaCao}) it is proved that as $n\rightarrow\infty$, given $\left(S_{1}^{n}/n>a_{n}\right) $ all terms $X_{i_{\text{}}}$ concentrate around $a_{n}$ with probability going to 1. This paper explores the asymptotic distribution of $X_{1}$ under the conditioning events $\left(S_{1}^{n}/n=a_{n}\right) $ and $\left(S_{1}^{n}/n\geq a_{n}\right)$ . It is proved that under some regulatity property, the asymptotic conditional distribution of $X_{1}$ given $\left(S_{1}^{n}/n=a_{n}\right) $ can be approximated in variation norm by the tilted distribution at point $a_{n}$, extending therefore the classical LDP case developed in Diaconis and Freedman (1988) . Also under $\left(S_{1}^{n}/n\geq a_{n}\right) $ the dominating point property holds. It also considers the case when the $X_{i}$'s are $\mathbb{R}^{d}-$valued, $f$ is a real valued function defined on $\mathbb{R}^{d}$ and the conditioning event writes $\left(U_{1}^{n}/n=a_{n}\right) $ or $\left(U_{1}^{n}/n\geq a_{n}\right)$ with $U_{1}^{n}:=\left(f(X_{1})+..+f(X_{n})\right) /n$ and $f(X_{1})$ has a light tail distribution$.$ As a by-product some attention is paid to the estimation of high level sets of functions., Comment: arXiv admin note: substantial text overlap with arXiv:1206.6951, arXiv:1302.1337

This paper explores Maximum Likelihood in parametric models in the context of Sanov type Large Deviation Probabilities. MLE in parametric models under weighted sampling is shown to be associated with the minimization of a specific divergence criterion defined with respect to the distribution of the weights. Some properties of the resulting inferential procedure are presented; Bahadur efficiency of tests are also considered in this context.

This paper explores a conditional Gibbs theorem for a random walkinduced by i.i.d. (X_{1},..,X_{n}) conditioned on an extreme deviation of its sum (S_{1}^{n}=na_{n}) or (S_{1}^{n}>na_{n}) where a_{n}\rightarrow\infty. It is proved that when the summands have light tails with some additional regulatity property, then the asymptotic conditional distribution of X_{1} can be approximated in variation norm by the tilted distribution at point a_{n}, extending therefore the classical LDP case.

This paper explores the joint behaviour of the summands of a random walk when their mean value goes to infinity as its length increases. It is proved that all the summands must share the same value, which extends previous results in the context of large exceedances of finite sums of i.i.d. random variables. Some consequences are drawn pertaining to the local behaviour of a random walk conditioned on a large deviation constraint on its end value. It is shown that the sample paths exhibit local oblic segments with increasing size and slope as the length of the random walk increases.

This paper presents a new approach to conditional inference, based on the simulation of samples conditioned by a statistics of the data. Also an explicit expression for the approximation of the conditional likelihood of long runs of the sample given the observed statistics is provided. It is shown that when the conditioning statistics is sufficient for a given parameter, the approximating density is still invariant with respect to the parameter. A new Rao-Blackwellisation procedure is proposed and simulation shows that Lehmann Scheff\'{e} Theorem is valid for this approximation. Conditional inference for exponential families with nuisance parameter is also studied, leading to Monte carlo tests. Finally the estimation of the parameter of interest through conditional likelihood is considered. Comparison with the parametric bootstrap method is discussed.

This paper presents a sharp approximation of the density of long runs of a random walk conditioned on its end value or by an average of a function of its summands as their number tends to infinity. In the large deviation range of the conditioning event it extends the Gibbs conditional principle in the sense that it provides a description of the distribution of the random walk on long subsequences. An approximation of the density of the runs is also obtained when the conditioning event states that the end value of the random walk belongs to a thin or a thick set with a nonempty interior. The approximations hold either in probability under the conditional distribution of the random walk, or in total variation norm between measures. An application of the approximation scheme to the evaluation of rare event probabilities through importance sampling is provided. When the conditioning event is in the range of the central limit theorem, it provides a tool for statistical inference in the sense that it produces an effective way to implement the Rao-Blackwell theorem for the improvement of estimators; it also leads to conditional inference procedures in models with nuisance parameters. An algorithm for the simulation of such long runs is presented, together with an algorithm determining the maximal length for which the approximation is valid up to a prescribed accuracy., Comment: Published in at http://dx.doi.org/10.1214/13-AAP975 the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org). arXiv admin note: text overlap with arXiv:1010.3616

In this note we prove the dual representation formula of the divergence between two distributions in a parametric model. Resulting estimators for the divergence as for the parameter are derived. These estimators do not make use of any grouping nor smoothing. It is proved that all differentiable divergences induce the same estimator of the parameter on any regular exponential family, which is nothing else but the MLE., Comment: Submitted

The aim of this paper is to introduce new statistical criterions for estimation, suitable for inference in models with common continuous support. This proposal is in the direct line of a renewed interest for divergence based inference tools imbedding the most classical ones, such as maximum likelihood, Chi-square or Kullback Leibler. General pseudodistances with decomposable structure are considered, they allowing to define minimum pseudodistance estimators, without using nonparametric density estimators. A special class of pseudodistances indexed by {\alpha}>0, leading for {\alpha}\downarrow0 to the Kulback Leibler divergence, is presented in detail. Corresponding estimation criteria are developed and asymptotic properties are studied. The estimation method is then extended to regression models. Finally, some examples based on Monte Carlo simulations are discussed.

Improving Importance Sampling estimators for rare event probabilities requires sharp approximations of conditional densities. This is achieved for events E_{n}:=(f(X_{1})+...+f(X_{n}))\inA_{n} where the summands are i.i.d. and E_{n} is a large or moderate deviation event. The approximation of the conditional density of the real r.v's X_{i} 's, for 1\leqi\leqk_{n} with repect to E_{n} on long runs, when k_{n}/n\to1, is handled. The maximal value of k compatible with a given accuracy is discussed; algorithms and simulated results are presented.

We propose a new definition of the chi-square divergence between distributions. Based on convexity properties and duality, this version of the {\chi}^2 is well suited both for the classical applications of the {\chi}^2 for the analysis of contingency tables and for the statistical tests for parametric models, for which it has been advocated to be robust against inliers. We present two applications in testing. In the first one we deal with tests for finite and infinite numbers of linear constraints, while, in the second one, we apply {\chi}^2-methodology for parametric testing against contamination.

This paper deals with probabilistic upper bounds for the error in functional estimation defined on some interpolation and extrapolation designs, when the function to estimate is supposed to be analytic. The error pertaining to the estimate may depend on various factors: the frequency of observations on the knots, the position and number of the knots, and also on the error committed when approximating the function through its Taylor expansion. When the number of observations is fixed, then all these parameters are determined by the choice of the design and by the choice estimator of the unknown function. The scope of the paper is therefore to determine a rule for the minimal number of observation required to achieve an upper bound of the error on the estimate with a given maximal probability.

This paper presents a sharp approximation of the density of long runs of a random walk conditioned on its end value or by an average of a functions of its summands as their number tends to infinity. The conditioning event is of moderate or large deviation type. The result extends the Gibbs conditional principle in the sense that it provides a description of the distribution of the random walk on long subsequences. An algorithm for the simulation of such long runs is presented, together with an algorithm determining their maximal length for which the approximation is valid up to a prescribed accuracy.

We consider the minimization problem of $\phi$-divergences between a given probability measure $P$ and subsets $\Omega$ of the vector space $\mathcal{M}_\mathcal{F}$ of all signed finite measures which integrate a given class $\mathcal{F}$ of bounded or unbounded measurable functions. The vector space $\mathcal{M}_\mathcal{F}$ is endowed with the weak topology induced by the class $\mathcal{F}\cup \mathcal{B}_b$ where $\mathcal{B}_b$ is the class of all bounded measurable functions. We treat the problems of existence and characterization of the $\phi$-projections of $P$ on $\Omega$. We consider also the dual equality and the dual attainment problems when $\Omega$ is defined by linear constraints.

We introduce estimation and test procedures through divergence minimiza- tion for models satisfying linear constraints with unknown parameter. These procedures extend the empirical likelihood (EL) method and share common features with generalized empirical likelihood approach. We treat the problems of existence and characterization of the divergence projections of probability distributions on sets of signed finite measures. We give a precise characterization of duality, for the proposed class of estimates and test statistics, which is used to derive their limiting distributions (including the EL estimate and the EL ratio statistic) both under the null hypotheses and under alterna- tives or misspecification. An approximation to the power function is deduced as well as the sample size which ensures a desired power for a given alternative., Comment: 37 pages, 4 figures

The class of dual $\phi$-divergence estimators (introduced in Broniatowski and Keziou (2009) is explored with respect to robustness through the influence function approach. For scale and location models, this class is investigated in terms of robustness and asymptotic relative efficiency. Some hypothesis tests based on dual divergence criterions are proposed and their robustness properties are studied. The empirical performances of these estimators and tests are illustrated by Monte Carlo simulation for both noncontaminated and contaminated data.

This paper introduces a new class of Cox models for dependent bivariate data. The impact of the covariate on the dependence of the variables is captured through the modification of their copula. Various classes of well known copulas are stable under the model (archimedean type and extreme value copulas), meaning that the role of the covariate acts in a simple and explicit way on the copula in the class; specific parametric classes are considered.

This paper deals with four types of point estimators based on minimization of information-theoretic divergences between hypothetical and empirical distributions. These were introduced (i) by Liese & Vajda (2006) and independently Broniatowski & Keziou (2006), called here power superdivergence estimators, (ii) by Broniatowski & Keziou (2009), called here power subdivergence estimators, (iii) by Basu et al. (1998), called here power pseudodistance estimators, and (iv) by Vajda (2008) called here Renyi pseudodistance estimators. The paper studies and compares general properties of these estimators such as consistency and influence curves, and illustrates these properties by detailed analysis of the applications to the estimation of normal location and scale.

This paper introduces a new Importance Sampling scheme, called Adaptive Twisted Importance Sampling, which is adequate for the improved estimation of rare event probabilities in he range of moderate deviations pertaining to the empirical mean of real i.i.d. summands. It is based on a sharp approximation of the density of long runs extracted from a random walk conditioned on its end value.

We introduce estimation and test procedures through divergence optimization for discrete or continuous parametric models. This approach is based on a new dual representation for divergences. We treat point estimation and tests for simple and composite hypotheses, extending maximum likelihood technique. An other view at the maximum likelihood approach, for estimation and test, is given. We prove existence and consistency of the proposed estimates. The limit laws of the estimates and test statistics (including the generalized likelihood ratio one) are given both under the null and the alternative hypotheses, and approximation of the power functions is deduced. A new procedure of construction of confidence regions, when the parameter may be a boundary value of the parameter space, is proposed. Also, a solution to the irregularity problem of the generalized likelihood ratio test pertaining to the number of components in a mixture is given, and a new test is proposed, based on $\chi ^{2}$-divergence on signed finite measures and duality technique.

We introduce estimation and test procedures through divergence minimization for models satisfying linear constraints with unknown parameter. Several statistical examples and motivations are given. These procedures extend the empirical likelihood (EL) method and share common features with generalized empirical likelihood (GEL). We treat the problems of existence and characterization of the divergence projections of probability measures on sets of signed finite measures. Our approach allows for a study of the estimates under misspecification. The asymptotic behavior of the proposed estimates are studied using the dual representation of the divergences and the explicit forms of the divergence projections. We discuss the problem of the choice of the divergence under various respects. Also we handle efficiency and robustness properties of minimum divergence estimates. A simulation study shows that the Hellinger divergence enjoys good efficiency and robustness properties.