Search

This paper considers the problem of estimating a high-dimensional vector of parameters \theta \in \mathbb R^n from a noisy observation. The noise vector is independent identically distributed Gaussian with known variance. For a squared-error loss function, the James–Stein (JS) estimator is known to dominate the simple maximum-likelihood (ML) estimator when the dimension n exceeds two. The JS-estimator shrinks the observed vector toward the origin, and the risk reduction over the ML-estimator is greatest for {\theta } that lie close to the origin. JS-estimators can be generalized to shrink the data toward any target subspace. Such estimators also dominate the ML-estimator, but the risk reduction is significant only when {\theta } lies close to the subspace. This leads to the question: in the absence of prior information about \theta , how do we design estimators that give significant risk reduction over the ML-estimator for a wide range of \theta ? In this paper, we propose shrinkage estimators that attempt to infer the structure of \theta from the observed data in order to construct a good attracting subspace. In particular, the components of the observed vector are separated into clusters, and the elements in each cluster shrunk toward a common attractor. The number of clusters and the attractor for each cluster are determined from the observed vector. We provide concentration results for the squared-error loss and convergence results for the risk of the proposed estimators. The results show that the estimators give significant risk reduction over the ML-estimator for a wide range of \theta , particularly for large $n$ . Simulation results are provided to support the theoretical claims. [ABSTRACT FROM AUTHOR]

We consider the fundamental problem of matching a template to a signal. We do so by M-estimation, which encompasses procedures that are robust to gross errors (i.e., outliers). Using standard results from empirical process theory, we derive the convergence rate and the asymptotic distribution of the M-estimator under relatively mild assumptions. We also discuss the optimality of the estimator, both in finite samples in the minimax sense and in the large-sample limit in terms of local minimaxity and relative efficiency. Although most of the paper is dedicated to the study of the basic shift model in the context of a random design, we consider many extensions towards the end of the paper, including more flexible templates, fixed designs, the agnostic setting, and more. [ABSTRACT FROM AUTHOR]

This paper studies forward and reverse projections for the Rényi divergence of order $\alpha \in (0, \infty )$ on $\alpha $ -convex sets. The forward projection on such a set is motivated by some works of Tsallis et al. in statistical physics, and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremoës proved a Pythagorean inequality for Rényi divergences on $\alpha $ -convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of a forward projection is proved for probability measures on a general alphabet. For $\alpha \in (1, \infty )$ , the proof relies on a new Apollonius theorem for the Hellinger divergence, and for $\alpha \in (0,1)$ , the proof relies on the Banach–Alaoglu theorem from the functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific $\alpha $ -convex set, which is termed an $\alpha $ -linear family, generalizing a result by Csiszár to $\alpha \neq 1$ . The solution to this problem yields a parametric family of probability measures, which turns out to be an extension of the exponential family, and it is termed an $\alpha $ -exponential family. An orthogonality relationship between the $\alpha $ -exponential and $\alpha $ -linear families is established, and it is used to turn the reverse projection on an $\alpha $ -exponential family into a forward projection on an $\alpha $ -linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of $\alpha $ -linear families. [ABSTRACT FROM PUBLISHER]

This paper considers the asymptotic properties of the recursive maximum-likelihood estimator for hidden Markov models. The paper is focused on the analytic properties of the asymptotic log-likelihood and on the point-convergence and convergence rate of the recursive maximum-likelihood estimator. Using the principle of analytic continuation, the analyticity of the asymptotic log-likelihood is shown for analytically parameterized hidden Markov models. Relying on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point convergence of the recursive maximum-likelihood algorithm is demonstrated, and relatively tight bounds on the convergence rate are derived. As opposed to the existing result on the asymptotic behavior of maximum-likelihood estimation in hidden Markov models, the results of this paper are obtained without assuming that the log-likelihood function has an isolated maximum at which the Hessian is strictly negative definite. [ABSTRACT FROM AUTHOR]

In this paper, under mild assumptions, we derive a law of large numbers, a central limit theorem with an error estimate, an almost sure invariance principle, and a variant of the Chernoff bound in finite-state hidden Markov models. These limit theorems are of interest in certain areas of information theory and statistics. Particularly, we apply the limit theorems to derive the rate of convergence of the maximum likelihood estimator in finite-state hidden Markov models. [ABSTRACT FROM AUTHOR]

In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The error or risk is considered as a function of the prior probabilities. We show that the risk function is locally Lipschitz in the vicinity of the true prior probabilities, and the error of detectors based on estimated prior probabilities depends on the behavior of the risk function in this locality. In general, we show that the error of detectors based on the maximum likelihood estimate (MLE) of the prior probabilities converges to the Bayes error at a rate of n^-1/2, where n is the number of training data. If the behavior of the risk function is more favorable, then detectors based on the MLE have errors converging to the corresponding Bayes errors at optimal rates of the form n^-(1+\alpha)/2, where \alpha > 0 is a parameter governing the behavior of the risk function with a typical value \alpha = 1. The limit \alpha \rightarrow \infty corresponds to a situation where the risk function is flat near the true probabilities, and thus insensitive to small errors in the MLE; in this case, the error of the detector based on the MLE converges to the Bayes error exponentially fast with n. We show that the bounds are achievable no matter given labeled or unlabeled training data and are minimax-optimal in the labeled case. [ABSTRACT FROM AUTHOR]

We consider the problem of discrete distribution estimation under \ell 1 loss. We provide tight upper and lower bounds on the maximum risk of the empirical distribution (the maximum likelihood estimator), and the minimax risk in regimes where the support size $S$ may grow with the number of observations $n$ . We show that among distributions with bounded entropy $H$ , the asymptotic maximum risk for the empirical distribution is $2H/\ln n$ , while the asymptotic minimax risk is $H/\ln n$ . Moreover, we show that a hard-thresholding estimator oblivious to the unknown upper bound $H$ , is essentially minimax. However, if we constrain the estimates to lie in the simplex of probability distributions, then the asymptotic minimax risk is again $2H/\ln n$ . We draw connections between our work and the literature on density estimation, entropy estimation, total variation distance ( \ell 1 divergence) estimation, joint distribution estimation in stochastic processes, normal mean estimation, and adaptive estimation. [ABSTRACT FROM PUBLISHER]

We study the phase synchronization problem with noisy measurements $Y=z^{*}z^{* { \mathrm {\scriptscriptstyle H} }}+\sigma W\in \mathbb {C}^{n\times n}$ , where $z^{*}$ is an $n$ -dimensional complex unit-modulus vector and $W$ is a complex-valued Gaussian random matrix. It is assumed that each entry $Y_{jk}$ is observed with probability $p$. We prove that an SDP relaxation of the MLE achieves the error bound $(1+o(1))\frac {\sigma ^{2}}{2np}$ under a normalized squared $\ell _{2}$ loss. This result matches the minimax lower bound of the problem, and even the leading constant is sharp. The analysis of the SDP is based on an equivalent non-convex programming whose solution can be characterized as a fixed point of the generalized power iteration lifted to a higher dimensional space. This viewpoint unifies the proofs of the statistical optimality of three different methods: MLE, SDP, and generalized power method. The technique is also applied to the analysis of the SDP for $\mathbb {Z}_{2}$ synchronization, and we achieve the minimax optimal error $\exp \left ({-(1-o(1))\frac {np}{2\sigma ^{2}}}\right)$ with a sharp constant in the exponent. [ABSTRACT FROM AUTHOR]

Using stochastic gradient search and the optimal filter derivative, it is possible to perform recursive maximum likelihood estimation in a non-linear state-space model. As the optimal filter and its derivative are analytically intractable for such a model, they need to be approximated numerically. In Poyiadjis et al. (G. Poyiadjis, A. Doucet, and S. S. Singh, Biometrika, vol. 98, no. 1, pp. 65–80, 2011), a recursive maximum likelihood algorithm based on a particle approximation to the optimal filter derivative has been proposed and studied through numerical simulations. This algorithm and its asymptotic behavior are here analyzed theoretically. Under regularity conditions, we show that the algorithm accurately estimates maxima of the underlying log-likelihood rate when the number of particles is sufficiently large. We also provide qualitative upper bounds on the estimation error in terms of the number of particles. [ABSTRACT FROM AUTHOR]

Strict Minimum Message Length (SMML) is an information-theoretic statistical inference method widely cited (but only with informal arguments) as providing estimations that are consistent for general estimation problems. It is, however, almost invariably intractable to compute, for which reason only approximations of it (known as MML algorithms) are ever used in practice. Using novel techniques that allow for the first time direct, non-approximated analysis of SMML solutions, we investigate the Neyman-Scott estimation problem, an oft-cited showcase for the consistency of MML, and show that even with a natural choice of prior neither SMML nor its popular approximations are consistent for it, thereby providing a counterexample to the general claim. This is the first known explicit construction of an SMML solution for a natural, high-dimensional problem. [ABSTRACT FROM AUTHOR]

The problem of estimating the directed information rate between two discrete processes \Xn\ and \Yn\ via the plug-in (or maximum-likelihood) estimator is considered. When the joint process \(Xn,Yn)\ is a Markov chain of a given memory length, the plug-in estimator is shown to be asymptotically Gaussian and to converge at the optimal rate O(1/\sqrt n) under appropriate conditions; this is the first estimator that has been shown to achieve this rate. An important connection is drawn between the problem of estimating the directed information rate and that of performing a hypothesis test for the presence of causal influence between the two processes. Under fairly general conditions, the null hypothesis, which corresponds to the absence of causal influence, is equivalent to the requirement that the directed information rate be equal to zero. In that case, a finer result is established, showing that the plug-in converges at the faster rate $O(1/n)$ and that it is asymptotically \chi ^{2} -distributed. This is proved by showing that this estimator is equal to (a scalar multiple of) the classical likelihood ratio statistic for the above hypothesis test. Finally, it is noted that these results facilitate the design of an actual likelihood ratio test for the presence or absence of causal influence. [ABSTRACT FROM AUTHOR]