Search

We consider the problem of estimating the $L_{1}$ distance between two discrete probability measures $P$ and $Q$ from empirical data in a nonasymptotic and large alphabet setting. When $Q$ is known and one obtains $n$ samples from $P$ , we show that for every $Q$ , the minimax rate-optimal estimator with $n$ samples achieves performance comparable to that of the maximum likelihood estimator with $n\ln n$ samples. When both $P$ and $Q$ are unknown, we construct minimax rate-optimal estimators, whose worst case performance is essentially that of the known $Q$ case with $Q$ being uniform, implying that $Q$ being uniform is essentially the most difficult case. The effective sample size enlargement phenomenon, identified by Jiao et al., holds both in the known $Q$ case for every $Q$ and the $Q$ unknown case. However, the construction of optimal estimators for $\|P-Q\|_{1}$ requires new techniques and insights beyond the approximation-based method of functional estimation by Jiao et al. [ABSTRACT FROM AUTHOR]

Suppose that we are given a time series where consecutive samples are believed to come from a probabilistic source, that the source changes from time to time and that the total number of sources is fixed. Our objective is to estimate the distributions of the sources. A standard approach to this problem is to model the data as a hidden Markov model (HMM). However, since the data often lacks the Markov or the stationarity properties of an HMM, one can ask whether this approach is still suitable or perhaps another approach is required. In this paper, we show that a maximum likelihood HMM estimator can be used to approximate the source distributions in a much larger class of models than HMMs. Specifically, we propose a natural and fairly general non-stationary model of the data, where the only restriction is that the sources do not change too often. Our main result shows that for this model, a maximum-likelihood HMM estimator produces the correct second moment of the data, and the results can be extended to higher moments. [ABSTRACT FROM AUTHOR]

This paper examines a general class of noisy matrix completion tasks, where the goal is to estimate a matrix from observations obtained at a subset of its entries, each of which is subject to random noise or corruption. Our specific focus is on settings where the matrix to be estimated is well-approximated by a product of two (a priori unknown) matrices, one of which is sparse. Such structural models—referred to here as sparse factor models—have been widely used, for example, in subspace clustering applications, as well as in contemporary sparse modeling and dictionary learning tasks. Our main theoretical contributions are estimation error bounds for sparsity-regularized maximum likelihood estimators for the problems of this form, which are applicable to a number of different observation noise or corruption models. Several specific implications are examined, including scenarios where observations are corrupted by additive Gaussian noise or additive heavier-tailed (Laplace) noise, Poisson-distributed observations, and highly quantized (e.g., 1 b) observations. We also propose a simple algorithmic approach based on the alternating direction method of multipliers for these tasks, and provide experimental evidence to support our error analyses. [ABSTRACT FROM AUTHOR]

This paper considers the asymptotic properties of the recursive maximum-likelihood estimator for hidden Markov models. The paper is focused on the analytic properties of the asymptotic log-likelihood and on the point-convergence and convergence rate of the recursive maximum-likelihood estimator. Using the principle of analytic continuation, the analyticity of the asymptotic log-likelihood is shown for analytically parameterized hidden Markov models. Relying on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point convergence of the recursive maximum-likelihood algorithm is demonstrated, and relatively tight bounds on the convergence rate are derived. As opposed to the existing result on the asymptotic behavior of maximum-likelihood estimation in hidden Markov models, the results of this paper are obtained without assuming that the log-likelihood function has an isolated maximum at which the Hessian is strictly negative definite. [ABSTRACT FROM AUTHOR]

This paper studies the partial estimation of Gaussian graphical models from high-dimensional empirical observations. We derive a convex formulation for this problem using \ell1-regularized maximum-likelihood estimation, which can be solved via a smoothing approximation algorithm. Statistical estimation performance can be established for our method. The proposed approach has competitive empirical performance compared with existing methods, as demonstrated by various experiments on synthetic and real data sets. [ABSTRACT FROM AUTHOR]

We study online learning under logarithmic loss with regular parametric models. In this setting, each strategy corresponds to a joint distribution on sequences. The minimax optimal strategy is the normalized maximum likelihood (NML) strategy. We show that the sequential NML (SNML) strategy predicts minimax optimally (i.e., as NML) if and only if the joint distribution on sequences defined by SNML is exchangeable. This property also characterizes the optimality of a Bayesian prediction strategy. In that case, the optimal prior distribution is Jeffreys prior for a broad class of parametric models for which the maximum likelihood estimator is asymptotically normal. The optimal prediction strategy, NML, depends on the number $n$ of rounds of the game, in general. However, when a Bayesian strategy is optimal, NML becomes independent of $n$ . Our proof uses this to exploit the asymptotics of NML. The asymptotic normality of the maximum likelihood estimator is responsible for the necessity of Jeffreys prior. [ABSTRACT FROM PUBLISHER]

This paper presents the theoretical foundation for time-domain, phase-based estimation of the frequency and phase of a single sinusoid in additive white Gaussian noise (AWGN), analogous to the theoretical foundation provided by Rife and Boorstyn for frequency-domain, Fourier-transform-based estimation. It is shown from the maximum a posteriori probability (MAP) and the maximum likelihood (ML) estimation principles that with the additive observation phase noise (AOPN), due to the AWGN, being described by its a posteriori distribution conditioned on the received signal magnitude, the received signal phase is a sufficient statistic for estimating the single-sinusoid angle parameters. Using a geometric approach, the exact statistical model for the AOPN is derived, where the a posteriori probability density function (pdf) and the corresponding a priori pdf are given by explicit, closed-form expressions that are valid for arbitrary signal-to-noise ratios (SNRs). The a posteriori pdf is Tikhonov, and is of particular interest as it establishes the AOPN model for phase-based frequency/phase MAP/ML estimation in the time domain. It is further illustrated that the results derived can yield various AOPN models as special cases, and the underlying physical insights and interconnections that exist among these models are revealed. It is shown that the model derived by Tretter is an ultimate specialization in the high SNR limit of the AOPN models developed here. For high SNR, the a posteriori Tikhonov pdf can be accurately approximated by a Gaussian distribution, which leads to the best linearized AOPN model. The applications of these AOPN models to the design of linear estimators, including the linear minimum mean square error (LMMSE) estimator, the linear minimum variance estimator, and the LMMSE implementation of the weighted phase averager are presented, and their estimation performances are compared through computer simulations, with the Cramer–Rao lower bound (CRLB) and the Bayesian CRLB as the benchmark. To facilitate estimator design, the a priori statistical models of the frequency and phase are proposed from the information-theoretic perspective, and an improved phase unwrapping algorithm over that given by Fu and Kam is presented. It is shown that by incorporating all the information available in the AOPN, the estimation accuracy can be much improved. [ABSTRACT FROM PUBLISHER]

In this paper, a new structured approach for obtaining uniformly best non-Bayesian biased estimators, which attain minimum-mean-square-error performance at any point in the parameter space, is established. We show that if a uniformly best biased (UBB) estimator exists, then it is unique, and it can be directly obtained from any locally best biased (LBB) estimator. A necessary and sufficient condition for the existence of a UBB estimator is derived. It is shown that if there exists an optimal bias, such that this condition is satisfied, then it is unique, and its closed-form expression is obtained. The proposed approach is exemplified in two nonlinear estimation problems, where uniformly minimum-variance-unbiased estimators do not exist. In the considered examples, we show that the UBB estimators outperform the corresponding maximum-likelihood estimators in the MSE sense. [ABSTRACT FROM AUTHOR]

Strict Minimum Message Length (SMML) is an information-theoretic statistical inference method widely cited (but only with informal arguments) as providing estimations that are consistent for general estimation problems. It is, however, almost invariably intractable to compute, for which reason only approximations of it (known as MML algorithms) are ever used in practice. Using novel techniques that allow for the first time direct, non-approximated analysis of SMML solutions, we investigate the Neyman-Scott estimation problem, an oft-cited showcase for the consistency of MML, and show that even with a natural choice of prior neither SMML nor its popular approximations are consistent for it, thereby providing a counterexample to the general claim. This is the first known explicit construction of an SMML solution for a natural, high-dimensional problem. [ABSTRACT FROM AUTHOR]

The quickest change detection problem is considered in the context of monitoring large-scale independent normal distributed data streams with possible changes in some of the means. It is assumed that for each individual local data stream, either there are no local changes, or there is a big local change that is larger than a pre-specified lower bound. Two different types of scenarios are studied: one is the sparse post-change case when the unknown number of affected data streams is much smaller than the total number of data streams, and the other is when all local data streams are affected simultaneously although not necessarily identically. We propose a systematic approach to develop efficient global monitoring schemes for quickest change detection by combining hard thresholding with linear shrinkage estimators to estimating all post-change parameters simultaneously. Our theoretical analysis demonstrates that the shrinkage estimation can balance the tradeoff between the first-order and second-order terms of the asymptotic expression on the detection delays, and our numerical simulation studies illustrate the usefulness of shrinkage estimation and the challenge of Monte Carlo simulation of the average run length to false alarm in the context of online monitoring large-scale data streams. [ABSTRACT FROM AUTHOR]

We consider the problem of discrete distribution estimation under \ell 1 loss. We provide tight upper and lower bounds on the maximum risk of the empirical distribution (the maximum likelihood estimator), and the minimax risk in regimes where the support size $S$ may grow with the number of observations $n$ . We show that among distributions with bounded entropy $H$ , the asymptotic maximum risk for the empirical distribution is $2H/\ln n$ , while the asymptotic minimax risk is $H/\ln n$ . Moreover, we show that a hard-thresholding estimator oblivious to the unknown upper bound $H$ , is essentially minimax. However, if we constrain the estimates to lie in the simplex of probability distributions, then the asymptotic minimax risk is again $2H/\ln n$ . We draw connections between our work and the literature on density estimation, entropy estimation, total variation distance ( \ell 1 divergence) estimation, joint distribution estimation in stochastic processes, normal mean estimation, and adaptive estimation. [ABSTRACT FROM PUBLISHER]

We study the maximum Markov model order that can be estimated for an (individual) input sequence $x$ of length $n$ over a finite alphabet of size $\alpha $ , for popular estimators of tree models, where the estimated order is determined by the depth of a context tree estimate, and in the special case of plain Markov models, where the tree is constrained to be perfect (with all leaves at the same depth). First, we consider penalized maximum likelihood estimators where a context tree $\hat {T}$ is obtained by minimizing a cost of the form $-\log \hat {P}_{T}(x) + f(n)|S_{T}|$ , where $\hat {P}_{T}(x)$ is the ML of $x$ under a model with context tree $T$ , $S_{T}$ is the set of leaves of $T$ , and $f(n)$ is an increasing (penalization) function of $n$ (the popular BIC estimator is a special case with $f(n)=\frac {\alpha -1}{2}\log n$). For plain Markov models, a simple argument yields a known upper bound $\overline {k}(n) = O(\log n)$ on the maximum order that can be estimated for $x$ , and we show that, in fact, this simple bound is not far from tight. For general context trees, we derive an asymptotic upper bound, $n^{1/2 - o(1)}$ , on the estimated depth, and we exhibit explicit input sequences that asymptotically attain the bound up to a multiplicative constant factor. We show that a similar upper bound applies also to MDL estimators based on the KT probability assignment and, moreover, the same example sequences asymptotically approach the upper bound also in this case. [ABSTRACT FROM AUTHOR]

The use of information theory concepts for universal estimation of delay for classes of discrete channels is discussed. The problem is presented as one of hypothesis testing. Although the channel statistics are not known, for large enough signal duration, the exponent of the average error probability is equal to that associated with the optimal maximum-likelihood (ML) decision procedure which utilizes full knowledge of the channel parameters. Two categories of problems are discussed: the single-channel problem, where the random transmitted signal is known to the receiver, and the two-sensor problem, where the random signal is unknown. [ABSTRACT FROM PUBLISHER]