Search

In 1922 R. A. Fisher introduced the modern regression model, synthesizing the regression theory of Pearson and Yule and the least squares theory of Gauss. The innovation was based on Fisher’s realization that the distribution associated with the regression coefficient was unaffected by the distribution of X. Subsequently Fisher interpreted the fixed X assumption in terms of his notion of ancillarity. This paper considers these developments against the background of the development of statistical theory in the early twentieth century.

We consider multidimensional M-functional parameters defined by expectations of score functions associated with multivariate M-estimators and tests for hypotheses concerning multidimensional smooth functions of these parameters. We propose a test statistic suggested by the exponent in the saddlepoint approximation to the density of the function of the M-estimates. This statistic is analogous to the log likelihood ratio in the parametric case. We show that this statistic is approximately distributed as a chi-squared variate and obtain a Lugannani-Rice style adjustment giving a relative error of order $n^{-1}$. We propose an empirical exponential likelihood statistic and consider a test based on this statistic. Finally we present numerical results for three examples including one in robust regression.

For any generalized linear model, the Pearson goodness of fit statistic is the score test statistic for testing the current model against the saturated model. The relationship between the Pearson statistic and the residual deviance is therefore the relationship between the score test and the likelihood ratio test statistics, and this clarifies the role of the Pearson statistic in generalized linear models. The result is extended to cases in which there are multiple reponse observations for the same combination of explanatory variables.

A widely held notion of classical conditional theory is that statistical inference in the presence of ancillary statistics should be independent of the distribution of those ancillary statistcs. In this paper, ancillary paradoxes which contradict this notion are presented for two scenarios involving confidence estimation. These results are related to Brown’s ancillary paradox in point estimation. Moreover, the confidence coefficient, the usual constant coverage probability estimator, is shown to be inadmissible for confidence estimation in the multiple regression model with random predictor variables if the dimension of the slope parameters is greater than five. Some estimators better than the confidence coefficient are provided in this paper. These new estimators are constructed based on empirical Bayes estimators.

Under mild misspecifications of model assumptions, maximum likelihood estimates often remain consistent and asymptotically normal. Asymptotic normality will often hold for the signed root of the likelihood ratio statistic and the score statistic as well. However, standard estimates of asymptotic variance are usually inconsistent. This occurs when Bartlett's second identity fails. In the manner of McCullagh and Tibshirani, a variance correction may be used to adjust the profile likelihood so this identity obtains. The resulting likelihood yields the robust versions of the signed root, Wald and score statistic suggested by Kent and Royall. ¶ Assuming model correctness, asymptotic expansions for the first three cumulants of each robust statistic are derived. It is seen that bias and skewness are not severely affected by using a robust statistic. An invariant expression derived for the asymptotic relative efficiency of a robust method allows assessment in numerous examples considered. Even for moderately large sample sizes, losses in efficiency are significant, making the misuse of a robust variance estimate potentially costly. Computer algebra is used in many of the calculations reported in this paper.

Simple conditions on the observed information ensure asymptotic normality of the conditional distributions of the randomly normed score statistic and maximum likelihood estimator given a suitable asymptotically ancillary statistic. In particular, asymptotic normality holds conditional on any asymptotically ancillary statistic asymptotically equivalent to observed information. The results apply to inference from a general stochastic process and are of particular relevance in the case of nonergodic models.

Conditions for the existence of the density of a minimum contrast estimator in a parametric statistical family are given together with a formula for this density. The formula is exact if multiple local minima cannot occur; otherwise the formula is an exact expression for the point process of local minima of the contrast function. Although it is not in general feasible to compute the expression for the density, the formula can be used as a basis for further expansion of the large deviation type. When the estimate is sufficient, either in the original model or after conditioning on an approximate or exact ancillary, the formula simplifies drastically. In particular, it is shown how Barndorff-Nielsen's formula for the density of the maximum likelihood estimator given an ancillary statistic is derived from the formula given here. In this way the nature of Barndorff-Nielsen's formula as an asymptotic approximation and its appearance as an exact formula for certain cases are demonstrated.

A case of nonregular estimation arises in attempting to estimate a single unknown parameter, $\theta$, in the probability distribution of a single chance variable in which one or both of the extremities of the range of the distribution are functions of the unknown parameter. The case treated in this paper is the one in which a probability density of exponential type exists. When one extremity alone of the range depends non-trivially upon $\theta$, a necessary and sufficient condition is given in order that a single order statistic be a sufficient statistic for $\theta$. In this case conditions are given for the existence of a unique unbiased estimate of $\theta$ possessing minimum variance uniformly in $\theta$. In the case in which both extremities of the range depend upon $\theta$, a necessary and sufficient condition is given that the smallest and largest order statistics constitute a set of sufficient statistics for $\theta$. In this case Pitman [1] has shown that a single sufficient statistic exists if one extremity of the range is a monotone decreasing function of the other extremity. It is shown that under the above condition a unique unbiased estimate exists possessing minimum variance. Moreover a surmise of Pitman is proved that only under this condition does a single sufficient statistic exist. When a single sufficient statistic does not exist, an unbiased estimate of a known function of $\theta$ is obtained which has less variance than any analytic function of the set of sufficient statistics for $\theta$.

The problem of finding a Chebyshev type inequality for random variables with unknown or non-existent variance was considered by Z. W. Birnbaum (1970). In this present paper, a statistic, $T$, similar to, but simpler than Birnbaum's, is considered. The statistic is independent of location and scale parameters for families of bell-shaped distributions and so may be considered to be a competitor to Student's $t$. An inequality establishing an upper bound for $P(|T| > \lambda)$ is proved. This bound is considerably smaller than the corresponding bound found by Birnbaum. Finally, an improvement of the latter is offered.

1. Introduction. In 1960, Bahadur [3] proposed two measures of the asymptotic performance of tests: one approximate measure, based on the limiting distribution of the test statistic; and one exact, based on the limiting form of the probability of a large deviation of the statistic from its asymptotic mean. Although knowledge of the exact measure is more desirable, it is often difficult to compute, whereas the approximate measure is usually trivially available. It is to be hoped that the approximate measure will nearly always be a good approximation to the exact measure, and there is considerable evidence in support of this conjecture, but counterexamples do exist (see e.g. [1], p. 20). This paper explores the question for the Kolmogorov-Smirnov (K-S) and Kuiper statistics-two closely related measures of goodness-of-fit based on deviations of the sample distribution functions from the null case-in one- and two-sample situations. In the case of the weighted one-sample K-S statistic and the two-sample Kuiper statistic, it is possible to obtain exact measures although the corresponding approximate measures are not available. The relative efficiency (in a sense defined in Section 2) of the weighted one-sample K-S statistic to the unweighted K-S statistic is computed in a number of cases, and the relative efficiency of the Kuiper statistic to the unweighted K-S statistic is also examined, from which it appears that the Kuiper statistic is always at least as good (and often much better) than the K-S statistic. In the cause of brevity, a great deal of the theory and all the computational detail (generally very tedious) has been omitted. Much of it is available in [1]. However, a short resum6 of the theory of the Bahadur measures of performance and their properties is given in Section 2 for the sake of completeness, although the details here are also omitted.

Joint intervals of bounded confidence are suggested for the medians of a bivariate population with continuous marginal distributions. The two intervals are of the classic type based on sample order statistics.

For the two-sample problem, Wilcoxon [21], Fisher and Yates [6], Terry [19], Hoeffding [10], Hodges and Lehmann [8], Savage [17], Chernoff and Savage [3], Lehmann [13], Capon [2] and others have considered rank-sum statistics equivalent to $S_N(H) = m^{-1} \sum E(Z(R(X_i)) \mid H) - n^{-1} \sum E(Z(R(Y_i)) \mid H)$, where $E(Z(j) \mid H)$ is the expectation of the $j$th order statistic of a sample of size $N = m + n$ from a population with cpf (cumulative probability function) $H$ and $R(X_i) \lbrack R(Y_j)\rbrack$ is the rank of $X_i\lbrack Y_j\rbrack$ in the combined sample of $X$'s and $Y$'s. In order to perform tests based on these statistics one needs special tables of expected values as well as tables of the hypothesis distribution. Further, in general, exact desired significance levels can only be achieved through randomization. The object of this note is to introduce rank-sum statistics in which one randomizes initially and circumvents the necessity of two special tables. These new randomized statistics, which are formed by deleting the expectation signs "$E$" in $S_N(H)$, generally satisfy the same asymptotic goodness criteria as their nonrandomized counterparts. Moreover, they have the added advantage that for appropriate choices of the parameters they have null hypothesis distributions which are continuous, known and tabulated (e.g., normal, $\chi^2$, etc.) In particular, one of the new two-sample statistics has an exact normal distribution and is asymptotically uniformly more efficient than the $t$-test for translation alternatives. This idea is extended to obtain randomized rank-sum statistics for the independence, randomness, $k$-sample and two-factor problems analogous to the statistics of Friedman [7], Puri [15], Stuart [18] and others. As in the references listed above, prime interest will be in those cases for which $H$ is normal, uniform or exponential. However, the methodology is equally applicable to other continuous cpf's $H$.

0. Summary. In 1949, M. Kac defined a statistic which appears useful for statistical problems arising in insurance, biology, and telephone engineering [2]. In those fields, the natural observation peiiod is a fixed time period during which a random number of observations would be obtained. The distribution of the number of observations is given by a Poisson distribution. Distributions of this statistic can be used to determine upper and lower confidence contours for an unkrnown distribution, or in testing a distribution hypothesis. The purpose of this note is to extend the authors' earlier results, [1], to the two-sided Kac statistic.

Generalizing earlier observations by Fisher and Hotelling, Tukey [1] showed that if a family of distributions admits a set of sufficient statistics, then the family obtained by truncation to a fixed set, or by a fixed selection, also admits the same set of sufficient statistics (this wording is Tukey's; we give a precise mathematical statement later). Tukey's proof assumed the relevant family of probability measures to be dominated by a fixed measure function and made use of the factorization theorem concerning sufficient statistics in this case. In the present short note we shall first re-prove Tukey's result without assuming domination (and, hence, without appealing to the factorization theorem). Then we shall show that, under general conditions, if a sufficient statistic has one or more of the properties of completeness, bounded completeness, or minimality, before truncation, then it preserves such after truncation. The treatment is on the lines of the abstract discussion of sufficient statistics given by Halmos and Savage [2]. We shall assume familiarity with the results given in this latter paper. For definitions of completeness, bounded completeness, and minimality, and for a discussion of the significance of these concepts we refer to Lehmann and Scheffe [3].

The concept of "functional sufficiency" or "f-sufficiency" for a class of density functions is introduced and conditions given under which it corresponds to sufficiency as defined by Halmos and Savage [2]. A minimal f-sufficient statistic is defined and shown to exist, and its construction is given. The minimal f-sufficient statistic for a class of densities for which the region of positive density varies with the parameter is shown to be equivalent to the combination of a "statistic of selection" and the minimal f-sufficient statistic for a class of densities for which the region of positive density is fixed. The construction of sufficient statistics in this latter case subject to certain conditions of regularity has been treated by Koopman [1]. If the parameter is a parameter of selection from a fixed distribution, then the statistic of selection is the minimal f-sufficient statistic. If in addition the regions of positive density are monotone and indexed monotonely by a real parameter, then the statistic of selection is sufficient according to the Halmos and Savage definition. Three examples are given to illustrate the results.

Cramer ([6], p. 474 ff.), Darmois [8], Frechet [10], and Rao [14] derived independently a lower bound for the mean square error of an estimate $t$ of a parameter which appears in a frequency function of a specified form. This expression, alternately termed the Cramer-Rao inequality or the information limit, is \begin{equation*}\tag{1.1}]E(t - \alpha)^2 \geqq \lbrack E(t) - \alpha\rbrack^2 + \frac{\big\lbrack\frac{\partial E(t)}{\partial\alpha}\big\rbrack^2}{E\big(\frac{\partial\ln\phi} {\partial\alpha}\big)^2}\end{equation*} where $\phi$ is the likelihood of the sample. The expression $E(\partial \ln \phi/\partial\alpha)^2$ is called the information on $\alpha$ and is sometimes denoted by $I(\alpha)$. Under rather general conditions it can be shown equal to $E( - \partial^2 \ln \phi/\partial\alpha^2)$. The equality in (1.1) is reached if and only if, \begin{equation*}\tag{1.2}\phi = \phi_1e^{tV(\alpha) + W(\alpha)}\end{equation*} where $t$ and $\phi_1$ are functions of the observations alone and $V(\alpha)$ and $W(\alpha)$ are functions of $\alpha$ alone. By the results of Pitman [13] and Koopman [12], the form of (1.2) implies that $t$ must be a sufficient statistic. The fact that this form of the likelihood yields a minimum variance estimate was first pointed out by Aitken and Silverstone [1]. If we have $n$ observations which are independently and identically distributed, the frequency function of the underlying population must be of the so-called Pitman-Koopman form, \begin{equation*}\tag{1.3}f(x; \alpha) = (\alpha)h(x)e^{P(\alpha)g(x)}\end{equation*} and $t$ must be a function of $\sum^n_{i = 1} g(x_i)$ for the equality in (1.2) to hold. Several extensions of the basic inequality have been derived. Bhattacharyya [4] and Chapman and Robbins [5] have derived results which yield more stringent inequalities in certain instances. Wolfowitz [21] has extended the result to sequential sampling situations. Cramer [7], Darmois [8], and Barankin [2] have considered joint bounds on sets of estimates of parameters and Hammersley [11] has derived a lower bound of the mean square error of an estimate for the situation in which the parameter to be estimated can only assume discrete values. Barankin [3] has also considered lower bounds on the general absolute central moments of the estimate. All these results assume that the parameters involved are constants. Here we shall consider the case where the parameters are random variables. Thus the lower bound of the mean square error of an estimate will take into account the variability due to both the observations and the parameters involved. Necessary and sufficient conditions for equality of the extended inequality are derived. Most unfavorable distributions, i.e., distributions which maximize the lower bound, are defined, and several examples are given. Extensions analogous to those of Bhattacharyya [4] and Wolfowitz [21] are also considered. Finally, bounds on the variance of linear estimates of the mean of the parameter are derived.

For non-identically distributed random variables, some inequalities on the median and the tails of the distribution of a sample order statistic are derived, and a simple condition for the existence of its moments is studied.

A Cramer type large deviation theorem is proved under alternatives as well as under hypothesis for the generalized linear rank statistic which includes as special cases (unsigned) linear rank statistics, signed linear rank statistics, linear combination of functions of order statistics, and a rank combinatorial statistic.

It is shown in this paper that the product family of countably many families of perfect probability measures defined on countably generated $\sigma$-fields admits a minimal sufficient statistic if and only if each component family admits a minimal sufficient statistic. Moreover, the minimal sufficient statistic of the product family is the "product" of the minimal sufficient statistics for the component families. Examples show that the assumptions on the component families cannot be omitted.

Geometrical foundations of asymptotic inference are described in simple cases, without the machinery of differential geometry. A primary statistical goal is to provide a deeper understanding of the ideas of Fisher and Jeffreys. The role of differential geometry in generalizing results is indicated, further applications are mentioned, and geometrical methods in nonlinear regression are related to those developed for general parametric families.

Two approximate ancillaries, the Efron-Hinkley ancillary and the (signed) likelihood ratio, are compared in a specific example by means of their level curves, their marginal densities and the conditional and unconditional likelihood functions. It is shown that in the present model the latter of the two ancillaries is the better. It is almost exactly distribution constant and has more stable conditional likelihood functions.

It is shown that two distributions both of which have a finite expectation are equal if and only if for every $n \geqq 1$ there exists $1 \leqq k \leqq n$ such that the $k$th order statistics from samples of size $n$ of each distribution have equal expectations. Similarly, it is shown that a distribution with finite expectation is symmetric about zero if and only if for every $n \geqq 0$ there exists $0 \leqq k \leqq 2n + 1$ such that the sum of the expectations of the $k$th smallest and the $k$th largest observations in a sample of size $2n + 1$ is zero.

Hajek (1968) proved that under weak conditions the distribution of a simple linear rank statistic $S$ is asymptotically normal, centered at the mean $ES$. He left open the question whether under the same conditions the centering constant $ES$ may be replaced by a simpler constant $\mu$, as was found to be true in the two-sample case and under different conditions by Chernoff and Savage (1958) and Govindarajulu, LeCam and Rhagavachari (1966). In this paper it is shown that the replacement of $ES$ by $\mu$ is permissible if one of Hajek's conditions is slightly strengthened.