Search

Data-driven most powerful tests are statistical hypothesis decision-making tools that deliver the greatest power against a fixed null hypothesis among all corresponding data-based tests of a given size. When the underlying data distributions are known, the likelihood ratio principle can be applied to conduct most powerful tests. Reversing this notion, we consider the following questions. (a) Assuming a test statistic, say T, is given, how can we transform T to improve the power of the test? (b) Can T be used to generate the most powerful test? (c) How does one compare test statistics with respect to an attribute of the desired most powerful decision-making procedure? To examine these questions, we propose one-to-one mapping of the term "most powerful" to the distribution properties of a given test statistic via matching characterization. This form of characterization has practical applicability and aligns well with the general principle of sufficiency. Findings indicate that to improve a given test, we can employ relevant ancillary statistics that do not have changes in their distributions with respect to tested hypotheses. As an example, the present method is illustrated by modifying the usual t-test under nonparametric settings. Numerical studies based on generated data and a real-data set confirm that the proposed approach can be useful in practice. [ABSTRACT FROM AUTHOR]

The meaning of randomization tests has become obscure in statistics education and practice over the last century. This article makes a fresh attempt at rectifying this core concept of statistics. A new term—"quasi-randomization test"—is introduced to define significance tests based on theoretical models and distinguish these tests from the "randomization tests" based on the physical act of randomization. The practical importance of this distinction is illustrated through a real stepped-wedge cluster-randomized trial. Building on the recent literature on randomization inference, a general framework of conditional randomization tests is developed and some practical methods to construct conditioning events are given. The proposed terminology and framework are then applied to understand several widely used (quasi-)randomization tests, including Fisher's exact test, permutation tests for treatment effect, quasi-randomization tests for independence and conditional independence, adaptive randomization, and conformal prediction. for this article are available online. [ABSTRACT FROM AUTHOR]

We present a new approach known as greedy segmentation (GS) to identify multiple changepoints for a functional data sequence. The proposed multiple changepoint detection criterion links detectability with the projection onto a suitably chosen subspace and the changepoint locations. The changepoint estimator identifies the true changepoints for any predetermined number of changepoint candidates, either over-reporting or under-reporting. This theoretical finding supports the proposed GS estimator, which can be efficiently obtained in a greedy manner. The GS estimator's consistency holds without being restricted to the conventional at most one changepoint condition, and it is robust to the relative positions of the changepoints. Based on the GS estimator, the test statistic's asymptotic distribution leads to the novel GS algorithm, which identifies the number and locations of changepoints. Using intensive simulation studies, we compare the finite sample performance of the GS approach with other competing methods. We also apply our method to temporal changepoint detection in weather datasets. [ABSTRACT FROM AUTHOR]

Following the guiding hypothesis in NLP, similar word frequency vectors may have similar implicatures, but some scholars are more inclined conversational implicatures cannot be obtained only through lexical features. To judge which view is more reasonable and explore the reasons for the divergence between them, whether conversational implicatures can be obtained only through lexical features is verified empirically. Main work of this paper includes: First, based on 600 corpora in the annotated dataset, the values of 20 lexical features of each corpus are obtained by automatic calculation. Second, meta-transformer of logistic regression for selecting features is adopted for feature selection and ranking. Third, after determining the features, the text is classified by the binomial logistic regression with the type of implicatures as labels. Fourth, results are tested for significance to identify relationships between variables. Experiments show that there is a statistical dependence between lexical features and conversational implicatures, and the text classification of implicatures can be performed only based on lexical features. In addition, the results of text classification will not be different due to the difference in context utterance or the type of implicature, and the text classification of implicatures only based on "response utterance" is more efficient. [ABSTRACT FROM AUTHOR]

With the growing number of insurance purchasers, the sophisticated claim analysis system has become an imperative must for any insurance firm. Claims Analysis can be utilized to better understand the customer strata and incorporate the findings throughout the insurance policy enrollment, including the underwriting and approval or rejection stages. In recent years machine learning (ML) technologies are increasingly being used to claims Analysis. However, choosing the optimal techniques, whether the features selection techniques, feature discretization techniques, resampling mechanisms, and ML classifiers for insurance decision assistance, is difficult and can harm the quality of claim suggestions. This study aims to develop appropriate decision models by combining binary classification, feature selection, feature discretization, and data resampling techniques. We did Extensive tests on three different datasets to evaluate the viability of the selected models. We used multiple assessment metrics besides the statistical significance test from The ANOVA test and the Friedman test to evaluate the ML models. The findings show that the models perform highly better after applying the feature discretization technique, reducing dimensionality using feature selection methods and solving the unbalanced data problem with resampling methods. [ABSTRACT FROM AUTHOR]

To assess the clinical and healthcare resource burden among C5 inhibitor (C5i)-treated patients with paroxysmal nocturnal haemoglobinuria (PNH), using patient-reported data. This web-based, cross-sectional survey (01FEB2021–31MAR2021) of adults with PNH treated with eculizumab (France, Germany, UK) or ravulizumab (Germany) included: patient characteristics; treatment patterns/dosage; haematological outcomes (haemoglobin [Hb] levels, transfusions, thrombotic events, breakthrough haemolysis); and medical encounters. Treatment and Hb-level subgroup differences were assessed with statistical significance tests. Among 71 patients, 98.6% were C5i-treated for ≥3 months. The majority (with reported Hb levels) had levels ≤12.0 g/dL (85.7%; n = 54/63). The mean Hb level was 10.2 g/dL (standard deviation [SD]: 2.0; median 10.0 g/dL). Treatment with above label-recommended doses was reported by 30.4% (eculizumab) and 5.3% (ravulizumab) of patients. Within the past 12 months among patients treated with C5i for ≥1 year: 24.1% had ≥1 transfusion; 3.2% had ≥1 thrombosis; and 28.6% had ≥1 breakthrough haemolysis. Among all patients, 26.8% and 31.0% reported emergency department/room [ER] and inpatient visits, respectively. Mean annual, per-patient all-cause medical encounters were: 0.5 (ER); 1.9 (inpatient); and overall outpatient visits ranged by setting from 2.0 to 6.4. Most encounters were PNH-related, with means of 0.4 (ER); 1.8 (inpatient); and 1.6–5.4 (outpatient). Primary haematological and medical encounter outcomes were similar between treatment as well as Hb-level subgroups, with almost no statistically significant differences. Despite at least 3 months of C5i treatment, high proportions of patients with PNH reported low haemoglobin levels and required transfusions and hospitalizations, which suggests remaining unmet needs. [ABSTRACT FROM AUTHOR]

The Canonical Correlation Analysis (CCA) estimates the correlation between two vector variables by maximizing the correlation of linear combinations of their respective components. Here, the CCA is used to find correlation patterns in the last five successive, per pairs, earthquakes ( M ≥ 4.0) preceding 271 main shocks ( M ≥ 5.5) that occurred in the Greek territory during 1964–2018. The vector variables have two components, the earthquake magnitude and interevent time. The statistical significance of CCA is determined by the standard parametric test along with two proposed randomization tests, one using random shuffling of each paired dataset and one using randomly selected pairs of successive earthquakes. Simulations were designed on synthetic data from vector variables having the statistical characteristics of the real observations. The results on uncorrelated variables showed the correct size for the two randomization tests but larger type I error for the parametric significance test for small sample size. For correlated variables, the test power was equally high for both test types. The application of CCA and the significance tests to the Greek seismicity evidence the significant correlation among the last five successive preshocks, proving to be a promising tool in an a posteriori short-term earthquake forecasting. [ABSTRACT FROM AUTHOR]

Standardization has been a widely adopted practice in multiple testing, for it takes into account the variability in sampling and makes the test statistics comparable across different study units. However, despite conventional wisdom to the contrary, we show that there can be a significant loss in information from basing hypothesis tests on standardized statistics rather than the full data. We develop a new class of heteroscedasticity-adjusted ranking and thresholding (HART) rules that aim to improve existing methods by simultaneously exploiting commonalities and adjusting heterogeneities among the study units. The main idea of HART is to bypass standardization by directly incorporating both the summary statistic and its variance into the testing procedure. A key message is that the variance structure of the alternative distribution, which is subsumed under standardized statistics, is highly informative and can be exploited to achieve higher power. The proposed HART procedure is shown to be asymptotically valid and optimal for false discovery rate (FDR) control. Our simulation results demonstrate that HART achieves substantial power gain over existing methods at the same FDR level. We illustrate the implementation through a microarray analysis of myeloma. [ABSTRACT FROM AUTHOR]

In genome-wide epigenetic studies, it is of great scientific interest to assess whether the effect of an exposure on a clinical outcome is mediated through DNA methylations. However, statistical inference for causal mediation effects is challenged by the fact that one needs to test a large number of composite null hypotheses across the whole epigenome. Two popular tests, the Wald-type Sobel's test and the joint significant test using the traditional null distribution are underpowered and thus can miss important scientific discoveries. In this article, we show that the null distribution of Sobel's test is not the standard normal distribution and the null distribution of the joint significant test is not uniform under the composite null of no mediation effect, especially in finite samples and under the singular point null case that the exposure has no effect on the mediator and the mediator has no effect on the outcome. Our results explain why these two tests are underpowered, and more importantly motivate us to develop a more powerful divide-aggregate composite-null test (DACT) for the composite null hypothesis of no mediation effect by leveraging epigenome-wide data. We adopted Efron's empirical null framework for assessing statistical significance of the DACT test. We showed analytically that the proposed DACT method had improved power, and could well control Type I error rate. Our extensive simulation studies showed that, in finite samples, the DACT method properly controlled the Type I error rate and outperformed Sobel's test and the joint significance test for detecting mediation effects. We applied the DACT method to the U.S. Department of Veterans Affairs Normative Aging Study, an ongoing prospective cohort study which included men who were aged 21 to 80 years at entry. We identified multiple DNA methylation CpG sites that might mediate the effect of smoking on lung function with effect sizes ranging from –0.18 to –0.79 and false discovery rate controlled at the level 0.05, including the CpG sites in the genes AHRR and F2RL3. Our sensitivity analysis found small residual correlations (less than 0.01) of the error terms between the outcome and mediator regressions, suggesting that our results are robust to unmeasured confounding factors. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement. [ABSTRACT FROM AUTHOR]

The issue of detection of hydrometeorological trends remains relevant because of the importance of climate change in design, operation, and management studies related to water resources. This study examines the effects of changes in climate and land use on monthly flows (1962–2018) in the Eastern Black Sea basin, Turkey, using innovative trend analysis methods. In this context, innovative polygon trend analysis (IPTA) and innovative trend significance test (ITST) were used to detect the trends and compared with Mann-Kendall test. Only stations with homogeneous data that did not experience non-climatic changes are used in the analysis. IPTA and ITST approaches are much more sensitive than Mann-Kendall in detecting trends. Although the innovative methods are mostly compatible with each other (90%), IPTA presents additional information about trend transitions between successive parts of time series. Results indicate significant decreasing trends in summer months, likely due to diminishing precipitation and effective evaporation. [ABSTRACT FROM AUTHOR]

How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is rarely explained or justified. A new Bayesian decision-theoretic model, "causal binary loss function model," overcomes these issues. It compares the expected loss under a policy intervention with the one under no intervention. These losses are computed based on a particular range of the effect sizes of a policy, the probability mass of this effect size range, the cost of the policy, and the cost of the undesirable event the policy intends to address. The model is more applicable than common statistical decision-theoretic models using the standard loss functions or capturing costs in terms of false positives and false negatives. I exemplify the model's use through three applications and provide an R package. for this article are available online. [ABSTRACT FROM AUTHOR]

Significance testing is often criticized because p-values can be low even though posterior probabilities of the null hypothesis are not low according to some Bayesian models. Those models, however, would assign low prior probabilities to the observation that the p-value is sufficiently low. That conflict between the models and the data may indicate that the models needs revision. Indeed, if the p-value is sufficiently small while the posterior probability according to a model is insufficiently small, then the model will fail a model check. That result leads to a way to calibrate a p-value by transforming it into an upper bound on the posterior probability of the null hypothesis (conditional on rejection) for any model that would pass the check. The calibration may be calculated from a prior probability of the null hypothesis and the stringency of the check without more detailed modeling. An upper bound, as opposed to a lower bound, can justify concluding that the null hypothesis has a low posterior probability. [ABSTRACT FROM AUTHOR]

An important factor in house prices is its location. However, measurement errors arise frequently in the process of observing variables such as the latitude and longitude of the house. The single-index models with measurement errors are used to study the relationship between house location and house price. We obtain the estimators by a SIMEX method based on the local linear method and the estimating equation. To test the significance of the index coefficient and the linearity of the link function, we establish the generalized likelihood ratio (GLR) tests for the models. We demonstrate that the asymptotic null distributions of the established GLR tests follow χ 2 -distributions which are independent of nuisance parameters or functions. Finally, two simulated examples and a real estate valuation data set are given to illustrate the effect of GLR tests. [ABSTRACT FROM AUTHOR]

The discussion on the use and misuse of p-values in 2016 by the American Statistician Association was a timely assertion that statistical concept should be properly used in science. Some researchers, especially the economists, who adopt significance testing and p-values to report their results, may felt confused by the statement, leading to misinterpretations of the statement. In this study, we aim to re-examine the accuracy of the p-value and introduce an alternative way for testing the hypothesis. We conduct a simulation study to investigate the reliability of the p-value. Apart from investigating the performance of p-value, we also introduce some existing approaches, Minimum Bayes Factors and Belief functions, for replacing p-value. Results from the simulation study confirm unreliable p-value in some cases and that our proposed approaches seem to be useful as the substituted tool in the statistical inference. Moreover, our results show that the plausibility approach is more accurate for making decisions about the null hypothesis than the traditionally used p-values when the null hypothesis is true. However, the MBFs of Edwards et al. [Bayesian statistical inference for psychological research. Psychol. Rev. 70(3) (1963), pp. 193–242]; Vovk [A logic of probability, with application to the foundations of statistics. J. Royal Statistical Soc. Series B (Methodological) 55 (1993), pp. 317–351] and Sellke et al. [Calibration of p values for testing precise null hypotheses. Am. Stat. 55(1) (2001), pp. 62–71] provide more reliable results compared to all other methods when the null hypothesis is false. [ABSTRACT FROM AUTHOR]

Hypothesis tests are conducted not only to determine whether a null hypothesis (H0) is true but also to determine the direction or sign of an effect. A simple estimate of the posterior probability of a sign error is PSE = (1 – PH0)p/2 + PH0, depending only on a two-sided p-value and PH0, an estimate of the posterior probability of H0. A convenient option for PH0 is the posterior probability derived from estimating the Bayes factor to be its e p ln (1 / p) lower bound. In that case, PSE depends only on p and an estimate of the prior probability of H0. PSE provides a continuum between significance testing and traditional Bayesian testing. The former effectively assumes the prior probability of H0 is 0, as some statisticians argue. In that case, PSE is equal to a one-sided p-value. (In that sense, PSE is a calibrated p-value.) In traditional Bayesian testing, on the other hand, the prior probability of H0 is at least 50%, which usually brings PSE close to PH0. [ABSTRACT FROM AUTHOR]

Objective: The purpose of this study was to investigate the methodology, interpretation, and perceived value of Spurling's test toward diagnosis/classification and treatment. Methods: An anonymous web-based survey was made available to physical therapist members of the Academy of Orthopaedic Physical Therapy of the American Physical Therapy Association. Based on video demonstrations of technique and symptom distribution, questions included preferred method, criteria for test interpretation, and perceived value of Spurling's test and other examination findings toward clinical decision-making. Professional profile data were also collected. Results: Among the 452 participants completing the survey, no method of testing was preferred by more than 37%, with ipsilateral lateral flexion, rotation, and extension with compression being most frequently preferred followed by ipsilateral lateral flexion with compression at 32%. Proximal provocation of symptoms only without distal symptoms was interpreted as a positive test by 67%. Participants rated Spurling's test of moderate to low value toward diagnosis/classification and treatment. Discussion: Inconsistency with methodology and interpretation of Spurling's test is suggested to be pervasive in physical therapist practice. While an optimal test methodology has yet to be identified, result interpretation does have a basis for clarification toward diagnosis/classification and reduction of unwanted variance in practice. [ABSTRACT FROM AUTHOR]

Millions of measuring instruments are verified each year before being placed on the markets worldwide. In the EU, such initial conformity assessments are regulated by the Measuring Instruments Directive (MID). The MID modules F and F1 on product verification allow for statistical acceptance sampling, whereby only random subsets of instruments need to be inspected. This article re-interprets the acceptance sampling conditions formulated by the MID. The new interpretation is contrasted with the one advanced in WELMEC guide 8.10, and three advantages have become apparent. First, an economic advantage of the new interpretation is a producers' risk bounded from above, such that measuring instruments with sufficient quality are accepted with a guaranteed probability of no less than 95%. Second, a conceptual advantage is that the new MID interpretation fits into the well known, formal framework of statistical hypothesis testing. Thirdly, the new interpretation applies unambiguously to finite-sized lots, even very small ones. We conclude that the new interpretation is to be preferred and suggest re-formulating the statistical sampling conditions in the MID. Re-interpreting the MID conditions implies that currently available sampling plans are either not admissible or not optimal. We derive a new acceptance sampling scheme and recommend its application. for this article are available online. [ABSTRACT FROM AUTHOR]

To describe chronic lymphocytic leukemia (CLL) treatment patterns and patient outcomes in Latin America. This chart review study (NCT02559583; 2008–2015)evaluated time to progression (TTP) and overall survival (OS) outcomes among patients with CLL who initiate done (n = 261) to two (n = 96) lines of therapy (LOT) since diagnosis. Differences in TTP and OS were assessed by Kaplan-Meier analysis, with a log-rank test for statistical significance. Association between therapeutic regimen and risk for disease progression or death was estimated using Cox proportional hazard regression. The most commonly prescribed therapies in both LOTs were chlorambucil-, followed by fludarabine- and cyclophosphamide (C)/CHOP-based therapies. Chlorambucil- and C/CHOP-based therapies were largely prescribed to elderly patients (≥65 years) while fludarabine-based therapy was predominantly used by younger patients (≤65 years). In LOT1, relative to chlorambucil-administered patients, those prescribed fludarabine-based therapies had lower risk of disease progression (hazard ratio [HR] and 95% confidence interval [CI] 0.32 [0.19–0.54]), whereas C/CHOP-prescribed patients had higher risk (HR 95%CI 1.88 [1.17–3.04]). Similar results were observed in LOT2. There was no difference in OS between treatments in both LOTs. Novel therapies such as kinase inhibitors were rarely prescribed in LOT1 or LOT2in Latin America. The greater TTP observed forfludarabine-based therapies could be attributed to the fact that fludarabine-based therapies are predominantly administered to young and healthy patients. Chlorambucil-based therapy, which has limited benefits, is frequently prescribed in Latin America. Prescribing novel agents for fludarabine-based therapy-ineligible patients with CLL is the need of the hour. Trial registration: ClinicalTrials.gov identifier: NCT02559583. [ABSTRACT FROM AUTHOR]

Many scientific and engineering challenges—ranging from pharmacokinetic drug dosage allocation and personalized medicine to marketing mix (4Ps) recommendations—require an understanding of the unobserved heterogeneity to develop the best decision making-processes. In this article, we develop a hypothesis test and the corresponding p-value for testing for the significance of the homogeneous structure in linear mixed models. A robust matching moment construction is used for creating a test that adapts to the size of the model sparsity. When unobserved heterogeneity at a cluster level is constant, we show that our test is both consistent and unbiased even when the dimension of the model is extremely high. Our theoretical results rely on a new family of adaptive sparse estimators of the fixed effects that do not require consistent estimation of the random effects. Moreover, our inference results do not require consistent model selection. We showcase that moment matching can be extended to nonlinear mixed effects models and to generalized linear mixed effects models. In numerical and real data experiments, we find that the developed method is extremely accurate, that it adapts to the size of the underlying model and is decidedly powerful in the presence of irrelevant covariates. for this article are available online. [ABSTRACT FROM AUTHOR]

Modern science frequently involves the analysis of large amount of quantitative information and the simultaneous testing of thousands or even hundreds of thousands null hypotheses. In this context, sometimes, naive deductions derived from the statistical reports substitute the rational thinking. The reproducibility crisis is a direct consequence of the misleading statistical conclusions. In this paper, the authors revisit some of the controversies on the implications derived from the statistical hypothesis testing. They focus on the role of the p-value on the massive multitesting problem and the loss of its standard probabilistic interpretation. The analogy between the hypothesis tests and the usual diagnostic process (both involve a decision-making) is used to point out some limitations in the probabilistic p-value interpretation and to introduce the receiver-operating characteristic, ROC, curve as a useful tool in the large-scale multitesting context. The analysis of the well-known Hedenfalk data illustrates the problem. [ABSTRACT FROM AUTHOR]

The scientific needs and computational limitations of the twentieth century fashioned classical statistical methodology. Both the needs and limitations have changed in the twenty-first, and so has the methodology. Large-scale prediction algorithms—neural nets, deep learning, boosting, support vector machines, random forests—have achieved star status in the popular press. They are recognizable as heirs to the regression tradition, but ones carried out at enormous scale and on titanic datasets. How do these algorithms compare with standard regression techniques such as ordinary least squares or logistic regression? Several key discrepancies will be examined, centering on the differences between prediction and estimation or prediction and attribution (significance testing). Most of the discussion is carried out through small numerical examples. [ABSTRACT FROM AUTHOR]

Previous research finds that low achievers and students from lower socioeconomic status backgrounds are more likely to bully peers, but fails to satisfactorily explain why. This paper tests the hypothesis that feelings of insecurity mediate the relationship between family background and low achievement, as predictors, and self-perception of being a bully, as the outcome. It operationalizes feelings of insecurity as self-esteem and academic-status insecurity. Using a large and diverse dataset of survey responses from secondary school males in the U.S. (N = 7,291, 45% White, 30% Black, 25% Latino from 6th-12th grade), the paper applies multi-level mediation analyses to test the hypothesis separately for Blacks, Whites, and Latinos. For all three groups, the insecurity measures account for virtually all of the relationship between a grade point average and self-perception of being a bully, and about a third of the relationship between family background and the same bullying measure. Implications are discussed. [ABSTRACT FROM AUTHOR]

Although choosing between several alternatives is the basic and most common process of decision-making, it has been well established that when having to decide, people are prone to various biases and seldom elect an alternative rationally. One of the most common and well-researched biases is the 'choice-bias' and many researches demonstrated its effect on decision-makers (both high-level and daily decisions) of the previous generations. This research examined the hypothesis that since the psychological development of 'generation Y' differs significantly from the researched generations, youth today are more resistant to this specific bias. A lifetime of constant exposure to the overabundance of suggestions and opportunities may have influenced decision-making capabilities. A study conducted on two separate groups of students (256 participants) revealed that no choice bias was observed, unlike similar studies conducted with previous generation populations. [ABSTRACT FROM AUTHOR]

In a test of significance, it is common practice to report the p-value as one way of summarizing the incompatibility between a set of data and a proposed model for the data constructed under a set of assumptions together with a null hypothesis. However, the p-value does have some flaws, one being in general its definition for two-sided tests and a related serious logical one of incoherence, in its interpretation as a statistical measure of evidence for its respective null hypothesis. We shall address these two issues in this article. [ABSTRACT FROM AUTHOR]

We give qualitative and quantitative improvements to theorems which enable significance testing in Markov chains, with a particular eye toward the goal of enabling strong, interpretable, and statistically rigorous claims of political gerrymandering. Our results can be used to demonstrate at a desired significance level that a given Markov chain state (e.g., a districting) is extremely unusual (rather than just atypical) with respect to the fragility of its characteristics in the chain. We also provide theorems specialized to leverage quantitative improvements when there is a product structure in the underlying probability space, as can occur due to geographical constraints on districtings. [ABSTRACT FROM AUTHOR]

Critical values and p-values of statistical hypothesis tests are often derived using asymptotic approximations of sampling distributions. However, this sometimes results in tests that are conservative (i.e., understate the frequency of an incorrectly rejected null hypothesis by employing too stringent of a threshold for rejection). Although computationally rigorous options (e.g., the bootstrap) are available for such situations, we illustrate that simple transformations can be used to improve both the size and power of such tests. Using a logarithmic transformation, we show that the transformed statistic is asymptotically equivalent to its untransformed analogue under the null hypothesis and is divergent from the untransformed version under the alternative (yielding a potentially substantial increase in power). The transformation is applied to several easily-accessible statistical hypothesis tests, a few of which are taught in introductory statistics courses. With theoretical arguments and simulations, we illustrate that the log transformation is preferable to other forms of correction (such as statistics that use a multiplier). Finally, we illustrate application of the method to a well-known dataset. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]

In comparing a collection of K populations, it is common practice to display in one visualization confidence intervals for the corresponding population parameters θ1, θ2, ..., θK. For a pair of confidence intervals that do (or do not) overlap, viewers of the visualization are cognitively compelled to declare that there is not (or there is) a statistically significant difference between the two corresponding population parameters. It is generally well known that the method of examining overlap of pairs of confidence intervals should not be used for formal hypothesis testing. However, use of a single visualization with overlapping and nonoverlapping confidence intervals leads many to draw such conclusions, despite the best efforts of statisticians toward preventing users from reaching such conclusions. In this article, we summarize some alternative visualizations from the literature that can be used to properly test equality between a pair of population parameters. We recommend that these visualizations be used with caution to avoid incorrect statistical inference. The methods presented require only that we have K sample estimates and their associated standard errors. We also assume that the sample estimators are independent, unbiased, and normally distributed. [ABSTRACT FROM AUTHOR]

Fisher succeeded early on in redefining Student's t-distribution in geometrical terms on a central hypersphere. Intriguingly, a noncentral analytical extension for this fundamental Fisher–Student's central hypersphere h-distribution does not exist. We therefore set to derive the noncentral h-distribution and use it to graphically illustrate the limitations of the Neyman–Pearson null hypothesis significance testing framework and the strengths of the Bayesian statistical hypothesis analysis framework on the hypersphere polar axis, a compact nontrivial one-dimensional parameter space. Using a geometrically meaningful maximal entropy prior, we requalify the apparent failure of an important psychological science reproducibility project. We proceed to show that the Bayes factor appropriately models the two-sample t-test p-value density of a gene expression profile produced by the high-throughput genomic-scale microarray technology, and provides a simple expression for a local false discovery rate addressing the multiple hypothesis testing problem brought about by such a technology. [ABSTRACT FROM AUTHOR]

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced. [ABSTRACT FROM AUTHOR]

As statisticians and scientists consider a world beyond p < 0.05, it is important to not lose sight of how we got to this point. Although significance testing and p-values are often presented as prescriptive procedures, they came about through a process of refinement and extension to other disciplines. Ronald A. Fisher and his contemporaries formalized these methods in the early twentieth century and Fisher's 1925 Statistical Methods for Research Workers brought the techniques to experimentalists in a variety of disciplines. Understanding how these methods arose, spread, and were argued over since then illuminates how p < 0.05 came to be a standard for scientific inference, the advantage it offered at the time, and how it was interpreted. This historical perspective can inform the work of statisticians today by encouraging thoughtful consideration of how their work, including proposed alternatives to the p-value, will be perceived and used by scientists. And it can engage students more fully and encourage critical thinking rather than rote applications of formulae. Incorporating history enables students, practitioners, and statisticians to treat the discipline as an ongoing endeavor, crafted by fallible humans, and provides a deeper understanding of the subject and its consequences for science and society. [ABSTRACT FROM AUTHOR]

The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of "significance levels" and "P-values" in authoritative sources, and the consequent misinterpretation of P-values as error probabilities. It then discusses several properties of P-values that have been presented as fatal flaws: That P-values exhibit extreme variation across samples (and thus are "unreliable"), confound effect size with sample size, are sensitive to sample size, and depend on investigator sampling intentions. These properties are often criticized from a likelihood or Bayesian framework, yet they are exactly the properties P-values should exhibit when they are constructed and interpreted correctly within their originating framework. Other common criticisms are that P-values force users to focus on irrelevant hypotheses and overstate evidence against those hypotheses. These problems are not however properties of P-values but are faults of researchers who focus on null hypotheses and overstate evidence based on misperceptions that p = 0.05 represents enough evidence to reject hypotheses. Those problems are easily seen without use of Bayesian concepts by translating the observed P-value p into the Shannon information (S-value or surprisal) –log2(p). [ABSTRACT FROM AUTHOR]

p-Values and Null Hypothesis Significance Testing (NHST), combined with a large number of institutional factors, jointly define the Generally Accepted Soft Social Science Publishing Process (GASSSPP) that is now dominant in the social sciences and is increasingly used elsewhere. The case against NHST and the GASSSPP has been abundantly articulated over past decades, and yet it continues to spread, supported by a large number of self-reinforcing institutional processes. In this article, the author presents a number of steps that may be taken to counter the spread of this corruption that directly address the institutional forces, both as individuals and through collaborative efforts. While individual efforts are indispensable to this undertaking, the author argues that these alone cannot succeed unless the institutional forces are also addressed. for this article are available online. [ABSTRACT FROM AUTHOR]

Most statistical analyses use hypothesis tests or estimation about parameters to form inferential conclusions. I think this is noble, but misguided. The point of view expressed here is that observables are fundamental, and that the goal of statistical modeling should be to predict future observations, given the current data and other relevant information. Further, the prediction of future observables provides multiple advantages to practicing scientists, and to science in general. These include an interpretable numerical summary of a quantity of direct interest to current and future researchers, a calibrated prediction of what's likely to happen in future experiments, a prediction that can be either "corroborated" or "refuted" through experimentation, and avoidance of inference about parameters; quantities that exists only as convenient indices of hypothetical distributions. Finally, the predictive probability of a future observable can be used as a standard for communicating the reliability of the current work, regardless of whether confirmatory experiments are conducted. Adoption of this paradigm would improve our rigor for scientific accuracy and reproducibility by shifting our focus from "finding differences" among hypothetical parameters to predicting observable events based on our current scientific understanding. [ABSTRACT FROM AUTHOR]

The author has previously proposed results blind manuscript evaluation (RBME) as a method of ameliorating often cited problems of statistical inference and scientific publication, notably publication bias, overuse/misuse of null hypothesis significance testing (NHST), and irreproducibility of reported scientific results. In RBME, manuscripts submitted to scientific journals are assessed for suitability for publication without regard to their reported results. Criteria for publication are based exclusively on the substantive importance of the research question addressed in the study, conveyed in the Introduction section of the manuscript, and the quality of the methodology, as reported in the Methods section. Practically, this policy is implemented by a two stage process whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers and a provisional decision regarding acceptance is made, followed by a second stage in which the complete manuscript is distributed for review but only if the decision of the first stage is for acceptance. The present paper expands upon this recommendation by addressing implications of this proposed policy with respect to statistical consultation and collaboration in research. It is suggested that under RBME, statisticians will become more integrated into research endeavors and called upon sooner for their input. [ABSTRACT FROM AUTHOR]

A central objective of empirical research on treatment response is to inform treatment choice. Unfortunately, researchers commonly use concepts of statistical inference whose foundations are distant from the problem of treatment choice. It has been particularly common to use hypothesis tests to compare treatments. Wald's development of statistical decision theory provides a coherent frequentist framework for use of sample data on treatment response to make treatment decisions. A body of recent research applies statistical decision theory to characterize uniformly satisfactory treatment choices, in the sense of maximum loss relative to optimal decisions (also known as maximum regret). This article describes the basic ideas and findings, which provide an appealing practical alternative to use of hypothesis tests. For simplicity, the article focuses on medical treatment with evidence from classical randomized clinical trials. The ideas apply generally, encompassing use of observational data and treatment choice in nonmedical contexts. [ABSTRACT FROM AUTHOR]

Recent efforts by the American Statistical Association to improve statistical practice, especially in countering the misuse and abuse of null hypothesis significance testing (NHST) and p-values, are to be welcomed. But will they be successful? The present study offers compelling evidence that this will be an extraordinarily difficult task. Dramatic citation-count data on 25 articles and books severely critical of NHST's negative impact on good science, underlining that this issue was/is well known, did nothing to stem its usage over the period 1960–2007. On the contrary, employment of NHST increased during this time. To be successful in this endeavor, as well as restoring the relevance of the statistics profession to the scientific community in the 21st century, the ASA must be prepared to dispense detailed advice. This includes specifying those situations, if they can be identified, in which the p-value plays a clearly valuable role in data analysis and interpretation. The ASA might also consider a statement that recommends abandoning the use of p-values. [ABSTRACT FROM AUTHOR]

Replication is complicated in psychological research because studies of a given psychological phenomenon can never be direct or exact replications of one another, and thus effect sizes vary from one study of the phenomenon to the next—an issue of clear importance for replication. Current large-scale replication projects represent an important step forward for assessing replicability, but provide only limited information because they have thus far been designed in a manner such that heterogeneity either cannot be assessed or is intended to be eliminated. Consequently, the nontrivial degree of heterogeneity found in these projects represents a lower bound on the true degree of heterogeneity. We recommend enriching large-scale replication projects going forward by embracing heterogeneity. We argue this is the key for assessing replicability: if effect sizes are sufficiently heterogeneous—even if the sign of the effect is consistent—the phenomenon in question does not seem particularly replicable and the theory underlying it seems poorly constructed and in need of enrichment. Uncovering why and revising theory in light of it will lead to improved theory that explains heterogeneity and increases replicability. Given this, large-scale replication projects can play an important role not only in assessing replicability but also in advancing theory. [ABSTRACT FROM AUTHOR]

As the debate over best statistical practices continues in academic journals, conferences, and the blogosphere, working researchers (e.g., psychologists) need to figure out how much time and effort to invest in attending to experts' arguments, how to design their next project, and how to craft a sustainable long-term strategy for data analysis and inference. The present special issue of The American Statistician promises help. In this article, we offer a modest proposal for a continued and informed use of the conventional p-value without the pitfalls of statistical rituals. Other statistical indices should complement reporting, and extra-statistical (e.g., theoretical) judgments ought to be made with care and clarity. [ABSTRACT FROM AUTHOR]

P values linked to null hypothesis significance testing (NHST) is the most widely (mis)used method of statistical inference. Empirical data suggest that across the biomedical literature (1990–2015), when abstracts use P values 96% of them have P values of 0.05 or less. The same percentage (96%) applies for full-text articles. Among 100 articles in PubMed, 55 report P values, while only 4 present confidence intervals for all the reported effect sizes, none use Bayesian methods and none use false-discovery rate. Over 25 years (1990–2015), use of P values in abstracts has doubled for all PubMed, and tripled for meta-analyses, while for some types of designs such as randomized trials the majority of abstracts report P values. There is major selective reporting for P values. Abstracts tend to highlight most favorable P values and inferences use even further spin to reach exaggerated, unreliable conclusions. The availability of large-scale data on P values from many papers has allowed the development and applications of methods that try to detect and model selection biases, for example, p-hacking, that cause patterns of excess significance. Inferences need to be cautious as they depend on the assumptions made by these models and can be affected by the presence of other biases (e.g., confounding in observational studies). While much of the unreliability of past and present research is driven by small, underpowered studies, NHST with P values may be also particularly problematic in the era of overpowered big data. NHST and P values are optimal only in a minority of current research. Using a more stringent threshold, as in the recently proposed shift from P < 0.05 to P < 0.005, is a temporizing measure to contain the flood and death-by-significance. NHST and P values may be replaced in many fields by other, more fit-for-purpose, inferential methods. However, curtailing selection biases requires additional measures, beyond changes in inferential methods, and in particular reproducible research practices. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editor discusses articles in the issue on topics including p-values, significance testing and statistics.

When the editors of Basic and Applied Social Psychology effectively banned the use of null hypothesis significance testing (NHST) from articles published in their journal, it set off a fire-storm of discussions both supporting the decision and defending the utility of NHST in scientific research. At the heart of NHST is the p-value which is the probability of obtaining an effect equal to or more extreme than the one observed in the sample data, given the null hypothesis and other model assumptions. Although this is conceptually different from the probability of the null hypothesis being true, given the sample, p-values nonetheless can provide evidential information, toward making an inference about a parameter. Applying a 10,000-case simulation described in this article, the authors found that p-values' inferential signals to either reject or not reject a null hypothesis about the mean (α = 0.05) were consistent for almost 70% of the cases with the parameter's true location for the sampled-from population. Success increases if a hybrid decision criterion, minimum effect size plus p-value (MESP), is used. Here, rejecting the null also requires the difference of the observed statistic from the exact null to be meaningfully large or practically significant, in the researcher's judgment and experience. The simulation compares performances of several methods: from p-value and/or effect size-based, to confidence-interval based, under various conditions of true location of the mean, test power, and comparative sizes of the meaningful distance and population variability. For any inference procedure that outputs a binary indicator, like flagging whether a p-value is significant, the output of one single experiment is not sufficient evidence for a definitive conclusion. Yet, if a tool like MESP generates a relatively reliable signal and is used knowledgeably as part of a research process, it can provide useful information. [ABSTRACT FROM AUTHOR]

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of "greater statistical stringency," including suggestions to require larger sample sizes and to lower the highly criticized "p < 0.05" significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this article, we develop a novel optimality model that, given current incentives to publish, predicts a researcher's most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable prestudy odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of "greater statistical stringency" with the goal of informing the ongoing debate. [ABSTRACT FROM AUTHOR]

It is now widely accepted that the techniques of null hypothesis significance testing (NHST) are routinely misused and misinterpreted by researchers seeking insight from data. There is, however, no consensus on acceptable alternatives, leaving researchers with little choice but to continue using NHST, regardless of its failings. I examine the potential for the Analysis of Credibility (AnCred) to resolve this impasse. Using real-life examples, I assess the ability of AnCred to provide researchers with a simple but robust framework for assessing study findings that goes beyond the standard dichotomy of statistical significance/nonsignificance. By extracting more insight from standard summary statistics while offering more protection against inferential fallacies, AnCred may encourage researchers to move toward the post p < 0.05 era. [ABSTRACT FROM AUTHOR]

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced. [ABSTRACT FROM AUTHOR]

As statisticians and scientists consider a world beyond p < 0.05, it is important to not lose sight of how we got to this point. Although significance testing and p-values are often presented as prescriptive procedures, they came about through a process of refinement and extension to other disciplines. Ronald A. Fisher and his contemporaries formalized these methods in the early twentieth century and Fisher's 1925 Statistical Methods for Research Workers brought the techniques to experimentalists in a variety of disciplines. Understanding how these methods arose, spread, and were argued over since then illuminates how p < 0.05 came to be a standard for scientific inference, the advantage it offered at the time, and how it was interpreted. This historical perspective can inform the work of statisticians today by encouraging thoughtful consideration of how their work, including proposed alternatives to the p-value, will be perceived and used by scientists. And it can engage students more fully and encourage critical thinking rather than rote applications of formulae. Incorporating history enables students, practitioners, and statisticians to treat the discipline as an ongoing endeavor, crafted by fallible humans, and provides a deeper understanding of the subject and its consequences for science and society. [ABSTRACT FROM AUTHOR]

Longstanding concerns with the role and interpretation of p-values in statistical practice prompted the American Statistical Association (ASA) to make a statement on p-values. The ASA statement spurred a flurry of responses and discussions by statisticians, with many wondering about the steps necessary to expand the adoption of these principles. Introductory statistics classrooms are key locations to introduce and emphasize the nuance related to p-values; in part because they engrain appropriate analysis choices at the earliest stages of statistics education, and also because they reach the broadest group of students. We propose a framework for statistics departments to conduct a content audit for p-value principles in their introductory curriculum. We then discuss the process and results from applying this course audit framework within our own statistics department. We also recommend meeting with client departments as a complement to the course audit. Discussions about analyses and practices common to particular fields can help to evaluate if our service courses are meeting the needs of client departments and to identify what is needed in our introductory courses to combat the misunderstanding and future misuse of p-values. [ABSTRACT FROM AUTHOR]

Evaluating the importance and the strength of empirical evidence requires asking three questions: First, what are the practical implications of the findings? Second, how precise are the estimates? Confidence intervals provide an intuitive way to communicate precision. Although nontechnical audiences often misinterpret confidence intervals (CIs), I argue that the result is less dangerous than the misunderstandings that arise from hypothesis tests. Third, is the model correctly specified? The validity of point estimates and CIs depends on the soundness of the underlying model. [ABSTRACT FROM AUTHOR]

The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of "significance levels" and "P-values" in authoritative sources, and the consequent misinterpretation of P-values as error probabilities. It then discusses several properties of P-values that have been presented as fatal flaws: That P-values exhibit extreme variation across samples (and thus are "unreliable"), confound effect size with sample size, are sensitive to sample size, and depend on investigator sampling intentions. These properties are often criticized from a likelihood or Bayesian framework, yet they are exactly the properties P-values should exhibit when they are constructed and interpreted correctly within their originating framework. Other common criticisms are that P-values force users to focus on irrelevant hypotheses and overstate evidence against those hypotheses. These problems are not however properties of P-values but are faults of researchers who focus on null hypotheses and overstate evidence based on misperceptions that p = 0.05 represents enough evidence to reject hypotheses. Those problems are easily seen without use of Bayesian concepts by translating the observed P-value p into the Shannon information (S-value or surprisal) -log2(p). [ABSTRACT FROM AUTHOR]

The American Statistical Association's Symposium on Statistical Inference (SSI) included a session on how editorial practices should change in a universe no longer dominated by null hypothesis significance testing (NHST). The underlying assumptions were first, that NHST is problematic; and second, that editorial practices really should change. The present article is based on my talk in this session, and on these assumptions. Consistent with the spirit of the SSI, my focus is not on what reviewers and editors should not do (e.g., NHST) but rather on what they should do, with an emphasis on changes that are not obvious. The recommended changes include a wider consideration of the nature of the contribution than submitted manuscripts usually receive; a greater tolerance of ambiguity; more of an emphasis on the thinking and execution of the study, with a decreased emphasis on the findings; replacing NHST with the a priori procedure; and a call for reviewers and editors to recognize that there are many cases where the basic assumptions of inferential statistical procedures simply are not met, and that inferential statistics (even the a priori procedure) may consequently be inappropriate. [ABSTRACT FROM AUTHOR]