Search

Null hypothesis significance testing (NHST) is the default approach to statistical analysis and reporting in marketing and the biomedical and social sciences more broadly. Despite its default role, NHST has long been criticized by both statisticians and applied researchers, including those within marketing. Therefore, the authors propose a major transition in statistical analysis and reporting. Specifically, they propose moving beyond binary: abandoning NHST as the default approach to statistical analysis and reporting. To facilitate this, they briefly review some of the principal problems associated with NHST. They next discuss some principles that they believe should underlie statistical analysis and reporting. They then use these principles to motivate some guidelines for statistical analysis and reporting. They next provide some examples that illustrate statistical analysis and reporting that adheres to their principles and guidelines. They conclude with a brief discussion. [ABSTRACT FROM AUTHOR]

In management research, fixed alpha levels in statistical testing are ubiquitous. However, in highly powered studies, they can lead to Lindley's paradox, a situation where the null hypothesis is rejected despite evidence in the test actually supporting it. We propose a sample-size-dependent alpha level that combines the benefits of both frequentist and Bayesian statistics, enabling strict hypothesis testing with known error rates while also quantifying the evidence for a hypothesis. We offer actionable guidelines of how to implement the sample-size-dependent alpha in practice and provide an R-package and web app to implement our method for regression models. By using this approach, researchers can avoid mindless defaults and instead justify alpha as a function of sample size, thus improving the reliability of statistical analysis in management research. [ABSTRACT FROM AUTHOR]

Most researchers receive formal training in frequentist statistics during their undergraduate studies. In particular, hypothesis testing is usually rooted on the null hypothesis significance testing paradigm and its p‐value. Null hypothesis Bayesian testing and its so‐called Bayes factor are now becoming increasingly popular. Although the Bayes factor is often introduced as being the Bayesian counterpart to the p‐value, its computation, use, and interpretation are quite distinct from the p‐value. There is now evidence confirming that the application of the Bayes factor in applied research is ill‐devised. To improve the current status quo, we have created a Shiny/R app called the Bayes factor, which offers a dynamic tutorial for learning all the basics about the Bayes factor. In this paper, we explain how the app works and we offer suggestions on how to use it in class or self‐study settings. The app is freely available at https://statsedge.org/shiny/LearnBF/. [ABSTRACT FROM AUTHOR]

We propose and illustrate a general framework in which spatial autocorrelation is measured by the Frobenius product of two kernels, a feature kernel and a spatial kernel. The resulting autocorrelation index δ$$ \delta $$ generalizes Moran's index in the weighted, multivariate setting, where regions, differing in importance, are characterized by multivariate features. Spatial kernels can traditionally be obtained from a matrix of spatial weights, or directly from geographical distances. In the former case, the Markov transition matrix defined by row‐normalized spatial weights must be made compatible with the regional weights, as well as reversible. Equivalently, space is specified by a symmetric exchange matrix containing the joint probabilities to select a pair of regions. Four original weight‐compatible constructions, based upon the binary adjacency matrix, are presented and analyzed. Weighted multidimensional scaling on kernels yields a low‐dimensional visualization of both the feature and the spatial configurations. The expected values of the first four moments of δ$$ \delta $$ under the null hypothesis of absence of spatial autocorrelation can be exactly computed under a new approach, invariant orthogonal integration, thus permitting to test the significance of δ$$ \delta $$ beyond the normal approximation, which only involves its expectation and expected variance. Various illustrations are provided, investigating the spatial autocorrelation of political and social features among French departments. [ABSTRACT FROM AUTHOR]

In the context of functional time series, we propose a significance test to distinguish between short memory with a change point and long range dependence. The test is based on coefficients of projections onto an optimal direction that captures the dependence structure of the latent stationary functions that are not observable due to a potential change point. The optimal direction must be estimated as well. The test statistic is constructed using the local Whittle estimator applied to these coefficients. It has standard normal distribution under the null hypothesis (change point) and diverges to infinity under the alternative (long range dependence). The article includes asymptotic theory, a simulation study and an application to curve‐valued time series derived from intraday asset prices. [ABSTRACT FROM AUTHOR]

The exponential random graph (ERGM) model is a commonly used statistical framework for studying the determinants of tie formations from social network data. To test scientific theories under ERGMs, statistical inferential techniques are generally used based on traditional significance testing using p -values. This methodology has certain limitations, however, such as its inconsistent behavior when the null hypothesis is true, its inability to quantify evidence in favor of a null hypothesis, and its inability to test multiple hypotheses with competing equality and/or order constraints on the parameters of interest in a direct manner. To tackle these shortcomings, this paper presents Bayes factors and posterior probabilities for testing scientific expectations under a Bayesian framework. The methodology is implemented in the R package BFpack. The applicability of the methodology is illustrated using empirical collaboration networks and policy networks. • Classical hypothesis testing using p -values under ERGMs has certain limitations. • A Bayes factor is proposed that avoids these limitations. • The Bayes factor can quantify the relative evidence that an ERGM coefficient is zero. • The Bayes factor can test hypotheses with order constraints on the ERGM coefficients. • The Bayesian test is readily available using the R package 'BFpack'. [ABSTRACT FROM AUTHOR]

Using compelling empirical strategies and sound reasoning, Simpson and Sims et al. ably contribute to our growing understanding of inferential errors that arise when filtering research findings using statistical significance. Drawing on the "Type M" (magnitude) and "Type S" (sign) errors described by Gelman and Carlin, both articles demonstrate that large Type M errors almost certainly exist when applying the statistical-significance filter in the field of education. In this comment, I discuss the pros and cons of four of the more actionable suggestions offered by the authors of both articles in their discussion sections. I also provide my own recommendations for drawing inferences about causal relationships that I believe will empower researchers and decision makers to more productively use evidence while avoiding the types of inferential mistakes examined by Sims et al. and Simpson. [ABSTRACT FROM AUTHOR]

Why you should read this article: • To understand how the permutation test can be used to test null hypothesis • To appreciate how much easier it is to understand than significance tests based on t statistics • To illustrate how the permutation test requires fewer assumptions than t-tests. Background: Quantitative researchers can use permutation tests to conduct null hypothesis significance testing without resorting to complicated distribution theory. A permutation test can reach conclusions in hypothesis testing that are the same as those of better-known tests such as the t-test but is much easier to understand and implement. Aim: To introduce and explain permutation tests using two real examples of independent and dependent t-tests and their corresponding permutation tests. Discussion: This article traces the history of permutation tests, explains the possible reason for their absence in textbooks and offers a simple example of their implementation. It provides simple code written in the R programming language to generate the null distributions and P-values for the permutation tests. Conclusion: Permutation tests do not require the strict model assumptions of t-tests and can be robust alternatives. Implications for practice: Permutation tests are a useful addition to practitioners’ research repertoire for testing hypotheses. [ABSTRACT FROM AUTHOR]

The statistical significance of a clinical trial analysis result is determined by a mathematical calculation and probability based on null hypothesis significance testing. However, statistical significance does not always align with meaningful clinical effects; thus, assigning clinical relevance to statistical significance is unreasonable. A statistical result incorporating a clinically meaningful difference is a better approach to present statistical significance. Thus, the minimal clinically important difference (MCID), which requires integrating minimum clinically relevant changes from the early stages of research design, has been introduced. As a follow-up to the previous statistical round article on P values, confidence intervals, and effect sizes, in this article, we present hands-on examples of MCID and various effect sizes and discuss the terms statistical significance and clinical relevance, including cautions regarding their use. [ABSTRACT FROM AUTHOR]

This paper considers the problem of testing statistical hypotheses about the variance components under linear models with crossed random effects. The objective here is two-fold. First, new derivations of exactly distributed F test statistics are presented. Second, an alternative Monte Carlo permutation procedure is proposed to approximate the distribution of the F statistics, which shows its usefulness when the error components distributions depart from normality. Transformations that uniquely decompose the covariance structure of the response vector and do not sacrifice any part of the data, as existing methods do, are presented in matrix form. The suggested transformations highlight the exchangeable covariance structure of the model under the null hypotheses of interest and thus motivate the use of the permutation procedure. Comments on the performance of the proposed procedure compared to existing tests as well as using a real data example are provided. [ABSTRACT FROM AUTHOR]

Edwin Locke's (1969, 1976, 1984) range-of-affect (RoA) hypothesis played a critical role in the literature on job satisfaction. Drawing on the parallel between job facets (e.g., freedom and promotion opportunities) to job and life domains (e.g., health and family) to life, the RoA hypothesis has also been applied to the subjective well-being (SWB) literature. The application of the RoA hypothesis to the SWB literature has focused on the topic of domain importance weighting which refers to the use of weighting of domain importance in aggregating domain-specific satisfaction to obtain overall satisfaction or SWB. The purpose of this study is to examine the applicability of the RoA hypothesis to the SWB literature, particularly concerning the topic of domain importance weighting. Based on the analysis of two survey data sets, this study found that not all empirical evidence supporting the RoA hypothesis from the job satisfaction literature could be replicated in the context of SWB. This study also pointed out that the empirical evidence presented to oppose domain importance weighting in the SWB context was based on an approach that deviated from the conventional approach of the null hypothesis significance test. The applicability of the RoA hypothesis from the job satisfaction context to the SWB context should not be assumed. [ABSTRACT FROM AUTHOR]

Because Null Hypothesis Significance Tests (NHST) can be misleading particularly when sample sizes are large, reporting effect sizes is recommended. Effect sizes such as Cohen's d, however, are abstract statistics and not relevant to decision-making. This article introduced an alternative formulation of Cohen's d based on a "minimal important percent difference" (MIPD) between groups that has an intuitive, practical meaning. Cohen's d was calculated based on detecting MIPDs of 5% and 10% differences in means. Power analyses were used to determine the number of responses needed to detect differences in means. Power-probability sample size requirements were examined for an 80% and a 50% chance of accepting a difference of MIPD at p <.05. We proposed a four-step process to show how both effect size and NHST jointly contribute to decision-making. To demonstrate the process, data were obtained for 6 years of annual waterfowl surveys in Illinois (N = 17,265, average response rate = 41%). [ABSTRACT FROM AUTHOR]

The.05 boundary within Null Hypothesis Statistical Testing (NHST) "has made a lot of people very angry and been widely regarded as a bad move" (to quote Douglas Adams). Here, we move past meta-scientific arguments and ask an empirical question: What is the psychological standing of the.05 boundary for statistical significance? We find that graduate students in the psychological sciences show a boundary effect when relating p-values across.05. We propose this psychological boundary is learned through statistical training in NHST and reading a scientific literature replete with "statistical significance". Consistent with this proposal, undergraduates do not show the same sensitivity to the.05 boundary. Additionally, the size of a graduate student's boundary effect is not associated with their explicit endorsement of questionable research practices. These findings suggest that training creates distortions in initial processing of p-values, but these might be dampened through scientific processes operating over longer timescales. [ABSTRACT FROM AUTHOR]

Ecologists and evolutionary biologists are regularly tasked with the comparison of binary data across groups. There is, however, some discussion in the biostatistics literature about the best methodology for the analysis of data comprising binary explanatory and response variables forming a 2 × 2 contingency table.We assess several methodologies for the analysis of 2 × 2 contingency tables using a simulation scheme of different sample sizes with outcomes evenly or unevenly distributed between groups. Specifically, we assess the commonly recommended logistic (generalised linear model [GLM]) regression analysis, the classical Pearson chi‐squared test and four conventional alternatives (Yates' correction, Fisher's exact, exact unconditional and mid‐p), as well as the widely discouraged linear model (LM) regression.We found that both LM and GLM analyses provided unbiased estimates of the difference in proportions between groups. LM and GLM analyses also provided accurate standard errors and confidence intervals when the experimental design was balanced. When the experimental design was unbalanced, sample size was small, and one of the two groups had a probability close to 1 or 0, LM analysis could substantially over‐ or under‐represent statistical uncertainty. For null hypothesis significance testing, the performance of the chi‐squared test and LM analysis were almost identical. Across all scenarios, both had high power to detect non‐null effects and reject false positives. By contrast, the GLM analysis was underpowered when using z‐based p‐values, in particular when one of the two groups had a probability near 1 or 0. The GLM using the LRT had better power to detect non‐null results.Our simulation results suggest that, wherever a chi‐squared test would be recommended, a linear regression is a suitable alternative for the analysis of 2 × 2 contingency table data. When researchers opt for more sophisticated procedures, we provide R functions to calculate the standard error of a difference between two probabilities from a Bernoulli GLM output using the delta method. We also explore approaches to compliment GLM analysis of 2 × 2 contingency tables with credible intervals on the probability scale. These additional operations should support researchers to make valid assessments of both statistical and practical significances. [ABSTRACT FROM AUTHOR]

This paper focuses on the likelihood ratio test (LRT) statistics for different hypothesis tests. Assuming that a random sample is from a normal population, we make the sample size n and the dimension p close to infinity and satisfy p < n − c for some 1 ≤ c ≤ 4. Based on this assumption, the moderate deviation principle (MDP) for the LRT will be given under the null hypothesis. The corresponding numerical simulation results are shown at the end of the paper. [ABSTRACT FROM AUTHOR]

Mediation analysis aims to assess if, and how, a certain exposure influences an outcome of interest through intermediate variables. This problem has recently gained a surge of attention due to the tremendous need for such analyses in scientific fields. Testing for the mediation effect (ME) is greatly challenged by the fact that the underlying null hypothesis (i.e. the absence of MEs) is composite. Most existing mediation tests are overly conservative and thus underpowered. To overcome this significant methodological hurdle, we develop an adaptive bootstrap testing framework that can accommodate different types of composite null hypotheses in the mediation pathway analysis. Applied to the product of coefficients test and the joint significance test, our adaptive testing procedures provide type I error control under the composite null, resulting in much improved statistical power compared to existing tests. Both theoretical properties and numerical examples of the proposed methodology are discussed. [ABSTRACT FROM AUTHOR]

Data-driven most powerful tests are statistical hypothesis decision-making tools that deliver the greatest power against a fixed null hypothesis among all corresponding data-based tests of a given size. When the underlying data distributions are known, the likelihood ratio principle can be applied to conduct most powerful tests. Reversing this notion, we consider the following questions. (a) Assuming a test statistic, say T, is given, how can we transform T to improve the power of the test? (b) Can T be used to generate the most powerful test? (c) How does one compare test statistics with respect to an attribute of the desired most powerful decision-making procedure? To examine these questions, we propose one-to-one mapping of the term "most powerful" to the distribution properties of a given test statistic via matching characterization. This form of characterization has practical applicability and aligns well with the general principle of sufficiency. Findings indicate that to improve a given test, we can employ relevant ancillary statistics that do not have changes in their distributions with respect to tested hypotheses. As an example, the present method is illustrated by modifying the usual t-test under nonparametric settings. Numerical studies based on generated data and a real-data set confirm that the proposed approach can be useful in practice. [ABSTRACT FROM AUTHOR]

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example. [ABSTRACT FROM AUTHOR]

The article discusses the phenomenon known as the "winner's curse" in medical research, where large effect sizes observed in small discovery trials often shrink or disappear in subsequent larger trials. The author explains that this effect is a consequence of null hypothesis significance testing and the need for larger trials to cross the significance threshold. The article also highlights the issue of publication bias, where trials reporting significant results are more likely to be published, leading to a skewed representation of the evidence. The author provides recommendations for interpreting effect sizes in small trials and emphasizes the importance of conducting larger trials to obtain more accurate estimates of treatment effects. [Extracted from the article]

The import or force of the result of a statistical test has long been portrayed as consistent with deductive reasoning. The simplest form of deductive argument has a first premise with conditional form, such as p → q, which means that "if p is true, then q must be true." Given the first premise, one can either affirm or deny the antecedent clause (p) or affirm or deny the consequent claim (q). This leads to four forms of deductive argument, two of which are valid forms of reasoning and two of which are invalid. The typical conclusion is that only a single form of argument—denying the consequent, also known as modus tollens —is a reasonable analog of decisions based on statistical hypothesis testing. Now, statistical evidence is never certain, but is associated with a probability (i.e., a p -level). Some have argued that modus tollens, when probabilified, loses its force and leads to ridiculous, nonsensical conclusions. Their argument is based on specious problem setup. This note is intended to correct this error and restore the position of modus tollens as a valid form of deductive inference in statistical matters, even when it is probabilified. [ABSTRACT FROM AUTHOR]

The success of preclinical research hinges on exploratory and confirmatory animal studies. Traditional null hypothesis significance testing is a common approach to eliminate the chaff from a collection of drugs, so that only the most promising treatments are funneled through to clinical research phases. Balancing the number of false discoveries and false omissions is an important aspect to consider during this process. In this paper, we compare several preclinical research pipelines, either based on null hypothesis significance testing or based on Bayesian statistical decision criteria. We build on a recently published large-scale meta-analysis of reported effect sizes in preclinical animal research and elicit a non-informative prior distribution under which both approaches are compared. After correcting for publication bias and shrinkage of effect sizes in replication studies, simulations show that (i) a shift towards statistical approaches which explicitly incorporate the minimum clinically important difference reduces the false discovery rate of frequentist approaches and (ii) a shift towards Bayesian statistical decision criteria can improve the reliability of preclinical animal research by reducing the number of false-positive findings. It is shown that these benefits hold while keeping the number of experimental units low which are required for a confirmatory follow-up study. Results show that Bayesian statistical decision criteria can help in improving the reliability of preclinical animal research and should be considered more frequently in practice. [ABSTRACT FROM AUTHOR]

Amid the growing emphasis on the environmental and health benefits of walking, enhanced network connectivity emerges as a significant determinant in promoting pedestrian activity, as proven by scholars statistically and theoretically. Recent academic endeavors have introduced metrics such as pedestrian route directness (PRD) to measure connectivity, which maps the directness of routes between critical origins and destinations of the urban layout. However, there have been concerns from scholars about the limitations of PRD in theory, especially its sensitivity to larger radii of analysis. Addressing this theoretical inconsistency, this paper employs Pearson's correlation and linear regression analysis to explore the correlation between the radius of analysis and variance of results, the significance of this correlation for subsequent research, and the geographical context's influence on metric selection. The findings have revealed an r-value of −0.82, suggesting a strong and negative correlation. Moreover, the p-value of 0.0003 demonstrates the significance of the correlation and the rejection of the null hypothesis. These results bridge the gap between theoretical discussions and empirical analysis, revealing that, as the analysis radius expands, the sensitivity of results diminishes. The findings of this study hold significant implications for policy development and regulation of PRD, offering crucial insights that particularly advance the field of street connectivity. [ABSTRACT FROM AUTHOR]

Paleomagnetic statistical inference is underpinned by a family of parametric null hypothesis tests. In many cases, however, paleomagnetic data do not meet the distributional assumptions of these tests, which can lead to spurious inferences. Earlier studies have proposed the bootstrap as a nonparametric alternative for paleomagnetic analysis, which can be applied even when the distributional form of the data is unknown. Key among these approaches is the bootstrap test for a common mean direction, which relies on assessment of the overlap of estimated confidence regions. In its current form, the bootstrap test for a common mean paleomagnetic direction does not consider a null hypothesis and can yield outcomes that cannot be interpreted in terms of a statistical significance level. To resolve these issues, we use recent advances to place such bootstrap tests within a null hypothesis significance testing framework, and unify them with the existing family of paleomagnetic statistical tests. Furthermore, using numerical experiments we demonstrate the applicability of such a nonparametric approach to moderately sized paleomagnetic data sets typical of modern and legacy studies. Finally, we demonstrate how a confidence region can be estimated for the common mean of two sets of directions and how known directions, such as the expected field produced by a geocentric axial dipole, can be compared to that mean. Plain Language Summary: Statistical inference is the process of using observations to infer the properties of an unobserved population. An example of this process in paleomagnetism (the study of Earth's ancient magnetic field based on information preserved in geological materials) is a test of whether two sets of field directions have a common mean. Traditional statistical methods used to address this question make assumptions about the properties of the observations, which are known to be invalid in many cases. An alternative approach is so‐called bootstrapping, which relaxes assumptions about the observations and can then be applied more widely. In this study, we show how bootstrapping can be applied to paleomagnetic observations within the interpretational structure of existing paleomagnetic statistics, thus, providing a consistent inference framework. Furthermore, using numerical experiments we investigate how many paleomagnetic observations are required for bootstrapping to yield reliable results. Key Points: Advances in bootstrapping directional data are consideredThe number of directions needed for bootstrap‐based inference is assessedA bootstrap common mean direction test (CMDT) for paleomagnetic directions is presented [ABSTRACT FROM AUTHOR]

Null hypothesis statistical testing (NHST) in its present form continues to be the predominant method of quantitative research. Bayesian methods, which emphasize the probability of a hypothesis given the data, may be growing in popularity due to the advent of statistical software employing these methods. Rather than discarding the present methods of NHST, the purpose of this article is to enhance understanding of NHST for counseling researchers and provide an augmentation to presenting results that address the known limitations of counseling research and other social science research. Such recommendations include summarizing previous recommendations of employing visuals to explain data, incorporating effect size confidence intervals, and utilizing the Bayes factor bound to supplement p-values and explain the probabilities related to acceptance or rejection of the null hypothesis. [ABSTRACT FROM AUTHOR]

Hydraulic fracturing (HF) can trigger induced seismicity, but documented occurrences tend to be localized compared with the regional extent of industry operations. Factors that determine intrinsic geological susceptibility of a given region to induced seismicity remain incompletely understood. To address this uncertainty, we have developed a stochastic modeling approach to enable statistical testing of hypotheses regarding the distribution of induced seismicity. For reference, we adopted a null hypothesis that HF-induced seismic events are randomly associated with HF wells. Realizations of synthetic induced-seismicity catalogs are generated based on the Gutenberg–Richter relationship for magnitudes and explicit assumed spatial relationship(s) between HF wells and other known features, such as mapped structural corridors. Uncertainties in observed event locations and magnitudes are also considered. Based on 1000 independent realizations for each test scenario, normalized correlation coefficients, Bayesian information criteria and other statistical measures are used to quantify the similarity of synthetic catalogs to the observed seismicity distribution. We applied this approach to induced seismicity associated with HF operations within the Montney Formation, in western Canada. Three hypotheses were tested, each showing a statistically significant improvement over the null hypothesis. A previous machine-learning-based model for Seismogenic Activation Potential (SAP) showed the highest correlation between observed and synthetically generated seismicity catalogs. Our method has been developed using cloud-based computing and is easily adapted to other regions and data types. [ABSTRACT FROM AUTHOR]

The effect of an independent variable on random slopes in growth modeling with latent variables is conventionally used to examine predictors of change over the course of a study. This article demonstrates that the same effect of a covariate on growth can be obtained by using final status centering for parameterization and regressing the random intercepts (or the intercept factor scores) on both the independent variable and a baseline covariate–the framework used to study change with classical regression analysis. Examples are provided that illustrate the application of an intercept-focused approach to obtain effect sizes–the unstandardized regression coefficient, the standardized regression coefficient, squared semi-partial correlation, and Cohen's f2–that estimate the same parameters as respective effect sizes from a classical regression analysis. Moreover, statistical power to detect the effect of the predictor on growth was greater when using random intercepts than the conventionally used random slopes. [ABSTRACT FROM AUTHOR]

The standard practice in discussing results has shifted from coefficients to substantive quantities of interest. Hypothesis testing nowadays entails computing substantive effects with the 95% confidence interval (CI) for alternative scenarios and comparing them. Absent an evaluation of the difference in estimates, current practice often cannot provide a definitive answer. When CIs overlap, the estimates may or may not be statistically different. This ambiguity invites mistakes, as analysts turn to ill-advised conjectures to infer whether estimates are distinct. My literature survey indicates this is a widespread problem, with more than half of the articles not providing the evidence required to assess significance of differences. One practical solution is to report instead significance of differences intervals (SDIs), which can be used for direct comparisons. I expand the SDI method to accommodate unpaired sample data, asymmetric distributions, and for substantive significance differences larger than zero. I also provide an easy-to-use software to compute SDIs. [ABSTRACT FROM AUTHOR]

Comparative psychology experiments typically test a null statistical hypothesis against an alternative. Coupled with Morgan’s canon, this is often taken to imply that the model positing the simpler psychological capacity should be treated as a ‘default’ that must be ruled out before any other model can be accepted. It has been posited that this practice neglects evidence. I argue that the problem is deeper, including the way it structures the evaluation of evidence that is considered; it frames model choice around the acceptance or rejection of a default. I oppose this default framing in all its forms and develop an evidentialist alternative. Default framing fails to respect the difference between experimental statistical hypotheses and substantive hypotheses such as models. It is not actually supported by the use of null hypothesis significance testing (which I retain), it distorts the weighting of evidence, and it systematically biases practice. [ABSTRACT FROM AUTHOR]

In this paper, we address the project schedule control problem under an uncertain environment. We propose a new method to set the tolerance limits based on the Earned Value Management/Earned Schedule (EVM/ES) schedule performance metrics. These tolerance limits can help a project manager to identify whether the schedule deviations from the baseline schedule are within the possible deviations derived from the expected variability of the project or if corrective actions must be taken to get the project back on track. We view the project control problem as a statistical hypothesis test with the null hypothesis being that the project progress is out of control. First, a simulation is performed to generate two types of empirical conditional distributions of the monitored schedule indicator. Afterwards, an algorithm that uses the derived conditional distributions as inputs is proposed to optimise the tolerance limits. An extensive computational experiment is carried out to assess the performance of the proposed approach. Additionally, sensitivity experiments are conducted to analyse four underlying factors that may influence the power of the proposed method. Experimental results show that our approach can keep the first type error under the required level ( α = 0.05) in any situation, meanwhile reducing the second type error significantly compared with three other methods in the literature. [ABSTRACT FROM AUTHOR]

We propose a kernel density based estimation for nonlinear regression models when neither the response variable nor the covariates can be directly observed, but are measured with multiplicative distortion measurement errors. For the hypothesis testing of parametric components, restricted estimators under the null hypothesis and test statistics are proposed. The asymptotic properties for the estimators and test statistics are established. A bootstrap procedure is proposed to calculate critical values. Lastly, we make some comparisons between the proposed estimator and nonlinear least squares estimator through the simulation. Our simulation studies indicate that the kernel density based estimator provides a better alternative under some settings, particularly for multimodal distribution of the error term. [ABSTRACT FROM AUTHOR]

Significance tests play a very important role in the scientific community, and the biomedical research community is not an exception. This is due, on the one hand, to the widespread use of the test in scientific methodology and the corresponding frequency of its application in research, and, on the other hand, to the general misinterpretation of the results obtained using this method. Misunderstanding of significance testing in academia and erroneous conclusions in research, regardless of the scientific field, are at the root of the distrust of this statistical method. This article aims to give insight into the relevance of this kind of method in the biomedical field and find a theoretical explanation for this phenomenon, and subsequently regulate the correct interpretation of the null hypothesis significance test (NHST), as well as consider alternative statistical methods. In addition, some relevant empirical studies from a geographical and multidisciplinary perspective will be presented to determine the real extent of misspecification at the academic level. In this way, both practical and theoretical arguments will be applied to address the problem of NHST at multiple levels. [ABSTRACT FROM AUTHOR]

Nondeterministic measurements are common in real-world scenarios: the performance of a stochastic optimization algorithm or the total reward of a reinforcement learning agent in a chaotic environment are just two examples in which unpredictable outcomes are common. These measures can be modeled as random variables and compared among each other via their expected values or more sophisticated tools such as null hypothesis statistical tests. In this article, we propose an alternative framework to visually compare two samples according to their estimated cumulative distribution functions. First, we introduce a dominance measure for two random variables that quantifies the proportion in which the cumulative distribution function of one of the random variables stochastically dominates the other one. Then, we present a graphical method that decomposes in quantiles (i) the proposed dominance measure and (ii) the probability that one of the random variables takes lower values than the other. With illustrative purposes, we reevaluate the experimentation of an already published work with the proposed methodology and we show that additional conclusions—missed by the rest of the methods—can be inferred. Additionally, the software package RVCompare was created as a convenient way of applying and experimenting with the proposed framework. [ABSTRACT FROM AUTHOR]

In classical statistics, there is a close link between null hypothesis significance testing (NHST) and parameter estimation via confidence intervals. However, for the Bayesian counterpart, a link between null hypothesis Bayesian testing (NHBT) and Bayesian estimation via a posterior distribution is less straightforward, but does exist, and has recently been reiterated by Rouder, Haaf, and Vandekerckhove (2018). It hinges on a combination of a point mass probability and a probability density function as prior (denoted as the spike-and-slab prior). In the present paper, it is first carefully explained how the spike-and-slab prior is defined, and how results can be derived for which proofs were not given in Rouder, Haaf, and Vandekerckhove (2018). Next, it is shown that this spike-and-slab prior can be approximated by a pure probability density function with a rectangular peak around the center towering highly above the remainder of the density function. Finally, we will indicate how this 'hill-and-chimney' prior may in turn be approximated by fully continuous priors. In this way, it is shown that NHBT results can be approximated well by results from estimation using a strongly peaked prior, and it is noted that the estimation itself offers more than merely the posterior odds on which NHBT is based. Thus, it complies with the strong APA requirement of not just mentioning testing results but also offering effect size information. It also offers a transparent perspective on the NHBT approach employing a prior with a strong peak around the chosen point null hypothesis value. [ABSTRACT FROM AUTHOR]

Why you should read this article: • To learn the rationale for complementing or replacing a P-value with a Bayesian inference • To understand the basic concepts and compelling statistical and practical benefits of Bayesian analysis • To learn about the two Bayesian approaches increasingly advocated as alternatives or complements to commonly applied P-value statistics • To identify user-friendly software available for Bayesian analysis Background: The statistical shortcomings of null hypothesis significance testing (NHST) are well documented, yet it continues to be the default paradigm in quantitative healthcare research. This is due partly to unfamiliarity with Bayesian statistics. Aim: To highlight some of the theoretical and practical benefits of using Bayesian analysis. Discussion: A growing body of literature demonstrates that Bayesian analysis offers statistical and practical benefits that are unavailable to researchers who rely solely on NHST. Bayesian analysis uses prior information in the inference process. It tests a hypothesis and yields the probability of that hypothesis, conditional on the observed data; in contrast, NHST checks observed data – and more extreme unobserved data – against a hypothesis and yields the long-term probability of the data based on repeated hypothetical experiments. Bayesian analysis provides quantification of the evidence for the null and alternative hypothesis, whereas NHST does not provide evidence for the null hypothesis. Bayesian analysis allows for multiplicity of testing without corrections, whereas NHST multiplicity requires corrections. Finally, it allows sequential data collection with variable stopping, whereas NHST sequential designs require specialised statistical approaches. Conclusion: The Bayesian approach provides statistical, practical and ethical advantages over NHST. Implications for practice: The quantification of uncertainty provided by Bayesian analysis – particularly Bayesian parameter estimation – should better inform evidence-based clinical decision-making. Bayesian analysis provides researchers with the freedom to analyse data in real time with optimal stopping when the data is persuasive and continuing when data is weak, thereby ensuring better use of the researcher's time and resources [ABSTRACT FROM AUTHOR]

Given the failure history for K≥2$$ K\ge 2 $$ independent and identical repairable systems, a nonparametric procedure is presented to test the null hypothesis of minimal repair (MR) against the alternative of imperfect repair. The main idea is that, under non‐harmful (harmful) repair, systems that failed later (earlier) are more reliable than those that have failed (later). This fact allows one, at any moment in time, to rank the systems from more to less reliable and, hence, to define a vector that counts how many times the system ranked r$$ r $$ failed. When all the systems are time‐truncated at the same time T$$ T $$, it is shown that, under the null hypothesis of MR, the vector of counts follows a multinomial distribution with class probabilities pr=K−1$$ {p}_r={K}^{-1} $$ for r=1,...,K$$ r=1,\dots, K $$. The test proceeds by computing a chi‐bar squared test statistic similar to the one used to test one‐sided alternatives in the multinomial setup, which allows us to compute p$$ p $$‐values using either asymptotic theory or a straightforward Monte Carlo simulation using the null multinomial distribution. Extension to the case of different truncation times is also discussed. The procedure is applied to two real datasets regarding equipment used in the mining industry. [ABSTRACT FROM AUTHOR]

How statistically non-significant results are reported and interpreted following null hypothesis significance testing is often criticized. This issue is important for animal cognition research because studies in the field are often underpowered to detect theoretically meaningful effect sizes, i.e., often produce non-significant p-values even when the null hypothesis is incorrect. Thus, we manually extracted and classified how researchers report and interpret non-significant p-values and examined the pvalue distribution of these non-significant results across published articles in animal cognition and related fields. We found a large amount of heterogeneity in how researchers report statistically non-significant p-values in the result sections of articles, and how they interpret them in the titles and abstracts. Reporting of the non-significant results as ‘‘No Effect’’ was common in the titles (84%), abstracts (64%), and results sections (41%) of papers, whereas reporting of the results as ‘‘Non-Significant’’ was less common in the titles (0%) and abstracts (26%), but was present in the results (52%). Discussions of effect sizes were rare (<5% of articles). A p-value distribution analysis was consistent with research being performed with low power of statistical tests to detect effect sizes of interest. These findings suggest that researchers in animal cognition should pay close attention to the evidence used to support claims of absence of effects in the literature, and—in their own work—report statistically non-significant results clearly and formally correct, as well as use more formal methods of assessing evidence against theoretical predictions. [ABSTRACT FROM AUTHOR]

In particular, the data allow us to calculate the I p i -value, which is defined as " I the probability under a specified statistical model that a statistical summary of the data would be equal to or more extreme than its observed value i " [[1]], and practically reflects the degree of data compatibility with the null hypothesis. (2) " I p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. i " They can only tell us how consistent the data are with the null hypothesis. Conventionally, if the I p i -value is lower than the 0.05 significance level, we reject the null hypothesis and accept the alternative hypothesis. At the beginning of our research training, we learned about hypothesis testing, I p i -values, and statistical inference. [Extracted from the article]

Much of the blame for failed attempts to replicate reports of scientific findings has been placed on ubiquitous and persistent misinterpretations of the p value. An increasingly popular solution is to transform a two-sided p value to a lower bound on a Bayes factor. Another solution is to interpret a one-sided p value as an approximate posterior probability. Combining the two solutions results in confidence intervals that are calibrated by an estimate of the posterior probability that the null hypothesis is true. The combination also provides a point estimate that is covered by the calibrated confidence interval at every level of confidence. Finally, the combination of solutions generates a two-sided p value that is calibrated by the estimate of the posterior probability of the null hypothesis. In the special case of a 50% prior probability of the null hypothesis and a simple lower bound on the Bayes factor, the calibrated two-sided p value is about (1 – abs(2.7 p ln p)) p + 2 abs(2.7 p ln p) for small p. The calibrations of confidence intervals, point estimates, and p values are proposed in an empirical Bayes framework without requiring multiple comparisons. [ABSTRACT FROM AUTHOR]

The full Bayesian significance test (FBST) for precise hypotheses is a Bayesian alternative to the traditional significance tests based on p-values. The FBST is characterized by the e-value as an evidence index in favor of the null hypothesis (H). An important practical issue for the implementation of the FBST is to establish how small the evidence against H must be in order to decide for its rejection. In this work, we present a method to find a cutoff value for the e-value in the FBST by minimizing the linear combination of the averaged type-I and type-II error probabilities for a given sample size and also for a given dimensionality of the parameter space. Furthermore, we compare our methodology with the results obtained from the test with adaptive significance level, which presents the capital-P P-value as a decision-making evidence measure. For this purpose, the scenario of linear regression models with unknown variance under the Bayesian approach is considered. [ABSTRACT FROM AUTHOR]

Many human disease conditions need to be measured by ordinal phenotypes, so analysis of ordinal phenotypes is valuable in genome-wide association studies (GWAS). However, existing association methods for dichotomous or quantitative phenotypes are not appropriate to ordinal phenotypes. Therefore, based on an aggregated Cauchy association test, we propose a fast and efficient association method to test the association between genetic variants and an ordinal phenotype. To enrich association signals of rare variants, we first use the burden method to aggregate rare variants. Then we respectively test the significance of the aggregated rare variants and other common variants. Finally, the combination of transformed variant-level P values is taken as test statistic, that approximately follows Cauchy distribution under the null hypothesis. Extensive simulation studies and analysis of GAW19 show that our proposed method is powerful and computationally fast as a gene-based method. Especially, in the presence of an extremely low proportion of causal variants in a gene, our method has better performance. [ABSTRACT FROM AUTHOR]

Hypothesis testing is an essential statistical method in experimental psychology and the cognitive sciences. The problems of traditional null hypothesis significance testing (NHST) have been discussed widely, and among the proposed solutions to the replication problems caused by the inappropriate use of significance tests and p-values is a shift toward Bayesian data analysis. However, Bayesian hypothesis testing is concerned with various posterior indices for significance and the size of an effect. This complicates Bayesian hypothesis testing in practice, as the availability of multiple Bayesian alternatives to the traditional p-value causes confusion which one to select and why. In this paper, various Bayesian posterior indices which have been proposed in the literature are compared and their benefits and limitations are discussed. The comparison shows that conceptually not all proposed Bayesian alternatives to NHST and p-values are beneficial, and the usefulness of some indices strongly depends on the study design and research goal. However, the comparison also reveals that there exist at least two candidates among the available Bayesian posterior indices which have appealing theoretical properties and are widely underused in the cognitive sciences. [ABSTRACT FROM AUTHOR]

In different scientific areas, empirical studies are typically carried out by statistical null hypothesis tests. Despite the long tradition of applications, misinterpretations and misuses of the concept have led to a substantial confidence crisis in its inferential quality. One of the discussed issues is the significance-relevance discrepancy of the results of standardly applied zero-effect null hypothesis tests. This means that statistically significant test results do not automatically also have to be of scientific relevance in the specific research context. Therefore, this article is aimed at practitioners of empirical research who might want to include the aspect of practical relevance in their statistical conclusions. Different approaches to include this aspect in the inferential process are discussed with an example from the field of educational research. [ABSTRACT FROM AUTHOR]

Diagnostic screening models for the interpretation of null hypothesis significance test (NHST) results have been influential in highlighting the effect of selective publication on the reproducibility of the published literature, leading to John Ioannidis' much-cited claim that most published research findings are false. These models, however, are typically based on the assumption that hypotheses are dichotomously true or false, without considering that effect sizes for different hypotheses are not the same. To address this limitation, we develop a simulation model that overcomes this by modeling effect sizes explicitly using different continuous distributions, while retaining other aspects of previous models such as publication bias and the pursuit of statistical significance. Our results show that the combination of selective publication, bias, low statistical power and unlikely hypotheses consistently leads to high proportions of false positives, irrespective of the effect size distribution assumed. Using continuous effect sizes also allows us to evaluate the degree of effect size overestimation and prevalence of estimates with the wrong sign in the literature, showing that the same factors that drive false-positive results also lead to errors in estimating effect size direction and magnitude. Nevertheless, the relative influence of these factors on different metrics varies depending on the distribution assumed for effect sizes. The model is made available as an R ShinyApp interface, allowing one to explore features of the literature in various scenarios. [ABSTRACT FROM AUTHOR]

Multivariate isotonic regression theory plays an important role in the field of testing statistical hypotheses under order restriction for several vector valued parameters and this problem may arise in several situations. Sasabuchi et al. derived a likelihood ratio type test for homogeneity of order restricted mean vectors under a multivariate normality assumption with unknown and common covariance matrices; however, its null distribution depends on the unknown covariance matrix. In order to overcome this difficulty, in the present paper we propose a test that conditions on the sufficient statistic for the covariance matrix. We remove the dependence of the null distribution of test statistic proposed by Sasabuchi et al. on the unknown value of covariance matrix and present a condition on a sufficient statistic for covariance matrix under the null hypothesis. A method based on Markov Chain Monte Carlo sampling is used to calculate the p - value of the test. We evaluate the operating characteristics of the proposed test through simulation studies. Our extensive simulation studies demonstrate that the power of this new test is generally higher than the test proposed by Sasabuchi et al. and also by Sasabuchi. A real data example is presented. [ABSTRACT FROM AUTHOR]

This paper deals with a new Bayesian approach to the one-sample test for proportion. More specifically, let x = (x 1 , ... , x n) be an independent random sample of size n from a Bernoulli distribution with an unknown parameter θ. For a fixed value θ 0 , the goal is to test the null hypothesis H 0 : θ = θ 0 against all possible alternatives. The proposed approach is based on using the well-known formula of the Kullback–Leibler divergence between two binomial distributions chosen in a certain way. Then, the difference of the distance from a priori to a posteriori is compared through the relative belief ratio (a measure of evidence). Some theoretical properties of the method are developed. Examples and simulation results are included. [ABSTRACT FROM AUTHOR]

Editor's note: This is the 16th article in a series on clinical research by nurses. The series is designed to be used as a resource for nurses to understand the concepts and principles essential to research. Each column will present the concepts that underpin evidence-based practice—from research design to data interpretation. To see all the articles in the series, go to https://links.lww.com/AJN/A204. [ABSTRACT FROM AUTHOR]

Background: Null Hypothesis Significance Testing (NHST) has been well criticised over the years yet remains a pillar of statistical inference. Although NHST is well described in terms of statistical models, most textbooks for non-statisticians present the null and alternative hypotheses (H0 and HA, respectively) in terms of differences between groups such as (μ1 = μ2) and (μ1 ≠ μ2) and HA is often stated to be the research hypothesis. Here we use propositional calculus to analyse the internal logic of NHST when couched in this popular terminology. The testable H0 is determined by analysing the scope and limits of the P-value and the test statistic's probability distribution curve. Results: We propose a minimum axiom set NHST in which it is taken as axiomatic that H0 is rejected if P-value< α. Using the common scenario of the comparison of the means of two sample groups as an example, the testable H0 is {(μ1 = μ2) and [( x ¯ 1 ≠ x ¯ 2) due to chance alone]}. The H0 and HA pair should be exhaustive to avoid false dichotomies. This entails that HA is ¬{(μ1 = μ2) and [( x ¯ 1 ≠ x ¯ 2) due to chance alone]}, rather than the research hypothesis (HT). To see the relationship between HA and HT, HA can be rewritten as the disjunction HA: ({(μ1 = μ2) ∧ [( x ¯ 1 ≠ x ¯ 2) not due to chance alone]} ∨ {(μ1 ≠ μ2) ∧ [ (x ¯ 1 ≠ x ¯ 2) not due to (μ1 ≠ μ2) alone]} ∨ {(μ1 ≠ μ2) ∧ [( x ¯ 1≠ x ¯ 2) due to (μ1 ≠ μ2) alone]}). This reveals that HT (the last disjunct in bold) is just one possibility within HA. It is only by adding premises to NHST that HT or other conclusions can be reached. Conclusions: Using this popular terminology for NHST, analysis shows that the definitions of H0 and HA differ from those found in textbooks. In this framework, achieving a statistically significant result only justifies the broad conclusion that the results are not due to chance alone, not that the research hypothesis is true. More transparency is needed concerning the premises added to NHST to rig particular conclusions such as HT. There are also ramifications for the interpretation of Type I and II errors, as well as power, which do not specifically refer to HT as claimed by texts. [ABSTRACT FROM AUTHOR]

The internal validity of observational study is often subject to debate. In this study, we define the counterfactuals as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST). We propose the probability of a robust inference for internal validity, that is, the PIV, as a robustness index of causal inference. Formally, the PIV is the probability of rejecting the null hypothesis again based on both the observed sample and the counterfactuals, provided the same null hypothesis has already been rejected based on the observed sample. Under either frequentist or Bayesian framework, one can bound the PIV of an inference based on his bounded belief about the counterfactuals, which is often needed when the unconfoundedness assumption is dubious. The PIV is equivalent to statistical power when the NHST is thought to be based on both the observed sample and the counterfactuals. We summarize the process of evaluating internal validity with the PIV into a six-step procedure and illustrate it with an empirical example. [ABSTRACT FROM AUTHOR]