Search

$$5\times 2$$ cross-validated F-test based on independent five replications of 2-fold cross-validation is recommended in choosing between two classification learning algorithms. However, the reusing of the same data in a $$5\times 2$$ cross-validation causes the real degree of freedom (DOF) of the test to be lower than the F(10, 5) distribution given by (Neural Comput 11:1885-1892, [1]). This easily leads the test to suffer from high type I and type II errors. Random partitions for $$5\times 2$$ cross-validation result in difficulty in analyzing the DOF for the test. In particular, Wang et al. (Neural Comput 26(1):208-235, [2]) proposed a new blocked $$3 \times 2$$ cross-validation, that considered the correlation between any two 2-fold cross-validations. Based on this, a calibrated balanced $$5\times 2$$ cross-validated F-test following F(7, 5) distribution is put forward in this study by calibrating the DOF for the F(10, 5) distribution. Simulated and real data studies demonstrate that the calibrated balanced $$5\times 2$$ cross-validated F-test has lower type I and type II errors than the $$5\times 2$$ cross-validated F-test following F(10, 5) in most cases. [ABSTRACT FROM AUTHOR]

If past treatment assignments are unmasked, selection bias may arise even in randomized controlled trials. The impact of such bias can be measured by considering the type I error probability. In case of a normally distributed outcome, there already exists a model accounting for selection bias that permits calculating the corresponding type I error probabilities. To model selection bias for trials with a time-to-event outcome, we introduce a new biasing policy for exponentially distributed data. Using this biasing policy, we derive an exact formula to compute type I error probabilities whenever an F-test is performed and no observations are censored. Two exemplary settings, with and without random censoring, are considered in order to illustrate how our results can be applied to compare distinct randomization procedures with respect to their performance in the presence of selection bias. © 2017 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd. [ABSTRACT FROM AUTHOR]

In an editorial (this issue), I argued that Eklund, Nichols, and Knutsson’s ‘null data’ reflected resting-state/default network activity that inflated their false-positive rates. Commentaries on that paper were received by Nichols, Eklund, and Knutsson (this issue), Hopfinger (this issue), and Cunningham and Koscik (this issue). In this author response, I consider these commentaries. Many issues stemming from Nichols et al. are identified including: (1) Nichols et al. did not provide convincing arguments that resting-state fMRI data reflect null data. (2) Eklund et al. presented one-sample t-test results in the main body of their paper showing that their permutation method was acceptable, while their supplementary results showed that this method produced false-positive rates that were similar to other methods. (3) Eklund et al. used the same event protocol for all the participants, which artifactually inflated the one-samplet-test false-positive rates. (4) Atp < .001, using two-samplet-tests (which corrected for the flawed analysis), all the methods employed to correct for multiple comparisons had acceptable false-positive rates. (5) Eklund et al. contrasted resting-state periods, which produced many significant clusters of activity, while null data should arguably be associated with few, if any, significant activations. Eklund et al.’s entire set of results show that commonly employed methods to correct for multiple comparisons have acceptable false-positive rates. Following Hopfinger along with Cunningham and Koscik, it is also highlighted that rather than focusing on only type I error, type I error and type II error should be balanced in fMRI analysis. [ABSTRACT FROM PUBLISHER]

The purpose of our study was to examine the probability of observing false positives in nonsimulated data using the dual-criteria methods. We extracted data from published studies to produce a series of 16,927 datasets and then assessed the proportion of false positives for various phase lengths. Our results indicate that collecting at least three data points in the first phase (Phase A) and at least five data points in the second phase (Phase B) is generally sufficient to produce acceptable levels of false positives. [ABSTRACT FROM AUTHOR]

Introduction: Clinical development of new drugs is a long and costly process. There is a need to find solutions which can improve and shorten this process. By introducing flexibility in to the design of clinical trials, adaptive design contributes to this improvement and allows to reach drug development decisions in a quicker way. Areas covered: We review the main methodological approaches to adaptive trial design, introducing key statistical concepts. For each phase of the clinical development, different uses and implementations of adaptive trial (AD) design are presented and examples of recent clinical trials are given. The guidance documents issued by the US and European regulatory authorities are also presented. Expert commentary: Despite inevitable challenges, prospects of this rapidly evolving approach to drug development are important. Controlling the risk of type 1 error and the potential operational risks which may be associated with adaptive trial strategy is paramount in late phase studies. However, with new methodological work, these risks are now well controlled and adaptive trial design will certainly shape the future of drug development. [ABSTRACT FROM AUTHOR]

In this article, the general test statistic introduced by Alizadeh Noughabi and Balakrishnan [Goodness of fit using a new estimate of Kullback-Leibler information based on Type II censored data. IEEE Trans Reliab. 2015;64:627–635.] is applied for testing goodness of fit of lifetime distributions based on Type II censored data. The test statistic is constructed based on an estimate of Kullback–Leibler (KL) information. We investigate the properties of the proposed test statistic such as the test statistic is nonnegative, just like KL information. We apply this test statistic to following distributions: Exponential, Weibull, Log-normal and Pareto. The critical values and Type I error of the proposed tests are obtained. It is shown that the proposed tests have an excellent Type I error and hence can be used confidently in practice. Then, by Monte Carlo simulations, the power values of the proposed tests are computed against several alternatives and compared with those of the existing tests. Finally, some real-world reliability data are used for illustrative purpose. [ABSTRACT FROM PUBLISHER]

This paper investigates the impact of the number of studies on meta-analysis and meta-regression within the random-effects model framework. It is frequently neglected that inference in random-effects models requires a substantial number of studies included in meta-analysis to guarantee reliable conclusions. Several authors warn about the risk of inaccurate results of the traditional DerSimonian and Laird approach especially in the common case of meta-analysis involving a limited number of studies. This paper presents a selection of likelihood and non-likelihood methods for inference in meta-analysis proposed to overcome the limitations of the DerSimonian and Laird procedure, with a focus on the effect of the number of studies. The applicability and the performance of the methods are investigated in terms of Type I error rates and empirical power to detect effects, according to scenarios of practical interest. Simulation studies and applications to real meta-analyses highlight that it is not possible to identify an approach uniformly superior to alternatives. The overall recommendation is to avoid the DerSimonian and Laird method when the number of meta-analysis studies is modest and prefer a more comprehensive procedure that compares alternative inferential approaches. R code for meta-analysis according to all of the inferential methods examined in the paper is provided. [ABSTRACT FROM AUTHOR]

To estimate or test the treatment effect in randomized clinical trials, it is important to adjust for the potential influence of covariates that are likely to affect the association between the treatment or control group and the response. If these covariates are known at the start of the trial, random assignment of the treatment within each stratum would be considered. On the other hand, if these covariates are not clear at the start of the trial, or if it is difficult to allocate the treatment within each stratum, completely randomized assignment of the treatment would be performed. In both sampling structures, the use of a stratified adjusted test is a useful way to evaluate the significance of the overall treatment effect by reducing the variance and/or bias of the result. If the trial has a binary endpoint, the Cochran and Mantel-Haenszel tests are generally used. These tests are constructed based on the assumption that the number of patients within a stratum is fixed. However, in practice, the stratum sizes are not fixed at the start of the trial in many situations, and are instead allowed to vary. Therefore, there is a risk that using these tests under such situations would result in an error in the estimated variation of the test statistics. To handle the problem, we propose new test statistics under both sampling structures based on multinomial distributions. Our proposed approach is based on the Cochran test, and the difference between the two tests tends to have similar values in the case of a large number of patients. When the total number of patients is small, our approach yields a more conservative result. Through simulation studies, we show that the new approach could correctly maintain the type I error better than the traditional approach. [ABSTRACT FROM AUTHOR]

In clinical trials it is not uncommon to face a multiple testing problem which can have an impact on both type I and type II error rates, leading to inappropriate interpretation of trial results. Multiplicity issues may need to be considered at the design, analysis and interpretation stages of a trial. The proportion of trial reports not adequately correcting for multiple testing remains substantial. The purpose of this article is to provide an introduction to multiple testing issues in clinical trials, and to reduce confusion around the need for multiplicity adjustments. We use a tutorial, question-and-answer approach to address the key issues of why, when and how to consider multiplicity adjustments in trials. We summarize the relevant circumstances under which multiplicity adjustments ought to be considered, as well as options for carrying out multiplicity adjustments in terms of trial design factors including Population, Intervention/Comparison, Outcome, Time frame and Analysis (PICOTA). Results are presented in an easy-to-use table and flow diagrams. Confusion about multiplicity issues can be reduced or avoided by considering the potential impact of multiplicity on type I and II errors and, if necessary pre-specifying statistical approaches to either avoid or adjust for multiplicity in the trial protocol or analysis plan. [ABSTRACT FROM AUTHOR]

Although many common uses of p -values for making statistical inferences in contemporary scientific research have been shown to be invalid, no one, to our knowledge, has adequately assessed the main original justification for their use, which is that they can help to control the Type I error rate (Neyman & Pearson, 1928, 1933). We address this issue head-on by asking a specific question: Across what domain, specifically, do we wish to control the Type I error rate? For example, do we wish to control it across all of science, across all of a specific discipline such as psychology, across a researcher's active lifetime, across a substantive research area, across an experiment, or across a set of hypotheses? In attempting to answer these questions, we show that each one leads to troubling dilemmas wherein controlling the Type I error rate turns out to be inconsistent with other scientific desiderata. This inconsistency implies that we must make a choice. In our view, the other scientific desiderata are much more valuable than controlling the Type I error rate and so it is the latter, rather than the former, with which we must dispense. But by doing so—that is, by eliminating the Type I error justification for computing and using p -values—there is even less reason to believe that p is useful for validly rejecting null hypotheses than previous critics have suggested. [ABSTRACT FROM AUTHOR]

Phylogenetically closely related species tend to be more similar to each other than to more distantly related ones, a pattern called phylogenetic signal. Appropriate tests to evaluate the association between phylogenetic relatedness and trait variation among species are employed in a myriad of eco-evolutionary studies. However, most tests available to date are only suitable for datasets describing continuous traits, and are most often applicable only for single trait analysis. The Mantel test is a useful method to measure phylogenetic signal for multiple (continuous, binary and/or categorical) traits. However, the classical Mantel test does not incorporate any evolutionary model (EM) in the analysis. Here, we describe a new analytical procedure, which incorporates explicitly an evolutionary model in the standard Mantel test (EM-Mantel). We run numerical simulations to evaluate its statistical properties, under different combinations of species pool size, trait type and number. Our results showed that EM-Mantel test has appropriate type I error and acceptable power, which increases with the strength of phylogenetic signal and with species pool size but depended on trait type. EM-Mantel test is a good alternative for measuring phylogenetic signal in binary and categorical traits and for datasets with multiple traits. [ABSTRACT FROM AUTHOR]

In recent decades, numerous methods have been developed for data mining of large drug safety databases, such as Food and Drug Administration's (FDA's) Adverse Event Reporting System, where data matrices are formed by drugs such as columns and adverse events as rows. Often, a large number of cells in these data matrices have zero cell counts and some of them are "true zeros" indicating that the drug-adverse event pairs cannot occur, and these zero counts are distinguished from the other zero counts that are modeled zero counts and simply indicate that the drug-adverse event pairs have not occurred yet or have not been reported yet. In this paper, a zero-inflated Poisson model based likelihood ratio test method is proposed to identify drug-adverse event pairs that have disproportionately high reporting rates, which are also called signals. The maximum likelihood estimates of the model parameters of zero-inflated Poisson model based likelihood ratio test are obtained using the expectation and maximization algorithm. The zero-inflated Poisson model based likelihood ratio test is also modified to handle the stratified analyses for binary and categorical covariates (e.g. gender and age) in the data. The proposed zero-inflated Poisson model based likelihood ratio test method is shown to asymptotically control the type I error and false discovery rate, and its finite sample performance for signal detection is evaluated through a simulation study. The simulation results show that the zero-inflated Poisson model based likelihood ratio test method performs similar to Poisson model based likelihood ratio test method when the estimated percentage of true zeros in the database is small. Both the zero-inflated Poisson model based likelihood ratio test and likelihood ratio test methods are applied to six selected drugs, from the 2006 to 2011 Adverse Event Reporting System database, with varying percentages of observed zero-count cells. [ABSTRACT FROM AUTHOR]

We discuss group-sequential three-arm noninferiority clinical trial designs that include active and placebo controls for evaluating both assay sensitivity and noninferiority. We extend two existing approaches, the fixed margin and fraction approaches, into a group-sequential setting with two decision-making frameworks. We investigate the operating characteristics including power, Type I error rate, maximum, and expected sample sizes, as design factors vary. In addition, we discuss sample size recalculation and its impact on the power and Type I error rate via a simulation study. [ABSTRACT FROM PUBLISHER]

The effect of correlation among covariates on covariate selection was examined with linear and nonlinear mixed effect models. Demographic covariates were extracted from the National Health and Nutrition Examination Survey III database. Concentration-time profiles were Monte Carlo simulated where only one covariate affected apparent oral clearance (CL/F). A series of univariate covariate population pharmacokinetic models was fit to the data and compared with the reduced model without covariate. The 'best' covariate was identified using either the likelihood ratio test statistic or AIC. Weight and body surface area (calculated using Gehan and George equation, 1970) were highly correlated ( r = 0.98). Body surface area was often selected as a better covariate than weight, sometimes as high as 1 in 5 times, when weight was the covariate used in the data generating mechanism. In a second simulation, parent drug concentration and three metabolites were simulated from a thorough QT study and used as covariates in a series of univariate linear mixed effects models of ddQTc interval prolongation. The covariate with the largest significant LRT statistic was deemed the 'best' predictor. When the metabolite was formation-rate limited and only parent concentrations affected ddQTc intervals the metabolite was chosen as a better predictor as often as 1 in 5 times depending on the slope of the relationship between parent concentrations and ddQTc intervals. A correlated covariate can be chosen as being a better predictor than another covariate in a linear or nonlinear population analysis by sheer correlation These results explain why for the same drug different covariates may be identified in different analyses. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

In this study, we develop a test based on computational approach for the equality of variances of several normal populations. The proposed method is numerically compared with the existing methods. The numeric results demonstrate that the proposed method performs very well in terms of type I error rate and power of test. Furthermore we study the robustness of the tests by using simulation study when the underlying data are from t, exponential and uniform distributions. Finally we analyze a real dataset that motivated our study using the proposed test. [ABSTRACT FROM AUTHOR]

For the two-way factorial design in analysis of variance, the current article explicates and compares three methods for controlling the Type I error rate for all possible simple interaction contrasts following a statistically significant interaction, including a proposed modification to the Bonferroni procedure that increases the power of statistical tests for deconstructing interaction effects when they are of primary substantive interest. Results indicate the general superiority of the modified Bonferroni procedure over Scheffé and Roy-type procedures, where the Bonferroni and Scheffé procedures have been modified to accommodate the logical implications of a false omnibus interaction null hypothesis. An applied example is provided and considerations for applied researchers are offered. [ABSTRACT FROM AUTHOR]

In linear regression models, covariate-adjusted analysis is not expected to change the estimates of the treatment effect in the clinical trials with randomized treatment assignment but rather to increase the precision of the estimates. However, the covariate-adjusted treatment effect estimates are generally not equivalent to the unadjusted estimates in logistic regression analysis for binary clinical trial data. In this article, we report the results of a simulation study conducted to quantify the magnitude of difference between the estimands underlying the two estimators in logistic regression. The simulation results demonstrated that both unadjusted and adjusted analyses preserved Type I error at the nominal level. The covariate-adjusted analysis produced unbiased, larger treatment effect estimates, larger standard error, and increased power compared with the unadjusted analysis when the sample size was large. The unadjusted analysis resulted in biased estimates of treatment effect. Analysis results for five phase 3 diabetes trials of the same compound were consistent with the simulation findings. Therefore, covariate-adjusted analysis is recommended for evaluating binary outcomes in clinical data. [ABSTRACT FROM PUBLISHER]

The draft adaptive design guidance released by FDA in 2010 included references to adaptive study designs that were described as "less well-understood." At that time, there was relatively little regulatory experience with such designs, and their properties were felt to be insufficiently understood. In order to promote greater use of adaptive designs, especially those categorized as less wellunderstood, the Best Practice Subteam of the DIA Adaptive Designs Scientific Working Group (ADSWG) has worked on describing and characterizing these designs, identifying challenges associated with them and suggesting improvements to design or study conduct aspects that might make them more acceptable. This paper summarizes the work from the subteam. [ABSTRACT FROM AUTHOR]

One of the most important statistical analyses when designing animal and human studies is the calculation of the required sample size. In this review, we define central terms in the context of sample size determination, including mean, standard deviation, statistical hypothesis testing, type I/II error, power, direction of effect, effect size, expected attrition, corrected sample size, and allocation ratio. We also provide practical examples of sample size calculations for animal and human studies based on pilot studies, larger studies similar to the proposed study-or if no previous studies are available-estimated magnitudes of the effect size per Cohen and Sawilowsky., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2023 Zhang and Hartmann.)

While it is often argued that a p-value is a probability; see Wasserstein and Lazar, we argue that a p-value is not defined as a probability. A p-value is a bijection of the sufficient statistic for a given test which maps to the same scale as the Type I error probability. As such, the use of p-values in a test should be no more a source of controversy than the use of a sufficient statistic. It is demonstrated that there is, in fact, no ambiguity about what a p-value is, contrary to what has been claimed in recent public debates in the applied statistics community. We give a simple example to illustrate that rejecting the use of p-values in testing for a normal mean parameter is conceptually no different from rejecting the use of a sample mean. The p-value is innocent; the problem arises from its misuse and misinterpretation. The way that p-values have been informally defined and interpreted appears to have led to tremendous confusion and controversy regarding their place in statistical analysis. [ABSTRACT FROM AUTHOR]

Background Self-contained tests estimate and test the association between a phenotype and mean expression level in a gene set defined a priori. Many self-contained gene set analysis methods have been developed but the performance of these methods for phenotypes that are continuous rather than discrete and with multiple nuisance covariates has not been well studied. Here, I use Monte Carlo simulation to evaluate the performance of both novel and previously published (and readily available via R) methods for inferring effects of a continuous predictor on mean expression in the presence of nuisance covariates. The motivating data are a high-profile dataset which was used to show opposing effects of hedonic and eudaimonic well-being (or happiness) on the mean expression level of a set of genes that has been correlated with social adversity (the CTRA gene set). The original analysis of these data used a linear model (GLS) of fixed effects with correlated error to infer effects of Hedonia and Eudaimonia on mean CTRA expression. Methods The standardized effects of Hedonia and Eudaimonia on CTRA gene set expression estimated by GLS were compared to estimates using multivariate (OLS) linear models and generalized estimating equation (GEE) models. The OLS estimates were tested using O’Brien’s OLS test, Anderson’s permutation ${r}_{F}^{2}$ r F 2 -test, two permutation F-tests (including GlobalAncova), and a rotation z-test (Roast). The GEE estimates were tested using a Wald test with robust standard errors. The performance (Type I, II, S, and M errors) of all tests was investigated using a Monte Carlo simulation of data explicitly modeled on the re-analyzed dataset. Results GLS estimates are inconsistent between data sets, and, in each dataset, at least one coefficient is large and highly statistically significant. By contrast, effects estimated by OLS or GEE are very small, especially relative to the standard errors. Bootstrap and permutation GLS distributions suggest that the GLS results in downward biased standard errors and inflated coefficients. The Monte Carlo simulation of error rates shows highly inflated Type I error from the GLS test and slightly inflated Type I error from the GEE test. By contrast, Type I error for all OLS tests are at the nominal level. The permutation F-tests have ∼1.9X the power of the other OLS tests. This increased power comes at a cost of high sign error (∼10%) if tested on small effects. Discussion The apparently replicated pattern of well-being effects on gene expression is most parsimoniously explained as “correlated noise” due to the geometry of multiple regression. The GLS for fixed effects with correlated error, or any linear mixed model for estimating fixed effects in designs with many repeated measures or outcomes, should be used cautiously because of the inflated Type I and M error. By contrast, all OLS tests perform well, and the permutation F-tests have superior performance, including moderate power for very small effects.

Sensitivity and specificity analysis is commonly used for screening and diagnostic tests. The main issue researchers face is to determine the sufficient sample sizes that are related with screening and diagnostic studies. Although the formula for sample size calculation is available but concerning majority of the researchers are not mathematicians or statisticians, hence, sample size calculation might not be easy for them. This review paper provides sample size tables with regards to sensitivity and specificity analysis. These tables were derived from formulation of sensitivity and specificity test using Power Analysis and Sample Size (PASS) software based on desired type I error, power and effect size. The approaches on how to use the tables were also discussed.

There is increasing concern about the replicability of studies in psychology and cognitive neuroscience. Hidden data dredging (also called p-hacking) is a major contributor to this crisis because it substantially increases Type I error resulting in a much larger proportion of false positive findings than the usually expected 5%. In order to build better intuition to avoid, detect and criticise some typical problems, here I systematically illustrate the large impact of some easy to implement and so, perhaps frequent data dredging techniques on boosting false positive findings. I illustrate several forms of two special cases of data dredging. First, researchers may violate the data collection stopping rules of null hypothesis significance testing by repeatedly checking for statistical significance with various numbers of participants. Second, researchers may group participants post-hoc along potential but unplanned independent grouping variables. The first approach 'hacks' the number of participants in studies, the second approach ‘hacks’ the number of variables in the analysis. I demonstrate the high amount of false positive findings generated by these techniques with data from true null distributions. I also illustrate that it is extremely easy to introduce strong bias into data by very mild selection and re-testing. Similar, usually undocumented data dredging steps can easily lead to having 20-50%, or more false positives.

Hypothesis testing is an important activity of empirical research and evidence-based medicine. A well worked up hypothesis is half the answer to the research question. For this, both knowledge of the subject derived from extensive review of the literature and working knowledge of basic statistical concepts are desirable. The present paper discusses the methods of working up a good hypothesis and statistical concepts of hypothesis testing.

One important topic in clinical trials is to show that the effects of new and standard treatments are equivalent in terms of clinical relevance. In literature, many equivalence tests based on the maximal difference between two survival functions for the two treatments over the whole time axis have been proposed. However, since survival times can only be observed until the end of follow-up, an equivalence test should be based on a comparison only in the observed time-window dictated by the end of follow-up. In this article, under the class of log transformation model, we propose an asymptotical α -level equivalence test for the difference between two survival functions that only addresses equivalence until the end of follow-up. We demonstrate that the hypothesis of equivalence of two survival functions before the end of follow-up can be formulated as interval-based hypothesis testing which involves the treatment effect parameter. Simulation results indicate that when sample size is sufficiently large the proposed test controls the type I error effectively and performs well at detecting the equivalence. The proposed test is applied to a dataset from veteran's administration lung cancer trial.

Researchers with mediation hypotheses must consider which design to use: within-subject or between-subject? In this paper, I argue that three factors should influence design choice: validity, causality, and statistical power. Threats to validity include carry-over effects, participant awareness, measurement, and more. Causality is a core element of mediation, and the assumptions required for causal inference differ between the two designs. Between-subject designs require more restrictive no-confounder assumptions, but within-subject designs require the assumption of no carry-over effects. Statistical power should be higher in within-subject designs, but the degree and conditions of this advantage are unknown for mediation analysis. A Monte Carlo simulation compares designs under a broad range of sample sizes, effect sizes, and correlations among repeated measurements. The results show within-subject designs require about half the sample size of between-subject designs to detect indirect effects of the same size, but this difference can vary with population parameters. I provide an empirical example and R script for conducting power analysis for within-subject mediation analysis. Researchers interested in conducting mediation analysis should not select within-subject designs merely because of higher power, but they should also consider validity and causality in their decision, both of which can favor between-subject designs.

We consider the problem of testing for a difference in means between clusters of observations identified via k -means clustering. In this setting, classical hypothesis tests lead to an inflated Type I error rate. In recent work, Gao et al. (2022) considered a related problem in the context of hierarchical clustering. Unfortunately, their solution is highly-tailored to the context of hierarchical clustering, and thus cannot be applied in the setting of k -means clustering. In this paper, we propose a p-value that conditions on all of the intermediate clustering assignments in the k -means algorithm. We show that the p-value controls the selective Type I error for a test of the difference in means between a pair of clusters obtained using k -means clustering in finite samples, and can be efficiently computed. We apply our proposal on hand-written digits data and on single-cell RNA-sequencing data.

Scientific research often involves testing more than one hypothesis at a time, which can inflate the probability that a Type I error (false discovery) will occur. To prevent this Type I error inflation, adjustments can be made to the testing procedure that compensate for the number of tests. Yet many researchers believe that such adjustments are inherently unnecessary if the tests were “planned” (i.e., if the hypotheses were specified before the study began). This longstanding misconception continues to be perpetuated in textbooks and continues to be cited in journal articles to justify disregard for Type I error inflation. I critically evaluate this myth and examine its rationales and variations. To emphasize the myth’s prevalence and relevance in current research practice, I provide examples from popular textbooks and from recent literature. I also make recommendations for improving research practice and pedagogy regarding this problem and regarding multiple testing in general.

Recent simulation studies have pointed to the higher power of the test for the mediated effect versus the test for the total effect, even in the presence of a direct effect. This has motivated applied researchers to investigate mediation in settings where there is no evidence of a total effect. In this paper we provide analytical insight into the circumstances under which higher power of the test for the mediated effect versus the test for the total effect can be expected in the absence of a direct effect. We argue that the acclaimed power gain is somewhat deceptive and comes with a big price. On the basis of the results, we recommend that when the primary interest lies in mediation only, a significant test for the total effect should not be used as a prerequisite for the test for the indirect effect. However, because the test for the indirect effect is vulnerable to bias when common causes of mediator and outcome are not measured or not accounted for, it should be evaluated in a sensitivity analysis.

Questions Mechanisms of community assembly are increasingly explored by combining community and species trait data with null models. By investigating if the traits of co-existing species are more similar (trait convergence) or more dissimilar (trait divergence) than expected by chance, these tests relate observed patterns to different co-existence mechanisms. Do null models accurately detect trait convergence and divergence? Are different null models equally good at detecting these two opposing patterns? How important are the species pool and other constraints that are considered by different null models? Methods We applied ten common randomizations to communities that were simulated in a process-based model. Results Null models good at detecting biotic processes differed from those null models that revealed abiotic processes. In particular, limiting similarity (detected through divergence) was better detected by randomizations that release the link between species abundance and trait values, whereas environmental filtering (detected through convergence of an environmental response trait) was identified by randomizations that keep this link. In general, using species abundance data provided better results than using presence-absence data, particularly within given limited environmental conditions. Weaker competitor exclusion (detected through convergence of a competition-related trait) was only detected when no environmental filtering was acting on the simulated assembly, which points to difficulties in disentangling biotic and abiotic convergence in natural communities, especially when data are randomized across habitats. Conclusions Overall the results manifest the importance of the pool of species over which randomizations are applied; in particular whether randomizations are conducted across or within given habitats. Taken together, our findings show that different null models must be combined and applied to a carefully chosen pool of species and species abundance data to ensure that co-existence mechanisms can be properly assessed. We utilize the results to (1) discuss how different constraints implied in the different null models affect the outcomes of our tests, and (2) provide some basic recommendations on how to choose null models, given the data available and questions being asked. [ABSTRACT FROM AUTHOR]

When there are four or more treatments under comparison, the use of a crossover design with a complete set of treatment-receipt sequences in binary data is of limited use because of too many treatment-receipt sequences. Thus, we may consider use of a 4 × 4 Latin square to reduce the number of treatment-receipt sequences when comparing three experimental treatments with a control treatment. Under a distribution-free random effects logistic regression model, we develop simple procedures for testing non-equality between any of the three experimental treatments and the control treatment in a crossover trial with dichotomous responses. We further derive interval estimators in closed forms for the relative effect between treatments. To evaluate the performance of these test procedures and interval estimators, we employ Monte Carlo simulation. We use the data taken from a crossover trial using a 4 × 4 Latin-square design for studying four-treatments to illustrate the use of test procedures and interval estimators developed here. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

In drug development and drug licensing, it sometimes occurs that a new drug does not demonstrate effectiveness for the full study population, but there appears to be benefit in a relevant, pre-defined subgroup. This raises the question, how strong the evidence from such a subgroup is, and which confirmatory testing strategies are the most appropriate ones. Hence, we considered the type I error and the power of a subgroup result in a trial with non-significant overall results and of suitable replication strategies. In the case of a single trial, the inflation of the overall type I error is substantial and can be up to twice as large, especially in relatively small subgroups. This also increases to the risk of starting a replication trial that should not be done, if such a second trial is not already available. The overall type I error is almost controlled by using an appropriate replication strategy. This confirms the required cautious interpretation of promising subgroups, even in the case that overall trial results were perceived to be close to significance. [ABSTRACT FROM AUTHOR]

Sensitivity and specificity analysis is commonly used for screening and diagnostic tests. The main issue researchers face is to determine the sufficient sample sizes that are related with screening and diagnostic studies. Although the formula for sample size calculation is available but concerning majority of the researchers are not mathematicians or statisticians, hence, sample size calculation might not be easy for them. This review paper provides sample size tables with regards to sensitivity and specificity analysis. These tables were derived from formulation of sensitivity and specificity test using Power Analysis and Sample Size (PASS) software based on desired type I error, power and effect size. The approaches on how to use the tables were also discussed. [ABSTRACT FROM AUTHOR]

There is increasing concern about the replicability of studies in psychology and cognitive neuroscience. Hidden data dredging (also called p-hacking) is a major contributor to this crisis because it substantially increases Type I error resulting in a much larger proportion of false positive findings than the usually expected 5%. In order to build better intuition to avoid, detect and criticize some typical problems, here I systematically illustrate the large impact of some easy to implement and so, perhaps frequent data dredging techniques on boosting false positive findings. I illustrate several forms of two special cases of data dredging. First, researchers may violate the data collection stopping rules of null hypothesis significance testing by repeatedly checking for statistical significance with various numbers of participants. Second, researchers may group participants post hoc along potential but unplanned independent grouping variables. The first approach 'hacks' the number of participants in studies, the second approach 'hacks' the number of variables in the analysis. I demonstrate the high amount of false positive findings generated by these techniques with data from true null distributions. I also illustrate that it is extremely easy to introduce strong bias into data by very mild selection and re-testing. Similar, usually undocumented data dredging steps can easily lead to having 20-50%, or more false positives. [ABSTRACT FROM AUTHOR]

The concept of controlling familywise type I and type II errors at the same time is essentially an integrated process to deal with multiplicity issues in clinical trials. The process will select a multiple testing procedure (MTP) which controls the familywise type I error and calculate the per hypothesis sample size such that the “studywise power” is maintained at desired level. The power of a study can be defined in several ways and it depends on the objective. In this article, we provide general guidance on how to make the selection of MTPs and calculate sample size simultaneously. We introduce the concept of strong and weak control of the familywise type II error and generalized familywise type II error. We also proposed the novel Bonferroni+ and optimal Bonferroni+ procedures to allocate per hypothesis type II error. We demonstrated the value of the proposed work as it cannot be replaced by simple simulations. A real clinical trial is discussed throughout the article as an example. [ABSTRACT FROM AUTHOR]

Recent criticisms of social psychological research are considered in relation to an earlier crisis in social psychology. The current replication crisis is particularly severe because (1) psychologists are questioning the accuracy of findings rather than the meaning of findings, and (2) researchers are responding to real scientific failures, rather than hypothetical scientific failures. I present an expanded model of statistical decision making that can be used to help researchers draw more reliable conclusions. Based on the premise that drawing conclusions on relatively bad evidence is an error, Type III and IV errors are introduced as categories representing statistical decisions that align with reality, but do not follow from the available evidence. Treating these as errors helps researchers and evaluators of research to draw more reliable conclusions. From the perspective of this model I discuss procedures that researchers can use to not only produce more replicable results, but also conduct more powerful statistical tests. [ABSTRACT FROM AUTHOR]

The increase in incidence of obesity and chronic diseases and their health care costs have raised the importance of quality diet on the health policy agendas. The healthy eating index is an important measure for diet quality which consists of 12 components derived from ratios of dependent variables with distributions hard to specify, measurement errors and excessive zero observations difficult to model parametrically. Hypothesis testing involving data of such nature poses challenges because the widely used multiple comparison procedures such as Hotelling's T(2) test and Bonferroni correction may suffer from substantial loss of efficiency. We propose a marginal rank-based inverse normal transformation approach to normalizing the marginal distribution of the data before employing a multivariate test procedure. Extensive simulation was conducted to demonstrate the ability of the proposed approach to adequately control the type I error rate as well as increase the power of the test, with data particularly from non-symmetric or heavy-tailed distributions. The methods are exemplified with data from a dietary intervention study for type I diabetic children. Published 2016. This article is a U.S. Government work and is in the public domain in the USA. [ABSTRACT FROM AUTHOR]

A sequential design is proposed to test whether the accuracy of a binary diagnostic biomarker meets the minimal level of acceptance. The accuracy of a binary diagnostic biomarker is a linear combination of the marker's sensitivity and specificity. The objective of the sequential method is to minimize the maximum expected sample size under the null hypothesis that the marker's accuracy is below the minimal level of acceptance. The exact results of two-stage designs based on Youden's index and efficiency indicate that the maximum expected sample sizes are smaller than the sample sizes of the fixed designs. Exact methods are also developed for estimation, confidence interval and p-value concerning the proposed accuracy index upon termination of the sequential testing. Published 2016. This article is a U.S. Government work and is in the public domain in the USA. [ABSTRACT FROM AUTHOR]

The two most common approaches for analysing count data are to use a generalized linear model ( GLM), or transform data, and use a linear model ( LM). The latter has recently been advocated to more reliably maintain control of type I error rates in tests for no association, while seemingly losing little in power. We make three points on this issue., Point 1 - Choice of statistical model should primarily be made on the grounds of data properties. Choice of testing procedure should be considered and addressed as a separate issue, after model choice. If models with the appropriate data properties nonetheless have statistical problems such as type I error control ( i.e. type I error rate greatly exceeds the intended significance level), the best solution is to keep the model but fix the problems., Point 2 - When a test has problems with type I error control, it can usually be corrected, but this may require departure from software default approaches. In particular, resampling is a good solution for small samples that can be easy to implement., Point 3 -Tests based on models that better fit the data ( e.g. a negative binomial for overdispersed count data) tend to have better power properties and in some instances have considerably higher power., We illustrate these issues for a 2 × 2 experiment with a count response. This seemingly simple problem becomes hard when the experimental design is unbalanced, and software default procedures using LMs or GLMs can have difficulties, although in both cases the issues can be fixed., We conclude that, when GLMs are thought to fit count data well, and when any necessary steps are taken to correct type I error rates, they should be used rather than LMs. Nonetheless, standard LM tests are often robust and can have good type I error control, so there is an argument for their use for counts when diagnostics are difficult and statistical models are complex, although at some risk of loss of power and interpretability. [ABSTRACT FROM AUTHOR]

In Phase III clinical trials for life-threatening conditions, some serious but expected adverse events, such as early deaths or congestive heart failure, are often treated as the secondary or co-primary endpoint, and are closely monitored by the Data and Safety Monitoring Committee (DSMC). A naïve group sequential design (GSD) for such a study is to specify univariate statistical boundaries for the efficacy and safety endpoints separately, and then implement the two boundaries during the study, even though the two endpoints are typically correlated. One problem with this naïve design, which has been noted in the statistical literature, is the potential loss of power. In this article, we develop an analytical tool to evaluate this negative impact for trials with non-trivial safety event rates, particularly when the safety monitoring is informal. Using a bivariate binary power function for the GSD with a random-effect component to account for subjective decision- making in safety monitoring, we demonstrate how, under common conditions, the power loss in the naïve design can be substantial. This tool may be helpful to entities such as the DSMCs when they wish to deviate from the prespecified stopping boundaries based on safety measures. [ABSTRACT FROM AUTHOR]

In this article, we propose a biomarker informed add-arm design for unimodal response. The new design contributes to optimizing the procedure of dose-finding when a biomarker of the study primary endpoint exists and prior evidence indicates a unimodal dose-response relationship. Designs with up to seven active treatment arms were considered. We propose the statistical approach for the Type I error control and carry out extensive simulation studies for the power performance of the design. The proposed design is shown to outperform the corresponding biomarker informed two-stage winner design in power on an average. [ABSTRACT FROM AUTHOR]

The likelihood ratio test (LRT), to the independence between two sets of variables, allows to identify whether there is a dependency relationship between them. The aim of this study was to calculate the type I error and power of the LRT for determining independence between two sets of variables under multivariate normal distributions in scenarios consisting of combinations of 16 sample sizes; 40 combinations of the number of variables of the two groups; and nine degrees of correlation between the variables (for the power). The rate of type I error and power were calculate at 640 and 5,760 scenarios, respectively. A performance evaluation of the LRT was conducted by computer simulation by the Monte Carlo method, using 2,000 simulations in each scenario. When the number of variables was large (24), the TRV controlled the rate of type I errors and showed high power in sizes greater than 100 samples. For small sample sizes (25, 30 and 50), the test showed good performance because the number of variables did not exceed 12. [ABSTRACT FROM AUTHOR]

Unlabelled: The results of a clinical trial may be subject to random error because of the variability in the measured data, which arises purely by chance. There are two types of random error - type I error and type II error. In this study, type I and type II errors are explained, and the important concepts of statistical power and sample size estimation are discussed.Conclusion: The most important way of minimising random errors is to ensure adequate sample size; that is, a sufficient large number of patients should be recruited for the study. [ABSTRACT FROM AUTHOR]

Model testing is a central step of spatial point pattern analysis, which allows ecologists to judge if their data agree with ecological hypotheses. We present a simple and elegant solution of a challenging problem: the construction of a goodness-of-fit envelope test with prescribed significance level α. Our new Analytical Global Envelope ( AGE) test is not restricted to the narrow frame of complete spatial randomness testing and its envelopes can be determined by mathematical calculations. This allows us to investigate the influence of key settings of the AGE test on the width of the envelope strip. To circumvent some assumptions of the simulation-free AGE test we present a corresponding Simulation-Based Global Envelope ( SBGE) test. The envelope strip of the AGE and the SBGE test encircles the range of a summary function such as the pair correlation function under the null model, and it has the desired property that the null hypothesis can be rejected with significance level α if the empirical summary function wanders outside the envelopes. The AGE test can be applied under the mild conditions that the values of the summary functions under the null model are (approximately) normally distributed and are (approximately) independent for different distance bins r j. The SBGE test requires only the independence assumption. The width of the strip of the AGE envelopes scales for a broad range of point processes with 1/ n, where n is the number of points. This casts doubt about attempts of goodness-of-fit testing with low n (say <100). The AGE and SBGE test operate with wider envelope strips than the classical 'pointwise' test. Therefore, the pointwise test has to be considered as too liberal. Furthermore, we show that the width of the AGE/ SBGE strip increases approximately with ln( b), where b is the number of distance bins. For example, the AGE/ SBGE envelopes are for b = 20 more than 50% wider than the corresponding pointwise envelopes. Our study opens up new avenues to the test problem in point pattern statistics and the new AGE and SBGE tests can be widely applied in ecology to improve the practice in null model testing. [ABSTRACT FROM AUTHOR]

Generalized estimating equations (GEE) is a general statistical method to fit marginal models for longitudinal data in biomedical studies. The variance-covariance matrix of the regression parameter coefficients is usually estimated by a robust "sandwich" variance estimator, which does not perform satisfactorily when the sample size is small. To reduce the downward bias and improve the efficiency, several modified variance estimators have been proposed for bias-correction or efficiency improvement. In this paper, we provide a comprehensive review on recent developments of modified variance estimators and compare their small-sample performance theoretically and numerically through simulation and real data examples. In particular, Wald tests and t-tests based on different variance estimators are used for hypothesis testing, and the guideline on appropriate sample sizes for each estimator is provided for preserving type I error in general cases based on numerical results. Moreover, we develop a user-friendly R package "geesmv" incorporating all of these variance estimators for public usage in practice. [ABSTRACT FROM AUTHOR]