Search

Recently, Asparouhov and Muthén Structural Equation Modeling: A Multidisciplinary Journal, 28, 1–14, (2021a, 2021b) proposed a variant of the Wald test that uses Markov chain Monte Carlo machinery to generate a chi-square test statistic for frequentist inference. Because the test's composition does not rely on analytic expressions for sampling variation and covariation, it potentially provides a way to get honest significance tests in cases where the likelihood-based test statistic's assumptions break down (e.g., in small samples). The goal of this study is to use simulation to compare the new MCM Wald test to its maximum likelihood counterparts, with respect to both their type I error rate and power. Our simulation examined the test statistics across different levels of sample size, effect size, and degrees of freedom (test complexity). An additional goal was to assess the robustness of the MCMC Wald test with nonnormal data. The simulation results uniformly demonstrated that the MCMC Wald test was superior to the maximum likelihood test statistic, especially with small samples (e.g., sample sizes less than 150) and complex models (e.g., models with five or more predictors). This conclusion held for nonnormal data as well. Lastly, we provide a brief application to a real data example. [ABSTRACT FROM AUTHOR]

AbstractThe ratio of two Poisson means is commonly used in biological, epidemiological, and medical. In this article, we consider the problem of testing the ratio of two Poisson means and propose a valid and efficient test based on the inference model (IM). The simulation studies show that the type I error and power of the IM-based test are competitive or even better than the six recommended tests: The likelihood ratio, mid-p, logarithmic transformation, and three tests based on the method of variance estimates recovery (MOVER). For R<1, the Rao-MOVER and fiducial-MOVER tests are slightly liberal, the mid-p test is somewhat conservative. For R>1, the Rao-MOVER and fiducial-MOVER tests are slightly conservative. The likelihood ratio and logarithmic transformation tests cannot control the type I error well. Therefore, the IM test could be recommended for practical applications. A real numerical example is presented to illustrate the flexibility of the proposed test. [ABSTRACT FROM AUTHOR]

Platform trials offer a framework to study multiple interventions in one trial with the opportunity of opening and closing arms. The use of common controls can increase efficiency as compared to individual controls. The need for multiplicity adjustment because of common controls is currently a debate among researchers, pharmaceutical companies, and regulators. The impact of common controls on the type one error in a fixed platform trial, i.e. when all treatments start and end recruitment at the same time, has been discussed in the literature before. We complement these findings by investigating the impact of a common control on the type one error and power in a flexible platform trial, i.e. when one arm joins the platform later. We derived the correlation of test statistics to assess the impact of the overlap and compared the results to a trial with individual controls. Furthermore, we evaluate the power, and the impact of multiplicity adjustment on the power in fixed and flexible platform trials. These methodological considerations are complemented by a regulatory guideline review. With multiple arms, the FWER is inflated when no multiplicity adjustment is applied. However, the FWER inflation is smaller with common controls than with individual controls. Even after multiplicity adjustment, a trial with common controls is often beneficial in terms of sample size and power. However, in some cases, the trial with common controls loses the efficiency gain and it might be advisable to run a separate trial rather than joining a platform trial. [ABSTRACT FROM AUTHOR]

Researchers with mediation hypotheses must consider which design to use: within-subject or between-subject? In this paper, I argue that three factors should influence design choice: validity, causality, and statistical power. Threats to validity include carry-over effects, participant awareness, measurement, and more. Causality is a core element of mediation, and the assumptions required for causal inference differ between the two designs. Between-subject designs require more restrictive no-confounder assumptions, but within-subject designs require the assumption of no carry-over effects. Statistical power should be higher in within-subject designs, but the degree and conditions of this advantage are unknown for mediation analysis. A Monte Carlo simulation compares designs under a broad range of sample sizes, effect sizes, and correlations among repeated measurements. The results show within-subject designs require about half the sample size of between-subject designs to detect indirect effects of the same size, but this difference can vary with population parameters. I provide an empirical example and R script for conducting power analysis for within-subject mediation analysis. Researchers interested in conducting mediation analysis should not select within-subject designs merely because of higher power, but they should also consider validity and causality in their decision, both of which can favor between-subject designs.

The analysis of multiple bivariate correlations is often carried out by conducting simple tests to check whether each of them is significantly different from zero. In addition, pairwise differences are often judged by eye or by comparing the p‐values of the individual tests of significance despite the existence of statistical tests for differences between correlations. This paper uses simulation methods to assess the accuracy (empirical Type I error rate), power, and robustness of 10 tests designed to check the significance of the difference between two dependent correlations with overlapping variables (i.e., the correlation between X1 and Y and the correlation between X2 and Y). Five of the tests turned out to be inadvisable because their empirical Type I error rates under normality differ greatly from the nominal alpha level of .05 either across the board or within certain sub‐ranges of the parameter space. The remaining five tests were acceptable and their merits were similar in terms of all comparison criteria, although none of them was robust across all forms of non‐normality explored in the study. Practical recommendations are given for the choice of a statistical test to compare dependent correlations with overlapping variables. [ABSTRACT FROM AUTHOR]

Careful consideration of the tradeoff between Type I and Type II error rates when designing experiments is critical for maximizing statistical decision accuracy. Typically, Type I error rates (e.g., .05) are significantly lower than Type II error rates (e.g., .20 for .80 power) in psychological science. Further, positive findings (true effects and Type I errors) are more likely to be the focus of replication. This conventional approach leads to very high rates of Type II error. Analyses show that increasing the Type I error rate to .10, thereby increasing power and decreasing the Type II error rate for each test, leads to higher overall rates of correct statistical decisions. This increase of Type I error rate is consistent with, and most beneficial in the context of, the replication and "New Statistics" movements in psychology. Making decisions under conditions of uncertainty (e.g., science) always involves a tradeoff between two types of errors. The decision maker can conclude something is present when it is not (false positive or Type I error) or conclude no difference is present, when in reality there is a difference (a miss or Type II error). We address the need to carefully consider the balance of these error types in scientific decision making. In psychology, it is typical to more stringently control the Type I error rate at .05, while the Type II error rate is .20 (i.e., when power is .80). We demonstrate that the overall percentage correct decisions in psychological science can be increased by increasing the Type I error rate to .10 to increase power, and decrease the Type II error rate. This is particularly relevant given the recent methodological and statistical changes in the field, particularly the long-needed emphasis on replication. [ABSTRACT FROM AUTHOR]

Intraclass correlation in bilateral data has been investigated in recent decades with various statistical methods. In practice, stratifying bilateral data by some control variables will provide more sophisticated statistical results to satisfy different research proposed in randomized clinical trials. In this article, we propose three test statistics (the likelihood ratio test, score test, and Wald-type test statistics) to evaluate the homogeneity of proportion ratios for stratified bilateral correlated data under an equal correlation assumption. Monte Carlo simulations of Type I error and power are performed, and the score test yields a robust outcome based on empirical Type I error and power. Lastly, two real data examples are conducted to illustrate the proposed three tests. [ABSTRACT FROM AUTHOR]

Sample size estimation remains as a cornerstone in the meticulous planning and execution of clinical trials, pivotal for ensuring studies possess the requisite statistical power to discern meaningful treatment effects. Insufficient sample sizes compromise the robustness of findings, whereas excessively large samples inflate costs and compromise data integrity. This article meticulously explains the multifaceted factors that outline sample size determination, encompassing various factors such as research design, types of hypotheses, error thresholds, effect size considerations, validity and precision. It investigates into the scope of methodologies available for sample size computation, spanning from intricate statistical formulas to pragmatic tabular approaches. Moreover, it underscores the significance of post-hoc power analysis in retrospectively evaluating completed studies, shedding light on their statistical robustness. This literature review furnishes a nuanced understanding of the sample size estimation landscape in clinical trials, delineating their strengths, limitations, and real-world applications. Anticipating participant attrition assumes paramount importance for proactively adjusting sample sizes, ensuring studies remain methodologically sound. Equipped with a profound grasp of these principles, researchers are empowered to conduct scientifically rigorous and impactful clinical trials, furnishing compelling evidence to inform judicious decision-making in healthcare interventions.

Ordinary least squares (OLS) regression is widely employed for statistical prediction and theoretical explanation in psychology studies. However, OLS regression has a critical drawback: it becomes less accurate in the presence of outliers and non-random error distribution. Several robust regression methods have been proposed as alternatives. However, each robust regression has its own strengths and limitations. Consequently, researchers are often at a loss as to which robust regression method to use for their studies. This study uses a Monte Carlo experiment to compare different types of robust regression methods with OLS regression based on relative efficiency (RE), bias, root mean squared error (RMSE), Type 1 error, power, coverage probability of the 95% confidence intervals (CIs), and the width of the CIs. The results show that, with sufficient samples per predictor (n = 100), the robust regression methods are as efficient as OLS regression. When errors follow non-normal distributions, i.e., mixed-normal, symmetric and heavy-tailed (SH), asymmetric and relatively light-tailed (AL), asymmetric and heavy-tailed (AH), and heteroscedastic, the robust method (GM-estimation) seems to consistently outperform OLS regression.

Ordinary least squares (OLS) regression is widely employed for statistical prediction and theoretical explanation in psychology studies. However, OLS regression has a critical drawback: it becomes less accurate in the presence of outliers and non-random error distribution. Several robust regression methods have been proposed as alternatives. However, each robust regression has its own strengths and limitations. Consequently, researchers are often at a loss as to which robust regression method to use for their studies. This study uses a Monte Carlo experiment to compare different types of robust regression methods with OLS regression based on relative efficiency (RE), bias, root mean squared error (RMSE), Type 1 error, power, coverage probability of the 95% confidence intervals (CIs), and the width of the CIs. The results show that, with sufficient samples per predictor (n = 100), the robust regression methods are as efficient as OLS regression. When errors follow non-normal distributions, i.e., mixed-normal, symmetric and heavy-tailed (SH), asymmetric and relatively light-tailed (AL), asymmetric and heavy-tailed (AH), and heteroscedastic, the robust method (GM-estimation) seems to consistently outperform OLS regression. [ABSTRACT FROM AUTHOR]

The matching paradigm can take a number of forms and has been used in many areas of psychology. When participants are asked to match or order sets of objects, researchers must correctly account for the number of matches expected purely by chance. Not accounting for the expected chance matches can lead to incorrectly drawing conclusions based on one's data. This study demonstrated that the z test can be an appropriate and easy test to use in the analysis of matching data from studies that require pairs of objects to be matched with each other. This article proves that in a matching paradigm the expected number of chance matches is 1.0 and the associated variance is also 1.0. The z test is shown to maintain the Type I error close to the nominal significance level when the null hypothesis is true and the sample size is at least 80 or 110. To attain power of .80, a sample size larger than 80 may be needed depending upon the effect size associated with the area of interest and the hypothesized alternative probability distribution. The matching paradigm can take a number of forms and has been used in many areas of psychology. When participants are asked to match, or order, sets of objects researchers must correctly account for the number of matches expected purely by chance. Not accounting for the expected chance matches can lead to incorrectly drawing conclusions based on one's data. This study demonstrated that the simple one-sample z test can be an appropriate and easy test to use in the analysis of matching data from studies that require pairs of objects to be matched with each other. When the null hypothesis is true (i.e., matches are occurring by chance) and the sample size is at least 80 or 110, the z test is shown to maintain the Type I error close to the alpha (significance) level researchers selected. To attain power of .80 a sample size larger than 80 may be needed depending upon the effect size associated with the area of interest. [ABSTRACT FROM AUTHOR]

This work examined the sensitivity of randomization test and F‐test in completely randomized design, randomized complete block design and Latin square design in order to ascertain which is more robust in terms of power (sensitivity) of the test and type I error rate in cases with normal error, skewed error (Chi‐square and Exponential distributions), by varying the sample sizes. Using Monte Carlo simulation, response observations were generated for normal error, skewed error at various sample sizes for each of the experimental designs. The results showed that for the normal error, the power and type I error rate of the F‐test were slightly smaller and greater respectively in each of the designs than that of the randomization test. Furthermore, for the skewed error, the randomization test was more sensitive than the F‐test, and also, the type I error was smaller in each of the designs. [ABSTRACT FROM AUTHOR]

In this study, the methods based on Classical Test Theory and Item Response Theory were used comparatively to determine Type I error and power rates in Differential Item Functioning. Logistic regression, MantelHaenszel, Lord's X², Breslow-Day and Raju's area index methods were used for the analyses, which were conducted using the R programming language. Determination of Type I error and power rates of these methods under certain conditions was carried out by simulation study. For data generation, analyzes were made under eight conditions in total by examining different sample sizes and DIF rates created with the WinGen 3 program. The results of the study indicate that, in general when the ratio of items containing DIF increased, Type I error increased and the power ratio decreased. Among the methods based on Item Response Theory, Lord's X² and Raju's area index methods gave better results than other methods with low error and high power. [ABSTRACT FROM AUTHOR]

A practical solution based on the method of variance estimates recovery is proposed for the Behrens–Fisher problem. Unlike the classical solution, the proposed test is not based on the central limit theorem and, therefore, it can control Type I error well for small or moderate sample sizes. In this sense, it performs better than the most commonly used Welch's approximate t-test, which may be liberal for small or moderate sample sizes. On the other hand, the powers of the proposed test appears to be close to that of the Welch's approximate t-test and better than those of the Fisher's solution and the generalized p-value solution; the latter two have no fixed p-value and need to simulate. The proposed test has a meaningful interpretation within and not just across experiments. It can be a competitive alternative solution. [ABSTRACT FROM AUTHOR]

A commonly used test for the independence of two sets of variables from a normal multivariate population is the Wilks Lambda test or the multivariate likelihood ratio test (LRT). However, this test's performance is highly influenced by outliers and non-normality of the data and thus, robust test statistics should be used, allied to Monte-Carlo methods. In this paper, we proposed and evaluated three new likelihood ratio test for the independence of two sets of multivariate variables (LRTR, T and TR), replacing the traditional covariance matrix estimator with the robust comedian estimators. The results showed that the proposed test T and TR control type I error rates also for non-normal distributions outperforming the ordinary test. [ABSTRACT FROM AUTHOR]

Abstract Background Due to the high cost and high failure rate of Phase III trials where a classical group sequential design (GSD) is usually used, seamless Phase II/III designs are more and more popular to improve trial efficiency. A potential attraction of Phase II/III design is to allow a randomized proof-of-concept stage prior to committing to the full cost of a Phase III trial. Population selection during the trial allows a trial to adapt and focus investment where it is most likely to provide patient benefit. Previous methods have been developed for this problem when there is a single primary endpoint and two possible populations. Methods To find the population that potentially benefits with one or two primary endpoints (e.g., progression free survival (PFS), overall survival (OS)), we propose a gated group sequential design for a seamless Phase II/III trial design with adaptive population selection. Results The investigated design controls the familywise error rate and allows multiple interim analyses to enable early stopping for efficacy or futility. Simulations and an illustrative example suggest that the proposed gated group sequential design has more power and requires less time and resources compared to the group sequential design and adaptive design. Conclusions Combining the group sequential design and adaptive design, the gated group sequential design has more power and higher efficiency while controlling for the familywise error rate. It has the potential to save drug development cost and more quickly fulfill unmet medical needs.

It has been reported that about half of biological discoveries are irreproducible. These irreproducible discoveries were partially attributed to poor statistical power. The poor powers are majorly owned to small sample sizes. However, in molecular biology and medicine, due to the limit of biological resources and budget, most molecular biological experiments have been conducted with small samples. Two-sample t-test controls bias by using a degree of freedom. However, this also implicates that t-test has low power in small samples. A discovery found with low statistical power suggests that it has a poor reproducibility. So, promotion of statistical power is not a feasible way to enhance reproducibility in small-sample experiments. An alternative way is to reduce type I error rate. For doing so, a so-called tα-test was developed. Both theoretical analysis and simulation study demonstrate that tα-test much outperforms t-test. However, tα-test is reduced to t-test when sample sizes are over 15. Large-scale simulation studies and real experiment data show that tα-test significantly reduced type I error rate compared to t-test and Wilcoxon test in small-sample experiments. tα-test had almost the same empirical power with t-test. Null p-value density distribution explains why tα-test had so lower type I error rate than t-test. One real experimental dataset provides a typical example to show that tα-test outperforms t-test and a microarray dataset showed that tα-test had the best performance among five statistical methods. In addition, the density distribution and probability cumulative function of tα-statistic were given in mathematics and the theoretical and observed distributions are well matched. [ABSTRACT FROM AUTHOR]

One of the most important statistical analyses when designing animal and human studies is the calculation of the required sample size. In this review, we define central terms in the context of sample size determination, including mean, standard deviation, statistical hypothesis testing, type I/II error, power, direction of effect, effect size, expected attrition, corrected sample size, and allocation ratio. We also provide practical examples of sample size calculations for animal and human studies based on pilot studies, larger studies similar to the proposed study—or if no previous studies are available—estimated magnitudes of the effect size per Cohen and Sawilowsky.

Intraclass correlation in bilateral data has been investigated in recent decades with various statistical methods. In practice, stratifying bilateral data by some control variables will provide more sophisticated statistical results to satisfy different research proposed in randomized clinical trials. In this article, we propose three test statistics (the likelihood ratio test, score test, and Wald-type test statistics) to evaluate the homogeneity of proportion ratios for stratified bilateral correlated data under an equal correlation assumption. Monte Carlo simulations of Type I error and power are performed, and the score test yields a robust outcome based on empirical Type I error and power. Lastly, two real data examples are conducted to illustrate the proposed three tests.

Analyses of longitudinal data with non-linear mixed-effects models (NLMEM) are typically associated with high power, but sometimes at the cost of inflated type I error. Approaches to overcome this problem were published recently, such as model-averaging across drug models (MAD), individual model-averaging (IMA), and combined Likelihood Ratio Test (cLRT). This work aimed to assess seven NLMEM approaches in the same framework: treatment effect assessment in balanced two-armed designs using real natural history data with or without the addition of simulated treatment effect. The approaches are MAD, IMA, cLRT, standard model selection (STDs), structural similarity selection (SSs), randomized cLRT (rcLRT), and model-averaging across placebo and drug models (MAPD). The assessment included type I error, using Alzheimer's Disease Assessment Scale-cognitive (ADAS-cog) scores from 817 untreated patients and power and accuracy in the treatment effect estimates after the addition of simulated treatment effects. The model selection and averaging among a set of pre-selected candidate models were driven by the Akaike information criteria (AIC). The type I error rate was controlled only for IMA and rcLRT; the inflation observed otherwise was explained by the placebo model misspecification and selection bias. Both IMA and rcLRT had reasonable power and accuracy except under a low typical treatment effect. [ABSTRACT FROM AUTHOR]

Fixed sample clinical trials involve humans accumulated over a considerably long period. Ignoring some of the economic and ethical issues make these designs impractical, bringing sequential designs into practice. At interim inspections, sequential trials either stop or continue. These usually have a lesser number of patients leading to savings in cost and time while improving ethical issues. When the response is binary the application of the binomial test in a sequential framework is merited. This research compares three popular repeated sequential types of designs namely, Pocock, O'Brien and Fleming, and Lan and DeMets. These are compared using type I error, power, mean information, and average duration to stop the trial. To assess the practical feasibility, all designs were applied to a real-life data set. All designs provide type I error rates which are maintained within the acceptable region while keeping high power. The results suggest that the Lan and DeMets design is slightly superior to the designs studied while assessing that all designs are practicable. [ABSTRACT FROM AUTHOR]

Background: Due to the high cost and high failure rate of Phase III trials where a classical group sequential design (GSD) is usually used, seamless Phase II/III designs are more and more popular to improve trial efficiency. A potential attraction of Phase II/III design is to allow a randomized proof-of-concept stage prior to committing to the full cost of a Phase III trial. Population selection during the trial allows a trial to adapt and focus investment where it is most likely to provide patient benefit. Previous methods have been developed for this problem when there is a single primary endpoint and two possible populations. Methods: To find the population that potentially benefits with one or two primary endpoints (e.g., progression free survival (PFS), overall survival (OS)), we propose a gated group sequential design for a seamless Phase II/III trial design with adaptive population selection. Results: The investigated design controls the familywise error rate and allows multiple interim analyses to enable early stopping for efficacy or futility. Simulations and an illustrative example suggest that the proposed gated group sequential design has more power and requires less time and resources compared to the group sequential design and adaptive design. Conclusions: Combining the group sequential design and adaptive design, the gated group sequential design has more power and higher efficiency while controlling for the familywise error rate. It has the potential to save drug development cost and more quickly fulfill unmet medical needs. [ABSTRACT FROM AUTHOR]

Background: Null Hypothesis Significance Testing (NHST) has been well criticised over the years yet remains a pillar of statistical inference. Although NHST is well described in terms of statistical models, most textbooks for non-statisticians present the null and alternative hypotheses (H0 and HA, respectively) in terms of differences between groups such as (μ1 = μ2) and (μ1 ≠ μ2) and HA is often stated to be the research hypothesis. Here we use propositional calculus to analyse the internal logic of NHST when couched in this popular terminology. The testable H0 is determined by analysing the scope and limits of the P-value and the test statistic's probability distribution curve. Results: We propose a minimum axiom set NHST in which it is taken as axiomatic that H0 is rejected if P-value< α. Using the common scenario of the comparison of the means of two sample groups as an example, the testable H0 is {(μ1 = μ2) and [( x ¯ 1 ≠ x ¯ 2) due to chance alone]}. The H0 and HA pair should be exhaustive to avoid false dichotomies. This entails that HA is ¬{(μ1 = μ2) and [( x ¯ 1 ≠ x ¯ 2) due to chance alone]}, rather than the research hypothesis (HT). To see the relationship between HA and HT, HA can be rewritten as the disjunction HA: ({(μ1 = μ2) ∧ [( x ¯ 1 ≠ x ¯ 2) not due to chance alone]} ∨ {(μ1 ≠ μ2) ∧ [ (x ¯ 1 ≠ x ¯ 2) not due to (μ1 ≠ μ2) alone]} ∨ {(μ1 ≠ μ2) ∧ [( x ¯ 1≠ x ¯ 2) due to (μ1 ≠ μ2) alone]}). This reveals that HT (the last disjunct in bold) is just one possibility within HA. It is only by adding premises to NHST that HT or other conclusions can be reached. Conclusions: Using this popular terminology for NHST, analysis shows that the definitions of H0 and HA differ from those found in textbooks. In this framework, achieving a statistically significant result only justifies the broad conclusion that the results are not due to chance alone, not that the research hypothesis is true. More transparency is needed concerning the premises added to NHST to rig particular conclusions such as HT. There are also ramifications for the interpretation of Type I and II errors, as well as power, which do not specifically refer to HT as claimed by texts. [ABSTRACT FROM AUTHOR]

In survival studies the treatment effect on the hazard ratio may not extend over the entire observation time, yet the effect over a shorter time may be medically important. If the analysis using all events over the observation time does not show a significant benefit, then post hoc analyses at earlier times, which show nominal significance, would only be viewed as post hoc and suggestive of an effect. Therefore, it would be advantageous to incorporate a shorter duration of treatment effect into the study design. A backward sequential logrank procedure, with testing at the end of study and if not significant then sequentially at prespecified earlier time points with a prespecified distribution of Type I error, is proposed. Critical values and power are obtained by simulation for repeated logrank tests at prespecified times and α distribution. When the treatment effect does not extend over the entire observation period the backward logrank testing procedure is more powerful compared to the power of a single logrank test at the end of the study. [ABSTRACT FROM AUTHOR]

Abstract Objective Allelic imbalance (AI) is the differential expression of the two alleles in a diploid. AI can vary between tissues, treatments, and environments. Methods for testing AI exist, but methods are needed to estimate type I error and power for detecting AI and difference of AI between conditions. As the costs of the technology plummet, what is more important: reads or replicates? Results We find that a minimum of 2400, 480, and 240 allele specific reads divided equally among 12, 5, and 3 replicates is needed to detect a 10, 20, and 30%, respectively, deviation from allelic balance in a condition with power > 80%. A minimum of 960 and 240 allele specific reads divided equally among 8 replicates is needed to detect a 20 or 30% difference in AI between conditions with comparable power. Higher numbers of replicates increase power more than adding coverage without affecting type I error. We provide a Python package that enables simulation of AI scenarios and enables individuals to estimate type I error and power in detecting AI and differences in AI between conditions.

A non-repeated item (NRI) design refers to an experimental design in which items used in one level of experimental conditions are not repeatedly used at other levels. Recent literature has suggested the use of generalized linear mixed-effects models (GLMMs) for experimental data analysis, but the existing specification of GLMMs does not account for all possible dependencies among the outcomes in NRI designs. Therefore, the current study proposed a GLMM with a level-specific item random effect for NRI designs. The hypothesis testing performance of the newly proposed model was evaluated via a simulation study to detect the experimental condition effect. The model with a level-specific item random effect performed better than the existing model in terms of power when the variance of the item effect was heterogeneous. Based on these results, we suggest that experimental researchers using NRI designs consider setting a level-specific item random effect in the model. [ABSTRACT FROM AUTHOR]

Power calculations are a key study design step in research studies. However, such power analysis is often inappropriately performed in the medical literature by attempting to help interpret the findings of a completed study, instead of attempting to aid in choosing an optimal sample size for a future study. The aim of this article is to provide a brief discussion of the drawbacks of performing these post hoc power calculations, and to correspondingly suggest best practices regarding the use of statistical power and the interpretation of study results. Specifically, power analysis should always be considered before any research study in order to choose an ideal sample size and/or to examine the feasibility of properly evaluating study aims, but it should never be used in order to help interpret the results of an already completed study. Alternatively, 95% confidence intervals for effect sizes (eg, odds ratio, hazard ratio, mean difference) or other relevant parameter estimates should be used when attempting to draw conclusions from results, such as the likelihood of a type II error (ie, a false negative finding). [ABSTRACT FROM AUTHOR]

For the sequential parallel comparison design with normally distributed outcomes and rerandomization at the start of phase 2, we propose fitting separate repeated measures models for the two phases. We show analytically the asymptotic independence between the treatment effect estimators in the two phases. We recommend prespecification of the weights for combining the treatment effects in the two phases and use of the t reference distribution with the Welch–Satterthwaite degrees of freedom for testing the overall treatment effect when the sample size is small. We use simulation studies to demonstrate that the proposed method controls Type I error and has proper coverage of 95% confidence intervals. [ABSTRACT FROM AUTHOR]

The non-inferiority of odds ratio in a matched-pairs design is a common question in medical research, and several different approaches are available to answer it. However, their performance is not always satisfactory. Some of the traditional approaches, such as the delta test and score test, do not perform well unless the sample size is large. The simulated significance level of the inferential model (IM) test is conservative, and its power is lower than that of the other tests. The results of the randomized IM test are satisfactory; however, since it is based on ω , which is randomly generated, the plausibility function is not fixed. This paper describes a fiducial test that is based on Fisher's fiducial argument. Our simulation studies illustrated that the fiducial test can control Type I error, and its power appears to be very close to that of the score test. All of the test procedures are illustrated with a real example. [ABSTRACT FROM AUTHOR]

Before making inferences about a population using a fitted model, it is necessary to determine whether the fitted model describes the data well. A poorly fitted model may lead to biased and invalid conclusions, resulting in incorrect inferences. Recent studies show the necessity of goodness-of-fit tests for high level binary multilevel models. The focus here was to develop a goodness-of-fit test to use in the model adequacy testing of high level binary multilevel models and to examine, whether the type I error and power hold for the newly developed goodness-of-fit test considering a three-level random intercept model. [ABSTRACT FROM AUTHOR]

The two one-sided tests (TOST) procedure is widely applied for assessing bioequivalence between two different drugs in clinical studies. The same notion can be applied to construct TOST method for establishing dispersion equivalence. In addition to the TOST extension, a uniformly most power test (UMPT) for equivalence in variability has been described in the literature. This article presents analytic and numerical comparisons to clarify the limitation and strength of the two procedures. The UMPT approach has comparatively superior control of Type I errors and demands more involved computations than the TOST method. To enhance the utility of the recommended UMPT procedure, a full set of computer algorithms are constructed to implement the described hypothesis testing and power calculation. Optimal sample size determinations are also developed for planning equivalence designs under different allocation and cost concerns. The suggested procedures are demonstrated with the data regarding the dispersion comparability between two measuring devices of breaking strengths in dental medicine. [ABSTRACT FROM AUTHOR]

There is an urgent need to evaluate new therapies in a time-sensitive and cost-effective manner. We propose the adaptive seamless phase II/III clinical trials with covariate adaptive randomization (CAR) to satisfy this need. CAR is one of the most popular designs in randomized controlled trials, enhancing covariance balance and ensuring valid treatment comparisons. However, it has several challenges: (1) the type I error rate of the commonly used Student's t -test following CAR can be in ated because of the seamless trials, but can also be decreased using CAR; (2) the complicated allocation mechanism induced by CAR causes extra difficulties to derive the asymptotic properties of a test procedure; and (3) previous theoretical studies of seamless trials rely mainly on the assumption of complete randomization, a procedure rarely used in real trials. We establish a theoretical foundation for adaptive seamless phase II/III trials with CAR. We also propose an approach that is easy to implement in order to control the type I error rate and improve the power when using Student's t -test. This important step will promote the application of this procedure. [ABSTRACT FROM AUTHOR]

Sample size calculation is an integral part of any clinical trial design, and determining the optimal sample size for a study ensures adequate power to detect statistical significance. It is a critical step in designing a planned research protocol, since using too many participants in a study is expensive, exposing more subjects to the procedure. If a study is underpowered, it will be statistically inconclusive and may cause the whole protocol to fail. Amidst the attempt to maximize power and the underlying effort to minimize the budget, the optimization of both has become a significant issue in the determination of sample size for clinical trials in recent decades. Although it is hard to generalize a single method for sample size calculation, this study is an attempt to offer something that might be a basis for finding a permanent answer to the contradictions of sample size determination, by the use of simulation studies under simple random and cluster sampling schemes, with different sizes of power and type I error. The effective sample size is much higher when the design effect of the sampling method is smaller, particularly less than 1. Sample size increases for cluster sampling when the number of clusters increases. [ABSTRACT FROM AUTHOR]

Objective: Allelic imbalance (AI) is the differential expression of the two alleles in a diploid. AI can vary between tissues, treatments, and environments. Methods for testing AI exist, but methods are needed to estimate type I error and power for detecting AI and difference of AI between conditions. As the costs of the technology plummet, what is more important: reads or replicates? Results: We find that a minimum of 2400, 480, and 240 allele specific reads divided equally among 12, 5, and 3 replicates is needed to detect a 10, 20, and 30%, respectively, deviation from allelic balance in a condition with power > 80%. A minimum of 960 and 240 allele specific reads divided equally among 8 replicates is needed to detect a 20 or 30% difference in AI between conditions with comparable power. Higher numbers of replicates increase power more than adding coverage without affecting type I error. We provide a Python package that enables simulation of AI scenarios and enables individuals to estimate type I error and power in detecting AI and differences in AI between conditions. [ABSTRACT FROM AUTHOR]

Despite the vast amount of literature on supersaturated designs (SSDs), there is a scant record of their use in practice. We contend this imbalance is due to conflicting recommendations regarding SSD use in the literature as well as the designs' inabilities to meet practitioners' analysis expectations. To address these issues, we first summarize practitioner concerns and expectations of SSDs as determined via an informal questionnaire. Next, we discuss and compare two recent SSDs that pair a design construction method with a particular analysis method. The choice of a design/analysis pairing is shown to depend on the screening objective. Group orthogonal SSDs, when paired with our new, modified analysis, are demonstrated to have high power even with many active factors. Constrained positive Var(s)-optimal designs, when paired with the Dantzig selector, are recommended when effect directions can be credibly specified in advance; this strategy reasonably controls Type I error rates while still identifying a high proportion of active factors. [ABSTRACT FROM AUTHOR]

• Hierarchical bootstrap as alternative to parametric Type I/II error simulations. • Identify inflated Type I & II errors of two most common approaches to RT analyses. • Identify substantial over-optimism of RT analyses in the literature. • Identify issues with past parametric validations of analysis approaches. • Provide recommendations for new standards in RT analyses. We investigate the statistical power and Type I error rate of the two most common approaches to reading time (RT) analyses: assuming normality of residuals and homogeneity of variance in raw or log-transformed RTs. We first show that the assumptions of such analyses—such as t -tests, ANOVAs, and linear mixed-effects models—are neither consistently met by raw RTs, nor by log-transformed RTs (or any other common power transforms, incl. inverse-transformed RTs). Only a non-power transform (log-shift) provides a decent fit for all data sets and data preparation steps we consider. We then compare the statistical power and Type I error rate for linear mixed-effects models over raw or log-transformed RTs. Previous studies on this matter relied on parametrically generated data. We show why this is problematic, and introduce as an alternative a hierarchical bootstrap approach over naturally distributed reading times. This approach yields substantially different—and arguably more informative—results than the parametric simulation approaches we compare it to. Our results suggests that it is time to heed the advice others have provided for reading research: for any but the simplest designs, we find both the rate of spurious significances and the rate of undetected true effects can strongly depend on the scale (e.g., raw or log-RTs) in which effects are assumed to be linear. Researchers should thus clearly motivate the choice of analysis based on theoretical grounds, assess the robustness of findings under different analysis approaches, and discuss potential mismatches between analyses. The R scripts and libraries shared in the accompanying OSF repo allow researchers to assess the reliability of their analyses via hierarchical bootstrap over their own data. [ABSTRACT FROM AUTHOR]

Abstract Background Pathway enrichment extensively used in the analysis of Omics data for gaining biological insights into the functional roles of pre-defined subsets of genes, proteins and metabolites. A large number of methods have been proposed in the literature for this task. The vast majority of these methods use as input expression levels of the biomolecules under study together with their membership in pathways of interest. The latest generation of pathway enrichment methods also leverages information on the topology of the underlying pathways, which as evidence from their evaluation reveals, lead to improved sensitivity and specificity. Nevertheless, a systematic empirical comparison of such methods is still lacking, making selection of the most suitable method for a specific experimental setting challenging. This comparative study of nine network-based methods for pathway enrichment analysis aims to provide a systematic evaluation of their performance based on three real data sets with different number of features (genes/metabolites) and number of samples. Results The findings highlight both methodological and empirical differences across the nine methods. In particular, certain methods assess pathway enrichment due to differences both across expression levels and in the strength of the interconnectedness of the members of the pathway, while others only leverage differential expression levels. In the more challenging setting involving a metabolomics data set, the results show that methods that utilize both pieces of information (with NetGSA being a prototypical one) exhibit superior statistical power in detecting pathway enrichment. Conclusion The analysis reveals that a number of methods perform equally well when testing large size pathways, which is the case with genomic data. On the other hand, NetGSA that takes into consideration both differential expression of the biomolecules in the pathway, as well as changes in the topology exhibits a superior performance when testing small size pathways, which is usually the case for metabolomics data.

Covariate‐adaptive randomization (CAR) is widely used in clinical trials to balance treatment allocation over covariates. Over the past decade, significant progress has been made on the theoretical properties of covariate‐adaptive design and associated inference. However, most results are established under the assumption that the covariates are correctly measured. In practice, measurement error is inevitable, resulting in misclassification for discrete covariates. When covariate misclassification is present in a clinical trial conducted using CAR, the impact is twofold: it impairs the intended covariate balance, and raises concerns over the validity of test procedures. In this paper, we consider the impact of misclassification on covariate‐adaptive randomized trials from the perspectives of both design and inference. We derive the asymptotic normality, and thereby the convergence rate, of the imbalance of the true covariates for a general family of covariate‐adaptive randomization methods, and show that a superior covariate balance can still be attained compared to complete randomization. We also show that the two sample t‐test is conservative, with a reduced Type I error, but that this can be corrected using a bootstrap method. Moreover, if the misclassified covariates are adjusted in the model used for analysis, the test maintains its nominal Type I error, with an increased power. Our results support the use of covariate‐adaptive randomization in clinical trials, even when the covariates are subject to misclassification. [ABSTRACT FROM AUTHOR]

Data-driven model modification plays an important role for a statistical methodology to advance the understanding of subjective matters. However, when the sample size is not sufficiently large model modification using the Lagrange multiplier (LM) test has been found not performing well due to capitalization on chance. With the recent development of lasso regression in statistical learning, lasso regularization for structural equation modeling (SEM) may seem to be a method that could avoid capitalizing on chance in finding an adequate model. But there is little evidence validating the goodness of lasso SEM. The purpose of this article is to examine the performance of lasso SEM by comparing it against the LM test, aiming to answer the following five questions: (1) Can we trust the results of lasso SEM for model modification? (2) Does the performance of lasso SEM depend more on the effect size or the absolute value of the parameter? (3) Does lasso SEM perform better than the widely used LM test for model modification? (4) Are lasso SEM and LM test affected by nonnormally distributed data in practice? and (5) Do lasso SEM and LM test perform better with robustly transformed data? By addressing these questions with real data, results indicate that lasso SEM is unable to deliver the expected promises, and it does not perform better than the LM test. [ABSTRACT FROM AUTHOR]

The normal distribution is considered to be one of the most important distributions, with numerous applications in various fields, including the field of agricultural sciences. The purpose of this study is to evaluate the most popular normality tests, comparing the performance in terms of the size (type I error) and the power against a large spectrum of distributions with simulations for various sample sizes and significance levels, as well as through empirical data from agricultural experiments. The simulation results show that the power of all normality tests is low for small sample size, but as the sample size increases, the power increases as well. Also, the results show that the Shapiro–Wilk test is powerful over a wide range of alternative distributions and sample sizes and especially in asymmetric distributions. Moreover the D'Agostino–Pearson Omnibus test is powerful for small sample sizes against symmetric alternative distributions, while the same is true for the Kurtosis test for moderate and large sample sizes. [ABSTRACT FROM AUTHOR]

Binomial regression models are commonly applied to proportion data such as those relating to the mortality and infection rates of diseases. However, it is often the case that the responses may exhibit excessive zeros; in such cases a zero‐inflated binomial (ZIB) regression model can be applied instead. In practice, it is essential to test if there are excessive zeros in the outcome to help choose an appropriate model. The binomial models can yield biased inference if there are excessive zeros, while ZIB models may be unnecessarily complex and hard to interpret, and even face convergence issues, if there are no excessive zeros. In this paper, we develop a new test for testing zero inflation in binomial regression models by directly comparing the amount of observed zeros with what would be expected under the binomial regression model. A closed form of the test statistic, as well as the asymptotic properties of the test, is derived based on estimating equations. Our systematic simulation studies show that the new test performs very well in most cases, and outperforms the classical Wald, likelihood ratio, and score tests, especially in controlling type I errors. Two real data examples are also included for illustrative purpose. [ABSTRACT FROM AUTHOR]

Abstract Background Most confirmatory randomised controlled clinical trials (RCTs) are designed with specified power, usually 80% or 90%, for a hypothesis test conducted at a given significance level, usually 2.5% for a one-sided test. Approval of the experimental treatment by regulatory agencies is then based on the result of such a significance test with other information to balance the risk of adverse events against the benefit of the treatment to future patients. In the setting of a rare disease, recruiting sufficient patients to achieve conventional error rates for clinically reasonable effect sizes may be infeasible, suggesting that the decision-making process should reflect the size of the target population. Methods We considered the use of a decision-theoretic value of information (VOI) method to obtain the optimal sample size and significance level for confirmatory RCTs in a range of settings. We assume the decision maker represents society. For simplicity we assume the primary endpoint to be normally distributed with unknown mean following some normal prior distribution representing information on the anticipated effectiveness of the therapy available before the trial. The method is illustrated by an application in an RCT in haemophilia A. We explicitly specify the utility in terms of improvement in primary outcome and compare this with the costs of treating patients, both financial and in terms of potential harm, during the trial and in the future. Results The optimal sample size for the clinical trial decreases as the size of the population decreases. For non-zero cost of treating future patients, either monetary or in terms of potential harmful effects, stronger evidence is required for approval as the population size increases, though this is not the case if the costs of treating future patients are ignored. Conclusions Decision-theoretic VOI methods offer a flexible approach with both type I error rate and power (or equivalently trial sample size) depending on the size of the future population for whom the treatment under investigation is intended. This might be particularly suitable for small populations when there is considerable information about the patient population.

The normal distribution is considered to be one of the most important distributions, with numerous applications in various fields, including the field of agricultural sciences. The purpose of this study is to evaluate the most popular normality tests, comparing the performance in terms of the size (type I error) and the power against a large spectrum of distributions with simulations for various sample sizes and significance levels, as well as through empirical data from agricultural experiments. The simulation results show that the power of all normality tests is low for small sample size, but as the sample size increases, the power increases as well. Also, the results show that the Shapiro–Wilk test is powerful over a wide range of alternative distributions and sample sizes and especially in asymmetric distributions. Moreover the D'Agostino–Pearson Omnibus test is powerful for small sample sizes against symmetric alternative distributions, while the same is true for the Kurtosis test for moderate and large sample sizes. [ABSTRACT FROM AUTHOR]

Summary: For pediatric drug development, the clinical effectiveness of the study medication for the adult population has already been demonstrated. Given the fact that it is usually not feasible to enroll a large number of pediatric patients, appropriately leveraging historical adult data into pediatric evaluation may be critical to success of pediatric drug development. In this manuscript, we propose a new empirical Bayesian approach, profile Bayesian estimation, to dynamically borrow adult information to the evaluation of treatment effect in pediatric patients. The new approach demonstrates an attractive balance between type I error control and power gain under the transfer‐ability assumption that the pediatric treatment effect size may differ from the adult treatment effect size. The decision making boundary mimics the real‐world practice in pediatric drug development. In addition, the posterior mean of the proposed empirical profile Bayesian is an unbiased estimator of the true pediatric treatment effect. We compare our approach to robust mixture prior with prior weight for informative borrowing set to 0.5 or 0.9, regular Bayesian approach, and frequentist for both type I error and power. [ABSTRACT FROM AUTHOR]