Search

Simultaneous confidence intervals (SCIs) that are compatible with a given closed test procedure are often non-informative. More precisely, for a one-sided null hypothesis, the bound of the SCI can stick to the border of the null hypothesis, irrespective of how far the point estimate deviates from the null hypothesis. This has been illustrated for the Bonferroni-Holm and fall-back procedures, for which alternative SCIs have been suggested, that are free of this deficiency. These informative SCIs are not fully compatible with the initial multiple test, but are close to it and hence provide similar power advantages. They provide a multiple hypothesis test with strong family-wise error rate control that can be used in replacement of the initial multiple test. The current paper extends previous work for informative SCIs to graphical test procedures. The information gained from the newly suggested SCIs is shown to be always increasing with increasing evidence against a null hypothesis. The new SCIs provide a compromise between information gain and the goal to reject as many hypotheses as possible. The SCIs are defined via a family of dual graphs and the projection method. A simple iterative algorithm for the computation of the intervals is provided. A simulation study illustrates the results for a complex graphical test procedure.

This note presents a method that provides optimal monotone conditional error functions for a large class of adaptive two stage designs. The presented method builds on a previously developed general theory for optimal adaptive two stage designs where sample sizes are reassessed for a specific conditional power and the goal is to minimize the expected sample size. The previous theory can easily lead to a non-monotonous conditional error function which is highly undesirable for logical reasons and can harm type I error rate control for composite null hypotheses. The here presented method extends the existing theory by introducing intermediate monotonising steps that can easily be implemented.

In this paper we consider online multiple testing with familywise error rate (FWER) control, where the probability of committing at least one type I error shall remain under control while testing a possibly infinite sequence of hypotheses over time. Currently, Adaptive-Discard (ADDIS) procedures seem to be the most promising online procedures with FWER control in terms of power. Now, our main contribution is a uniform improvement of the ADDIS principle and thus of all ADDIS procedures. This means, the methods we propose reject as least as much hypotheses as ADDIS procedures and in some cases even more, while maintaining FWER control. In addition, we show that there is no other FWER controlling procedure that enlarges the event of rejecting any hypothesis. Finally, we apply the new principle to derive uniform improvements of the ADDIS-Spending and ADDIS-Graph.

The population-wise error rate (PWER) is a type I error rate for clinical trials with multiple target populations. In such trials, one treatment is tested for its efficacy in each population. The PWER is defined as the probability that a randomly selected, future patient will be exposed to an inefficient treatment based on the study results. The PWER can be understood and computed as an average of strata-specific family-wise error rates and involves the prevalences of these strata. A major issue of this concept is that the prevalences are usually not known in practice, so that the PWER cannot be directly controlled. Instead, one could use an estimator based on the given sample, like their maximum-likelihood estimator under a multinomial distribution. In this paper we show in simulations that this does not substantially inflate the true PWER. We differentiate between the expected PWER, which is almost perfectly controlled, and study-specific values of the PWER which are conditioned on all subgroup sample sizes and vary within a narrow range. Thereby, we consider up to eight different overlapping patient populations and moderate to large sample sizes. In these settings, we also consider the maximum strata-wise family-wise error rate, which is found to be, on average, at least bounded by twice the significance level used for PWER control. Finally, we introduce an adjustment of the PWER that could be made when, by chance, no patients are recruited from a stratum, so that this stratum is not counted in PWER control. We would then reduce the PWER in order to control for multiplicity in this stratum as well., Comment: 14 pages, 7 figures

Assume that a set of $P$ process parameters $p_i$, $i=1,\dots,P$, determines the outcome of a set of $D$ descriptor variables $d_j$, $j=1,\dots,D$, via an unknown functional relationship $\phi: \mathbf{p} \mapsto \mathbf{d}, \, \mathbb{R}^{P} \to \mathbb{R}^{D}$, where $\mathbf{p}=(p_1,\dots,p_{P})$, $\mathbf{d}=(d_1,\dots,d_{D})$. It is desired to find appropriate values $\mathbf{\hat p} = ({\hat p}_1,\dots, {\hat p}_P)$ for the process parameters such that the corresponding values of the descriptor variables $\phi (\mathbf {\hat p})$ are close to a given target $\mathbf d^*=(d^*_1,\dots,d^*_D)$, assuming that at least one exact solution exists. A sequential approach using dimension reduction techniques has been developed to achieve this. In a simulation study, results of the suggested approach and the algorithms NSGA-II, SMS-EMOA and MOEA/D are compared., Comment: 29 pages, 26 figures, 1 table

In contemporary research, online error control is often required, where an error criterion, such as familywise error rate (FWER) or false discovery rate (FDR), shall remain under control while testing an a priori unbounded sequence of hypotheses. The existing online literature mainly considered large-scale designs and constructed blackbox-like algorithms for these. However, smaller studies, such as platform trials, require high flexibility and easy interpretability to take study objectives into account and facilitate the communication. Another challenge in platform trials is that due to the shared control arm some of the p-values are dependent and significance levels need to be prespecified before the decisions for all the past treatments are available. We propose ADDIS-Graphs with FWER control that due to their graphical structure perfectly adapt to such settings and provably uniformly improve the state-of-the-art method. We introduce several extensions of these ADDIS-Graphs, including the incorporation of information about the joint distribution of the p-values and a version for FDR control., Comment: 31 pages, 10 figures

The closure principle is fundamental in multiple testing and has been used to derive many efficient procedures with familywise error rate control. However, it is often unsuitable for modern research, which involves flexible multiple testing settings where not all hypotheses are known at the beginning of the evaluation. In this paper, we focus on online multiple testing where a possibly infinite sequence of hypotheses is tested over time. At each step, it must be decided on the current hypothesis without having any information about the hypotheses that have not been tested yet. Our main contribution is a general and stringent mathematical definition of online multiple testing and a new online closure principle which ensures that the resulting closed procedure can be applied in the online setting. We prove that any familywise error rate controlling online procedure can be derived by this online closure principle and provide admissibility results. In addition, we demonstrate how short-cuts of these online closed procedures can be obtained under a suitable consonance property., Comment: 21 pages, 8 figures

In machine learning, the selection of a promising model from a potentially large number of competing models and the assessment of its generalization performance are critical tasks that need careful consideration. Typically, model selection and evaluation are strictly separated endeavors, splitting the sample at hand into a training, validation, and evaluation set, and only compute a single confidence interval for the prediction performance of the final selected model. We however propose an algorithm how to compute valid lower confidence bounds for multiple models that have been selected based on their prediction performances in the evaluation set by interpreting the selection problem as a simultaneous inference problem. We use bootstrap tilting and a maxT-type multiplicity correction. The approach is universally applicable for any combination of prediction models, any model selection strategy, and any prediction performance measure that accepts weights. We conducted various simulation experiments which show that our proposed approach yields lower confidence bounds that are at least comparably good as bounds from standard approaches, and that reliably reach the nominal coverage probability. In addition, especially when sample size is small, our proposed approach yields better performing prediction models than the default selection of only one model for evaluation does., Comment: 17 pages, 13 figures, 3 tables. Submitted to the Springer Machine Learning Journal. Changes to version 2: made figures easier to read; corrected a minor typo

Three-arm `gold-standard' non-inferiority trials are recommended for indications where only unstable reference treatments are available and the use of a placebo group can be justified ethically. For such trials several study designs have been suggested that use the placebo group for testing 'assay sensitivity', i.e. the ability of the trial to replicate efficacy. Should the reference fail in the given trial, then non-inferiority could also be shown with an ineffective experimental treatment and hence becomes useless. In this paper we extend the so called Koch-R\"ohmel design where a proof of efficacy for the experimental treatment is required in order to qualify the non-inferiority test. While efficacy of the experimental treatment is an indication for assay sensitivity, it does not guarantee that the reference is sufficient efficient to let the non-inferiority claim be meaningful. It has therefore been suggested to adaptively test non-inferiority only if the reference demonstrates superiority to placebo and otherwise to test $\delta$-superiority of the experimental treatment over placebo, where $\delta$ is chosen in such a way that it provides proof of non-inferiority with regard to the reference's historical effect. In this paper we extend the previous work by complementing its adaptive test with compatible simultaneous confidence intervals. Confidence intervals are commonly used and suggested by regulatory guidelines for non-inferiority trials. We show how to adopt different approaches to simultaneous confidence intervals from the literature to the setting of three-arm non-inferiority trials and compare these methods in a simulation study. Finally we apply these methods to a real clinical trial example.

For indications where only unstable reference treatments are available and use of placebo is ethically justified, three-arm `gold standard' designs with an experimental, reference and placebo arm are recommended for non-inferiority trials. In such designs, the demonstration of efficacy of the reference or experimental treatment is a requirement. They have the disadvantage that only little can be concluded from the trial if the reference fails to be efficacious. To overcome this, we investigate novel single-stage, adaptive test strategies where non-inferiority is tested only if the reference shows sufficient efficacy and otherwise $\delta$-superiority of the experimental treatment over placebo is tested. With a properly chosen superiority margin, $\delta$-superiority indirectly shows non-inferiority. We optimize the sample size for several decision rules and find that the natural, data driven test strategy, which tests non-inferiority if the reference's efficacy test is significant, leads to the smallest overall and placebo sample sizes. We proof that under specific constraints on the sample sizes, this procedure controls the family-wise error rate. All optimal sample sizes are found to meet this constraint. We finally show how to account for a relevant placebo drop-out rate in an efficient way and apply the new test strategy to a real life data set.

We introduce a new multiple type I error criterion for clinical trials with multiple populations. Such trials are of interest in precision medicine where the goal is to develop treatments that are targeted to specific sub-populations defined by genetic and/or clinical biomarkers. The new criterion is based on the observation that not all type I errors are relevant to all patients in the overall population. If disjoint sub-populations are considered, no multiplicity adjustment appears necessary, since a claim in one sub-population does not affect patients in the other ones. For intersecting sub-populations we suggest to control the average multiple type error rate, i.e. the probably that a randomly selected patient will be exposed to an inefficient treatment. We call this the population-wise error rate, exemplify it by a number of examples and illustrate how to control it with an adjustment of critical boundaries or adjusted p-values. We furthermore define corresponding simultaneous confidence intervals. We finally illustrate the power gain achieved by passing from family-wise to population-wise error rate control with two simple examples and a recently suggest multiple testing approach for umbrella trials., Comment: 21 pages, 6 figures

The research on and application of artificial intelligence (AI) has triggered a comprehensive scientific, economic, social and political discussion. Here we argue that statistics, as an interdisciplinary scientific field, plays a substantial role both for the theoretical and practical understanding of AI and for its future development. Statistics might even be considered a core element of AI. With its specialist knowledge of data evaluation, starting with the precise formulation of the research question and passing through a study design stage on to analysis and interpretation of the results, statistics is a natural partner for other disciplines in teaching, research and practice. This paper aims at contributing to the current discussion by highlighting the relevance of statistical methodology in the context of AI development. In particular, we discuss contributions of statistics to the field of artificial intelligence concerning methodological development, planning and design of studies, assessment of data quality and data collection, differentiation of causality and associations and assessment of uncertainty in results. Moreover, the paper also deals with the equally necessary and meaningful extension of curricula in schools and universities.

In a Phase II dose-finding study with a placebo control, a new drug with several dose levels is compared with a placebo to test for the effectiveness of the new drug. The main focus of such studies often lies in the characterization of the dose-response relationship followed by the estimation of a target dose that leads to a clinically relevant effect over the placebo. This target dose is known as the minimum effective dose (MED) in a drug development study. Several approaches exist that combine multiple comparison procedures with modeling techniques to efficiently estimate the dose-response model and thereafter select the target dose. Despite the flexibility of the existing approaches, they cannot completely address the model uncertainty in the model-selection step and may lead to target dose estimates that are biased. In this article, we propose two new MED estimation approaches based on weighted regression modeling that are robust against deviations from the dose-response model assumptions. These approaches are compared with existing approaches with regard to their accuracy in point and interval estimation of the MED. We illustrate by a simulation study that by integrating one of the new dose estimation approaches with the existing dose-response profile estimation approaches one can take into account the uncertainty of the model selection step.

The COVID-19 pandemic has led to an unprecedented response in terms of clinical research activity. An important part of this research has been focused on randomized controlled clinical trials to evaluate potential therapies for COVID-19. The results from this research need to be obtained as rapidly as possible. This presents a number of challenges associated with considerable uncertainty over the natural history of the disease and the number and characteristics of patients affected, and the emergence of new potential therapies. These challenges make adaptive designs for clinical trials a particularly attractive option. Such designs allow a trial to be modified on the basis of interim analysis data or stopped as soon as sufficiently strong evidence has been observed to answer the research question, without compromising the trial's scientific validity or integrity. In this paper we describe some of the adaptive design approaches that are available and discuss particular issues and challenges associated with their use in the pandemic setting. Our discussion is illustrated by details of four ongoing COVID-19 trials that have used adaptive designs.

Major advances have been made regarding the utilization of artificial intelligence in health care. In particular, deep learning approaches have been successfully applied for automated and assisted disease diagnosis and prognosis based on complex and high-dimensional data. However, despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common. Automated medical testing devices based on machine-learned prediction models should thus undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co-primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, i.e. to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family-wise error rate. Moreover, we demonstrate in extensive simulation studies that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate. For this purpose, our algorithm optimizes the expected final model performance given previous (hold-out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are implemented., Comment: 32 pages, 5 figures; v2: minor linguistic revision, no content-related changes

Linear regression is a frequently used tool in statistics, however, its validity and interpretability relies on strong model assumptions. While robust estimates of the coefficients' covariance extend the validity of hypothesis tests and confidence intervals, a clear interpretation of the coefficients is lacking if the mean structure of the model is miss-specified. We therefore suggest a new intuitive and mathematical rigorous interpretation of the coefficients that is independent from specific model assumptions. It relies on a new population based measure of association. The idea is to quantify how much the population mean of the dependent variable Y can be changed by changing the distribution of the independent variable X. Restriction to linear functions for the distributional changes in X provides the link to linear regression. It leads to a conservative approximation of the newly defined and generally non-linear measure of association. The conservative linear approximation can then be estimated by linear regression. We show how this interpretation can be extended to multiple regression and how far and in which sense it leads to an adjustment for confounding. We point to perspectives for new analysis strategies and illustrate the utility and limitations of the new interpretation and strategies by examples and simulations.