Search

Objective Recently Doi et al. argued that risk ratios should be replaced with odds ratios in clinical research. We disagreed, and empirically documented the lack of portability of odds ratios, while Doi et al. defended their position. In this response we highlight important errors in their position. Study design and setting We counter Doi et al.’s arguments by further examining the correlations of odds ratios, and risk ratios, with baseline risks in 20,198 meta-analyses from the Cochrane Database of Systematic Reviews. Results Doi et al.’s claim that odds ratios are portable is invalid because 1) their reasoning is circular: they assume a model under which the odds ratio is constant and show that under such a model the odds ratio is portable; 2) the method they advocate to convert odds ratios to risk ratios is biased; 3) their empirical example is readily-refuted by counter-examples of meta-analyses in which the risk ratio is portable but the odds ratio isn't; and 4) they fail to consider the causal determinants of meta-analytic inclusion criteria: Doi et al. mistakenly claim that variation in odds ratios with different baseline risks in meta-analyses is due to collider bias. Empirical comparison between the correlations of odds ratios, and risk ratios, with baseline risks show that the portability of odds ratios and risk ratios varies across settings. Conclusion The suggestion to replace risk ratios with odds ratios is based on circular reasoning and a confusion of mathematical and empirical results. It is especially misleading for meta-analyses and clinical guidance. Neither the odds ratio nor the risk ratio is universally portable. To address this lack of portability, we reinforce our suggestion to report variation in effect measures conditioning on modifying factors such as baseline risk; understanding such variation is essential to patient-centered practice.

Randomised controlled trials are often presented as the gold standard for testing new medical treatments. In the early stages of research, however, reports from single trials are likely to show exaggerated effect estimates. Erik van Zwet, Simon Schwab and Sander Greenland explain why – and propose a remedy

To prevent statistical misinterpretations, it has long been advised to focus on estimation instead of statistical testing. This sound advice brings with it the need to choose the outcome and effect measures on which to focus. Measures based on odds or their logarithms have often been promoted due to their pleasing statistical properties, but have an undesirable property for risk summarization and communication: Noncollapsibility, defined as a failure of the measure when taken on a group to equal a simple average of the measure when taken on the group's members or subgroups. The present note illustrates this problem with a basic numeric example involving the odds, which is not collapsible when the odds vary across individuals and are not low in all subgroups. Its sequel will illustrate how this problem is amplified in odds ratios and logistic regression.

Quantitative bias analyses allow researchers to adjust for uncontrolled confounding, given specification of certain bias parameters. When researchers are concerned about unknown confounders, plausible values for these bias parameters will be difficult to specify. Ding and VanderWeele developed bounding factor and E-value approaches that require the user to specify only some of the bias parameters. We describe the mathematical meaning of bounding factors and E-values as well as the plausibility of these methods in an applied context. We encourage researchers to pay particular attention to the assumption made, when using E-values, that the prevalence of the uncontrolled confounder among the exposed is 100% (or, equivalently, the prevalence of the exposure among those without the confounder is 0%). We contrast methods that attempt to bound biases or effects and alternative approaches such as quantitative bias analysis. We provide an example where failure to make this distinction led to erroneous statements. If the primary concern in an analysis is with known but unmeasured potential confounders, then E-values are not needed and may be misleading. In cases where the concern is with unknown confounders, the E-value assumption of an extreme possible prevalence of the confounder limits its practical utility.

It is well known that the statistical analyses in health-science and medical journals are frequently misleading or even wrong. Despite many decades of reform efforts by hundreds of scientists and statisticians, attempts to fix the problem by avoiding obvious error and encouraging good practice have not altered this basic situation. Statistical teaching and reporting remain mired in damaging yet editorially enforced jargon of "significance", "confidence", and imbalanced focus on null (no-effect or "nil") hypotheses, leading to flawed attempts to simplify descriptions of results in ordinary terms. A positive development amidst all this has been the introduction of interval estimates alongside or in place of significance tests and P-values, but intervals have been beset by similar misinterpretations. Attempts to remedy this situation by calling for replacement of traditional statistics with competitors (such as pure-likelihood or Bayesian methods) have had little impact. Thus, rather than ban or replace P-values or confidence intervals, we propose to replace traditional jargon with more accurate and modest ordinary-language labels that describe these statistics as measures of compatibility between data and hypotheses or models, which have long been in use in the statistical modeling literature. Such descriptions emphasize the full range of possibilities compatible with observations. Additionally, a simple transform of the P-value called the surprisal or S-value provides a sense of how much or how little information the data supply against those possibilities. We illustrate these reforms using some examples from a highly charged topic: trials of ivermectin treatment for Covid-19.

Lash et al. (Am J Epidemiol. 2021;190(8):1604–1612) have presented detailed critiques of 3 bias analyses that they identify as “suboptimal.” This identification raises the question of what “optimal” means for bias analysis, because it is practically impossible to do statistically optimal analyses of typical population studies—with or without bias analysis. At best the analysis can only attempt to satisfy practice guidelines and account for available information both within and outside the study. One should not expect a full accounting for all sources of uncertainty; hence, interval estimates and distributions for causal effects should never be treated as valid uncertainty assessments—they are instead only example analyses that follow from collections of often questionable assumptions. These observations reinforce those of Lash et al. and point to the need for more development of methods for judging bias-parameter distributions and utilization of available information.

The "replication crisis" has been attributed to perverse incentives that lead to selective reporting and misinterpretations of P-values and confidence intervals. A crude fix offered for this problem is to lower testing cut-offs (α levels), either directly or in the form of null-biased multiple comparisons procedures such as naïve Bonferroni adjustments. Methodologists and statisticians have expressed positions that range from condemning all such procedures to demanding their application in almost all analyses. Navigating between these unjustifiable extremes requires defining analysis goals precisely enough to separate inappropriate from appropriate adjustments. To meet this need, I here review issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons. I also review considerations that should be made when examining arguments for and against modifications of decision cut-offs and adjustments for multiple comparisons. The goal is to provide researchers a better understanding of what is assumed by each side and to enable recognition of hidden assumptions. Basic issues of goal specification and error costs are illustrated with simple fixed cut-off hypothesis testing scenarios. These illustrations show how adjustment choices are extremely sensitive to implicit decision costs, making it inevitable that different stakeholders will vehemently disagree about what is necessary or appropriate. Because decisions cannot be justified without explicit costs, resolution of inference controversies is impossible without recognising this sensitivity. Pre-analysis statements of funding, scientific goals, and analysis plans can help counter demands for inappropriate adjustments, and can provide guidance as to what adjustments are advisable. Hierarchical (multilevel) regression methods (including Bayesian, semi-Bayes, and empirical-Bayes methods) provide preferable alternatives to conventional adjustments, insofar as they facilitate use of background information in the analysis model, and thus can provide better-informed estimates on which to base inferences and decisions.

In 2020, prior to COVID-19 vaccine rollout, the Brighton Collaboration created a priority list, endorsed by the World Health Organization, of potential adverse events relevant to COVID-19 vaccines. We adapted the Brighton Collaboration list to evaluate serious adverse events of special interest observed in mRNA COVID-19 vaccine trials.Secondary analysis of serious adverse events reported in the placebo-controlled, phase III randomized clinical trials of Pfizer and Moderna mRNA COVID-19 vaccines in adults (NCT04368728 and NCT04470427), focusing analysis on Brighton Collaboration adverse events of special interest.Pfizer and Moderna mRNA COVID-19 vaccines were associated with an excess risk of serious adverse events of special interest of 10.1 and 15.1 per 10,000 vaccinated over placebo baselines of 17.6 and 42.2 (95 % CI -0.4 to 20.6 and -3.6 to 33.8), respectively. Combined, the mRNA vaccines were associated with an excess risk of serious adverse events of special interest of 12.5 per 10,000 vaccinated (95 % CI 2.1 to 22.9); risk ratio 1.43 (95 % CI 1.07 to 1.92). The Pfizer trial exhibited a 36 % higher risk of serious adverse events in the vaccine group; risk difference 18.0 per 10,000 vaccinated (95 % CI 1.2 to 34.9); risk ratio 1.36 (95 % CI 1.02 to 1.83). The Moderna trial exhibited a 6 % higher risk of serious adverse events in the vaccine group: risk difference 7.1 per 10,000 (95 % CI -23.2 to 37.4); risk ratio 1.06 (95 % CI 0.84 to 1.33). Combined, there was a 16 % higher risk of serious adverse events in mRNA vaccine recipients: risk difference 13.2 (95 % CI -3.2 to 29.6); risk ratio 1.16 (95 % CI 0.97 to 1.39).The excess risk of serious adverse events found in our study points to the need for formal harm-benefit analyses, particularly those that are stratified according to risk of serious COVID-19 outcomes. These analyses will require public release of participant level datasets.

The 1970s and 1980s saw the appearance of many papers on the topics of synergy, antagonism, and similar concepts of causal interactions and interdependence of effects, with a special emphasis on distinguishing these concepts from that of statistical interaction - the need for a product term in a model. As an example, Miettinen defined "synergism" as "the existence of instances in which both risk factors are needed for the effect", whereas "antagonism" is where "at least one [factor] can block the solo effect of the other". In response, Greenland and Poole constructed a systematic analysis of 16 possible individual response patterns in a deterministic causal model for two binary exposure variables, and showed how these patterns can be mapped onto nine types of sufficient causes, which in turn can be simplified into four intuitive categories. Although these and other papers recognized that epidemiology cannot directly study biological mechanisms underlying interaction, they showed how it can usefully study causal and preventive interdependence - which, despite its mechanistic agnosticism, has important implications for clinical decision making as well as for public health.

Key Points Question Do children with atopic dermatitis experience impaired sleep duration and sleep quality throughout childhood, and do disease severity and activity affect their sleep? Findings In this longitudinal cohort study of 13 988 children, atopic dermatitis was statistically significantly associated with impaired sleep quality but not sleep duration throughout childhood. Sleep impairment was more common among children with more severe disease and with comorbid asthma or allergic rhinitis, and the risk remained elevated even among children with mild and inactive atopic dermatitis. Meaning These findings suggest that clinicians should consider sleep quality among all children with atopic dermatitis, especially those with comorbid asthma or allergic rhinitis and severe disease; it appears interventions to improve sleep quality are needed for this population., Importance Pruritus, a hallmark of atopic dermatitis (AD), is thought to disrupt sleep, yet little is known about how variations in disease activity and severity of this common childhood condition may be associated with sleep patterns over time. Objective To determine whether children with active AD have impaired sleep duration and quality at multiple time points throughout childhood and whether disease severity affects sleep outcomes. Design, Setting, and Participants This longitudinal cohort study used data of children enrolled in the Avon Longitudinal Study of Parents and Children, a population-based birth cohort in Avon, United Kingdom. Participants were children (N = 13 988) alive at 1 year and followed up with repeated measures of self-reported AD and sleep through 16 years of age. This study was based on data collected from 1990 to 2008. Data analysis was performed from September 2017 to September 2018. Main Outcomes and Measures Standardized measure of sleep duration and composite measure of sleep quality, including nighttime awakenings, early morning awakenings, difficulty falling asleep, and nightmares, were repeated at multiple time points between 2 and 16 years of age. Results The study sample comprised 13 988 children (7220 male [51.6%]) followed up for a median (interquartile range [IQR]) duration of 11 (5-14) years. Of this total, 4938 children (35.3%) met the definition of having atopic dermatitis between 2 and 16 years of age. Total sleep duration was similar between children with active AD and without AD at all ages, and the average estimated difference across childhood was a clinically negligible difference of 2 minutes less per day for children with AD (95% CI, −4 to 0 minutes). In contrast, children with active AD were more likely to report worse sleep quality at all time points, with a nearly 50% higher odds of experiencing more sleep-quality disturbances (adjusted odds ratio [aOR], 1.48; 95% CI, 1.33 to 1.66). Children with more severe active disease (quite bad or very bad AD: aOR, 1.68; 95% CI, 1.42 to 1.98) and with comorbid asthma or allergic rhinitis (aOR, 1.79; 95% CI, 1.54 to 2.09) had worse sleep quality. However, even children with mild AD (OR, 1.40; 95% CI, 1.27 to 1.54) or inactive AD (OR, 1.41; 95% CI, 1.28 to 1.55) had statistically significantly increased odds of impaired sleep quality. Conclusions and Relevance Atopic dermatitis appeared to be associated with impaired sleep quality throughout childhood; thus, it is suggested that clinicians should consider sleep quality among all children with AD, especially those with comorbid asthma or allergic rhinitis and severe disease, and that interventions to improve sleep quality are needed., This cohort study examines the prevalence and severity of atopic dermatitis as well as sleep duration and quality at multiple periods among young children enrolled in the Avon Longitudinal Study of Parents and Children in the United Kingdom.

Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and $P$-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review simple aids to statistical interpretations. These aids emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions. We use the Shannon transform of the $P$-value $p$, also known as the binary surprisal or $S$-value $s=-\log_{2}(p)$, to measure the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret $P$-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study. In line with other recent recommendations, we advise that teaching materials and research reports discuss $P$-values as measures of compatibility rather than significance, compute $P$-values for alternative hypotheses whenever they are computed for null hypotheses, and interpret interval estimates as showing values of high compatibility with data, rather than regions of confidence. Our recommendations emphasize cognitive devices for displaying the compatibility of the observed data with various hypotheses of interest, rather than focusing on single hypothesis tests or interval estimates. We believe these simple reforms are well worth the minor effort they require., 22 pages; 5 figures; 2 tables; 94 references; Published at BMC Medical Research Methodology

It has long been argued that we need to consider much more than an observed point estimate and a p-value to understand statistical results. One of the most persistent misconceptions about p-values is that they are necessarily calculated assuming a null hypothesis of no effect is true. Instead, p-values can and should be calculated for multiple hypothesized values for the effect size. For example, a p-value function allows us to visualize results continuously by examining how the p-value varies as we move across possible effect sizes. For more focused discussions, a 95% confidence interval shows the subset of possible effect sizes that have p-values larger than 0.05 as calculated from the same data and the same background statistical assumptions. In this sense a confidence interval can be taken as showing the effect sizes that are most compatible with the data, given the assumptions, and thus may be better termed a compatibility interval. The question that should then be asked is whether any or all of the effect sizes within the interval are substantial enough to be of practical importance.

This chapter provides an introduction to causal inference theory for public health research. Causal inference can be viewed as a prediction problem, addressing the question of what the likely outcome will be under one action vs. an alternative action. To answer this question usefully requires clarity and precision in both the statement of the causal hypothesis and the techniques used to attempt an answer. This chapter reviews considerations that have been invoked in discussions of causality based on epidemiologic evidence. It then describes the potential-outcome (counterfactual) framework for cause and effect, which shows how measures of effect and association can be distinguished. The potential-outcome framework illustrates problems inherent in attempts to quantify the changes in health expected under different actions or interventions. The chapter concludes with a discussion of how research findings may be translated into policy.

Personalized medicine asks if a new treatment will help a particular patient, rather than if it improves the average response in a population. Without a causal model to distinguish these questions, interpretational mistakes arise. These mistakes are seen in an article by Demidenko [2016] that recommends the “D-value,” which is the probability that a randomly chosen person from the new-treatment group has a higher value for the outcome than a randomly chosen person from the control-treatment group. The abstract states “The D-value has a clear interpretation as the proportion of patients who get worse after the treatment” with similar assertions appearing later. We show these statements are incorrect because they require assumptions about the potential outcomes which are neither testable in randomized experiments nor plausible in general. The D-value will not equal the proportion of patients who get worse after treatment if (as expected) those outcomes are correlated. Independence of potential outcomes is unrealistic and eliminates any personalized treatment effects; with dependence, the D-value can even imply treatment is better than control even though most patients are harmed by the treatment. Thus, D-values are misleading for personalized medicine. To prevent misunderstandings, we advise incorporating causal models into basic statistics education.

The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of “significance levels” and “P-values” in authoritative sources, and the consequent misinterp...

A previous note illustrated how the odds of an outcome have an undesirable property for risk summarization and communication: Noncollapsibility, defined as a failure of a group measure to represent a simple average of the measure over individuals or subgroups. The present sequel discusses how odds ratios amplify odds noncollapsibility and provides a basic numeric illustration of how noncollapsibility differs from confounding of effects (with which it is often confused). It also draws a connection of noncollapsibility to sparse-data bias in logistic, log-linear, and proportional-hazards regression.

IntroductionDespite a recent meta-analysis including 31 randomised controlled trials comparing methadone and buprenorphine for the treatment of opioid use disorder, important knowledge gaps remain regarding the long-term effectiveness of different treatment modalities across individuals, including rigorously collected data on retention rates and other treatment outcomes. Evidence from real-world data represents a valuable opportunity to improve personalised treatment and patient-centred guidelines for vulnerable populations and inform strategies to reduce opioid-related mortality. Our objective is to determine the comparative effectiveness of methadone versus buprenorphine/naloxone, both overall and within key populations, in a setting where both medications are simultaneously available in office-based practices and specialised clinics.Methods and analysisWe propose a retrospective cohort study of all adults living in British Columbia receiving opioid agonist treatment (OAT) with methadone or buprenorphine/naloxone between 1 January 2008 and 30 September 2018. The study will draw on seven linked population-level administrative databases. The primary outcomes include retention in OAT and all-cause mortality. We will determine the effectiveness of buprenorphine/naloxone vs methadone using intention-to-treat and per-protocol analyses—the former emulating flexible-dose trials and the latter focusing on the comparison of the two medication regimens offered at the optimal dose. Sensitivity analyses will be used to assess the robustness of results to heterogeneity in the patient population and threats to internal validity.Ethics and disseminationThe protocol, cohort creation and analysis plan have been approved and classified as a quality improvement initiative exempt from ethical review (Providence Health Care Research Institute and the Simon Fraser University Office of Research Ethics). Dissemination is planned via conferences and publications, and through direct engagement and collaboration with entities that issue clinical guidelines, such as professional medical societies and public health organisations.

Disease surveillance forms the basis for response to epidemics. COVID-19 provides a modern example of why the classic mantra of “person, place, and time” remains crucial: epidemic control requires ...

Measures of information and surprise, such as the Shannon information value (S value), quantify the signal present in a stream of noisy data. We illustrate the use of such information measures in the context of interpreting P values as compatibility indices. S values help communicate the limited information supplied by conventional statistics and cast a critical light on cutoffs used to judge and construct those statistics. Misinterpretations of statistics may be reduced by interpreting P values and interval estimates using compatibility concepts and S values instead of “significance” and “confidence.”

Study size has typically been planned based on statistical power and therefore has been heavily influenced by the philosophy of statistical hypothesis testing. A worthwhile alternative is to plan study size based on precision, for example by aiming to obtain a desired width of a confidence interval for the targeted effect. This article presents formulas for planning the size of an epidemiologic study based on the desired precision of the basic epidemiologic effect measures.

We thank Dr. Karp for his interest [1] in our paper [2]. We agree on some points, but our theoretical description differs from his in ways leading to important divergences for teaching and practice. We also see a danger of overextending abstract theory (with its inevitable and extensive simplifications) into practice [3], especially when the practical questions are causal but the theory applied lacks an explicit, sound longitudinal causal model to address these questions. As we will explain, a defect in the “study base” theory Dr. Karp adopts as a foundational belief system is that it takes as a foundation a parameter affected by baseline risk factors—including exposure when that has effects on follow-up or disease. It consequently leads to biases and misconceptions of the sort documented elsewhere [4, 5] and below, which require a coherent theory of longitudinal causality to address. Our divergence from Dr. Karp thus raises the issue of the role of theory and methods in research, although matching serves to illustrate our points in a familiar epidemiologic context.

Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly crucial effects. Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly crucial effects.

Misconceptions about the impact of case-control matching remain common. We discuss several subtle problems associated with matched case-control studies that do not arise or are minor in matched cohort studies: (1) matching, even for non-confounders, can create selection bias; (2) matching distorts dose-response relations between matching variables and the outcome; (3) unbiased estimation requires accounting for the actual matching protocol as well as for any residual confounding effects; (4) for efficiency, identically matched groups should be collapsed; (5) matching may harm precision and power; (6) matched analyses may suffer from sparse-data bias, even when using basic sparse-data methods. These problems support advice to limit case-control matching to a few strong well-measured confounders, which would devolve to no matching if no such confounders are measured. On the positive side, odds ratio modification by matched variables can be assessed in matched case-control studies without further data, and when one knows either the distribution of the matching factors or their relation to the outcome in the source population, one can estimate and study patterns in absolute rates. Throughout, we emphasize distinctions from the more intuitive impacts of cohort matching.

Marginal structural models for time-fixed treatments fit using inverse-probability weighted estimating equations are increasingly popular. Nonetheless, the resulting effect estimates are subject to finite-sample bias when data are sparse, as is typical for large-sample procedures. Here we propose a semi-Bayes estimation approach which penalizes or shrinks the estimated model parameters to improve finite-sample performance. This approach uses simple symmetric data-augmentation priors. Limited simulation experiments indicate that the proposed approach reduces finite-sample bias and improves confidence-interval coverage when the true values lie within the central "hill" of the prior distribution. We illustrate the approach with data from a nonexperimental study of HIV treatments.

Separation is encountered in regression models with a discrete outcome (such as logistic regression) where the covariates perfectly predict the outcome. It is most frequent under the same conditions that lead to small-sample and sparse-data bias, such as presence of a rare outcome, rare exposures, highly correlated covariates, or covariates with strong effects. In theory, separation will produce infinite estimates for some coefficients. In practice, however, separation may be unnoticed or mishandled because of software limits in recognizing and handling the problem and in notifying the user. We discuss causes of separation in logistic regression and describe how common software packages deal with it. We then describe methods that remove separation, focusing on the same penalized-likelihood techniques used to address more general sparse-data problems. These methods improve accuracy, avoid software problems, and allow interpretation as Bayesian analyses with weakly informative priors. We discuss likelihood penalties, including some that can be implemented easily with any software package, and their relative advantages and disadvantages. We provide an illustration of ideas and methods using data from a case-control study of contraceptive practices and urinary tract infection.

I present an overview of two methods controversies that are central to analysis and inference: That surrounding causal modeling as reflected in the "causal inference" movement, and that surrounding null bias in statistical methods as applied to causal questions. Human factors have expanded what might otherwise have been narrow technical discussions into broad philosophical debates. There seem to be misconceptions about the requirements and capabilities of formal methods, especially in notions that certain assumptions or models (such as potential-outcome models) are necessary or sufficient for valid inference. I argue that, once these misconceptions are removed, most elements of the opposing views can be reconciled. The chief problem of causal inference then becomes one of how to teach sound use of formal methods (such as causal modeling, statistical inference, and sensitivity analysis), and how to apply them without generating the overconfidence and misinterpretations that have ruined so many statistical practices.

To the Editor of JAMA Dr Ioannidis writes against our proposals to abandon statistical significance in scientific reasoning and publication, as endorsed in the editorial of a recent special issue of an American Statistical Association journal devoted to moving to a “post p Nonetheless, we disagree that a statistical significance-based “filtering process is useful to avoid drowning in noise” in science and instead view such filtering as harmful. First, the implicit rule to not publish nonsignificant results biases the literature with overestimated effect sizes and encourages “hacking” to get significance. Second, nonsignificant results are often wrongly treated as zero. Third, significant results are often wrongly treated as truth rather than as the noisy estimates they are, thereby creating unrealistic expectations of replicability. Fourth, filtering on statistical significance provides no guarantee against noise. Instead, it amplifies noise because the quantity on which the filtering is based (the p-value) is itself extremely noisy and is made more so by dichotomizing it. We also disagree that abandoning statistical significance will reduce science to “a state of statistical anarchy.” Indeed, the journal Epidemiology banned statistical significance in 1990 and is today recognized as a leader in the field. Valid synthesis requires accounting for all relevant evidence—not just the subset that attained statistical significance. Thus, researchers should report more, not less, providing estimates and uncertainty statements for all quantities, justifying any exceptions, and considering ways the results are wrong. Publication criteria should be based on evaluating study design, data quality, and scientific content—not statistical significance. Decisions are seldom necessary in scientific reporting. However, when they are required (as in clinical practice), they should be made based on the costs, benefits, and likelihoods of all possible outcomes, not via arbitrary cutoffs applied to statistical summaries such as p-values which capture little of this picture. The replication crisis in science is not the product of the publication of unreliable findings. The publication of unreliable findings is unavoidable: as the saying goes, if we knew what we were doing, it would not be called research. Rather, the replication crisis has arisen because unreliable findings are presented as reliable.

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced.

There are now many published applications of causal (structural) models for estimating effects of time-varying exposures in the presence of confounding by earlier exposures and confounders affected by earlier exposures. Results from these models can be highly sensitive to inclusion of lagged and baseline exposure terms for different visits. This sensitivity is often overlooked in practice; moreover, results from these models are not directly comparable to results from conventional time-dependent regression models, because the latter do not estimate the same causal parameter even when no bias is present. We thus explore the implications of including lagged and baseline exposure terms in causal and regression models, using a public data set (Caerphilly Heart Disease Study in the United Kingdom, 1979–1998) relating smoking to cardiovascular outcomes.

There is a massive crisis of confidence in statistical inference, which has largely been attributed to overemphasis on and abuse of hypothesis testing. Much of the abuse stems from failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Unedited and unselected results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. Considerable non-replication is thus to be expected even with honest and complete reporting practices, and generalizations from single studies are rarely if ever warranted. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between model predictions and data, rather than as generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, and what analysis methods were used.

Instrumental-variable (IV) methods were invented over 70 years ago, but remain uncommon in epidemiology. Over the past decade or so, non-parametric versions of IV methods have appeared that connect IV methods to causal and measurement-error models important in epidemiological applications. This paper provides an introduction to those developments, illustrated by an application of IV methods to non-parametric adjustment for non-compliance in randomized trials.

We present a command, penlogit, for approximate Bayesian logistic regression using penalized likelihood estimation via data augmentation. This command automatically adds specific prior-data records to a dataset. These records are computed so that they generate a penalty function for the log likelihood of a logistic model, which equals (up to an additive constant) a set of independent log prior distributions on the model parameters. This command overcomes the necessity of relying on specialized software and statistical tools (such as Markov chain Monte Carlo) for fitting Bayesian models, and allows one to assess the information content of a prior in terms of the data that would be required to generate the prior as a likelihood function. The command produces data equivalent to normal and generalized log- F priors for the model parameters, providing flexible translation of background information into prior data, which allows calculation of approximate posterior medians and intervals from ordinary maximum likelihood programs. We illustrate the command through an example using data from an observational study of neonatal mortality.

This article discusses definitions and concepts of effect modification, interaction, synergism, and related concepts and terms. Keywords: antagonism; causal coaction; effect-measure modification; effect modification; heterogeneity of effect; interaction; synergism

Job exposure matrices (JEMs) are used to measure exposures based on information about particular jobs and tasks. JEMs are especially useful when individual exposure data cannot be obtained. Nonetheless, there may be other workplace exposures associated with the study disease that are not measured in available JEMs. When these exposures are also associated with the exposures measured in the JEM, biases due to uncontrolled confounding will be introduced. Furthermore, individual exposures differ from JEM measurements due to differences in job conditions and worker practices. Uncertainty may also be present at the assessor level since exposure information for each job may be imprecise or incomplete. Assigning individuals a fixed exposure determined by the JEM ignores these uncertainty sources. We examine the uncertainty displayed by bias analyses in a study of occupational electric shocks, occupational magnetic fields, and amyotrophic lateral sclerosis.