Search

Researchers in the single-case design tradition have debated the size and importance of the observed autocorrelations in those designs. All of the past estimates of the autocorrelation in that literature have taken the observed autocorrelation estimates as the data to be used in the debate. However, estimates of the autocorrelation are subject to great sampling error when the design has a small number of time points, as is typically the situation in single-case designs. Thus, a given observed autocorrelation may greatly over- or underestimate the corresponding population parameter. This article presents Bayesian estimates of the autocorrelation that greatly reduce the role of sampling error, as compared to past estimators. Simpler empirical Bayes estimates are presented first, in order to illustrate the fundamental notions of autocorrelation sampling error and shrinkage, followed by fully Bayesian estimates, and the difference between the two is explained. Scripts to do the analyses are available as supplemental materials. The analyses are illustrated using two examples from the single-case design literature. Bayesian estimation warrants wider use, not only in debates about the size of autocorrelations, but also in statistical methods that require an independent estimate of the autocorrelation to analyze the data. (Contains 2 figures and 1 table.)

Single case designs are a set of research methods for evaluating treatment effects by assigning different treatments to the same individual and measuring outcomes over time and are used across fields such as behavior analysis, clinical psychology, special education, and medicine. Emerging standards for single case designs have focused attention on the need for effect sizes for summarizing and meta-analyzing findings from the designs; although many effect size measures have been proposed, there is little consensus regarding their use. This article proposes a new effect size measure for single case research that is directly comparable with the standardized mean difference (Cohen's "d") often used in between-subjects designs. Techniques are provided for estimating the new effect size, as well as its variance, from balanced or unbalanced treatment reversal designs. The proposed estimation methods are further evaluated using a simulation study and then demonstrated in two applications. The following are appended: (1) Derivation of the exact expectation and variance of ES from the moments of D and the inverse moments of S[superscript 2]; (2) Derivation of the effect size estimator for the general (AB)[superscript k] design; and (3) Formulas for estimators phi and rho in the unbalanced (AB)[superscript k] design. (Contains 1 table and 2 figures.)

The purpose of this study was to identify the characteristics of a representative sample of single-case designs that appear in the published literature. The study located, digitized, and coded all 809 single-case designs appearing in 113 studies in the year 2008 in 21 journals in a variety of fields in psychology and education. Coded variables included the specific kind of design, number of cases per study, number of outcomes, data points and phases per case, and autocorrelations for each case. Results indicate that although studies of the effects of interventions are a minority in these journals, within that category, single-case designs are used more frequently than randomized or nonrandomized experiments. The modal study uses a multiple-baseline design with 20 data points for each of three or four cases, where the aim of the intervention is to increase the frequency of a desired behavior; but these characteristics vary widely over studies. The average autocorrelation is near to but significantly different from zero; but autocorrelations are significantly heterogeneous. Results show that single-case designs are widely used to study the effects of interventions, and suggest that more work needs to be done to find ways to include such studies in reviews of evidence-based practice. (Contains 4 tables, 1 footnote, and 5 figures.)

Currently, the design standards for single-case experimental designs (SCEDs) are based on validity considerations as prescribed by the What Works Clearinghouse. However, there is a need for design considerations such as power based on statistical analyses. We compute and derive power using computations for (AB)k designs with multiple cases which are common in SCEDs. Our computations show that effect size has the maximum impact on power followed by the number of subjects and then the number of phase reversals. An effect size of 0.75 or higher, at least one set of phase reversals (i.e., where k > 1), and at least three subjects showed high power. The latter two conditions agree with current standards about either having at least an ABAB design or a multiple baseline design with three subjects to meet design standards. An effect size of 0.75 or higher is not uncommon in SCEDs either. Autocorrelations, the number of time-points per phase, and intraclass correlations had a smaller but non-negligible impact on power. In sum, power analyses in the present study show that conditions to meet power requirements are not unreasonable in SCEDs. The software code to compute power is available on GitHub for the use of the reader. [ABSTRACT FROM AUTHOR]

The published literature often underrepresents studies that do not find evidence for a treatment effect; this is often called "publication bias." Literature reviews that fail to include such studies may overestimate the size of an effect. Only a few studies have examined publication bias in single-case design (SCD) research, but those studies suggest that publication bias may occur. This study surveyed SCD researchers about publication preferences in response to simulated SCD results that show a range of small to large effects. Results suggest that SCD researchers are more likely to submit manuscripts that show large effects for publication and are more likely to recommend acceptance of manuscripts that show large effects when they act as a reviewer. A nontrivial minority of SCD researchers (4% to 15%) would drop 1 or 2 cases from the study if the effect size is small and then submit for publication. This article ends with a discussion of implications for publication practices in SCD research.

We developed a reporting guideline to provide authors with guidance about what should be reported when writing a paper for publication in a scientific journal using a particular type of research design: the single-case experimental design. This report describes the methods used to develop the Single-Case Reporting guideline In BEhavioural interventions (SCRIBE) 2016. As a result of 2 online surveys and a 2-day meeting of experts, the SCRIBE 2016 checklist was developed, which is a set of 26 items that authors need to address when writing about single-case research. This article complements the more detailed SCRIBE 2016 Explanation and Elaboration article (Tate et al., 2016) that provides a rationale for each of the items and examples of adequate reporting from the literature. Both these resources will assist authors to prepare reports of single-case research with clarity, completeness, accuracy, and transparency. They will also provide journal reviewers and editors with a practical checklist against which such reports may be critically evaluated. We recommend that the SCRIBE 2016 is used by authors preparing manuscripts describing single-case research for publication, as well as journal reviewers and editors who are evaluating such manuscripts.

This article looks at the impact of meta-analysis and then explores why meta-analysis was developed at the time and by the scholars it did in the social sciences in the 1970s. For the first problem, impact, it examines the impact of meta-analysis using citation network analysis. The impact is seen in the sciences, arts and humanities, and on such contemporaneous developments as multilevel modeling, medical statistics, qualitative methods, program evaluation, and single-case design. Using a constrained snowball sample of citations, we highlight key articles that are either most highly cited or most central to the systematic review network. Then, the article examines why meta-analysis came to be in the 1970s in the social sciences through the work of Gene Glass, Robert Rosenthal, and Frank Schmidt, each of whom developed similar theories of meta-analysis at about the same time. The article ends by explaining how Simonton's chance configuration theory and Campbell's evolutionary epistemology can illuminate why meta-analysis occurred with these scholars when it did and not in medical sciences.

In single-case research, the multiple baseline design is a widely used approach for evaluating the effects of interventions on individuals. Multiple baseline designs involve repeated measurement of outcomes over time and the controlled introduction of a treatment at different times for different individuals. This article outlines a general framework for defining effect sizes in multiple baseline designs that are directly comparable to the standardized mean difference from a between-subjects randomized experiment. The target, design-comparable effect size parameter can be estimated using restricted maximum likelihood together with a small sample correction analogous to Hedges's "g." The approach is demonstrated using hierarchical linear models that include baseline time trends and treatment-by-time interactions. A simulation compares the performance of the proposed estimator to that of an alternative, and an application illustrates the model-fitting process.

Single-case designs are a class of research methods for evaluating treatment effects by measuring outcomes repeatedly over time while systematically introducing different condition (e.g., treatment and control) to the same individual. The designs are used across fields such as behavior analysis, clinical psychology, special education, and medicine. Emerging standards for single-case designs have focused attention on methods for summarizing and meta-analyzing findings and on the need for effect sizes indices that are comparable to those used in between-subjects designs. In the previous work, we discussed how to define and estimate an effect size that is directly comparable to the standardized mean difference often used in between-subjects research based on the data from a particular type of single-case design, the treatment reversal or (AB)[superscript k] design. This paper extends the effect size measure to another type of single-case study, the multiple baseline design. We propose estimation methods for the effect size and its variance, study the estimators using simulation, and demonstrate the approach in two applications.

Although randomized studies have high internal validity, generalizability of the estimated causal effect from randomized clinical trials to real-world clinical or educational practice may be limited. We consider the implication of randomized assignment to treatment, as compared with choice of preferred treatment as it occurs in real-world conditions. Compliance, engagement, or motivation may be better with a preferred treatment, and this can complicate the generalizability of results from randomized trials. The doubly randomized preference trial (DRPT) is a hybrid randomized and nonrandomized design that allows for estimation of the causal effect of randomization versus treatment preference. In the DRPT, individuals are first randomized to either randomized assignment or choice assignment. Those in the randomized assignment group are then randomized to treatment or control, and those in the choice group receive their preference of treatment versus control. Using the potential outcomes framework, we apply the algebra of conditional independence to show how the DRPT can be used to derive an unbiased estimate of the causal effect of randomization versus preference for each of the treatment and comparison conditions. Also, we show how these results can be implemented using full matching on the propensity score. The methodology is illustrated with a DRPT of introductory psychology students who were randomized to randomized assignment or preference of mathematics versus vocabulary training. We found a small to moderate benefit of preference versus randomization with respect to the mathematics outcome for those who received mathematics training. (Contains 2 figures and 1 table.)

This article reviews several decades of the author's meta-analytic and experimental research on the conditions under which nonrandomized experiments can approximate the results from randomized experiments (REs). Several studies make clear that we can expect accurate effect estimates from the regression discontinuity design, though its statistical power is lower, it estimates a different parameter than the RE, and its analysis is considerably more complex. For other nonrandomized designs, the picture is more complex. They may yield accurate estimates if they are prospectively designed to include comprehensive and reliable measurement of the process by which participants selected into conditions, if they use large sample sizes, and if they carefully select control groups that are from the same location and with the same substantive characteristics. By contrast, we have little good reason to think that nonrandomized experiments using archival data without comprehensive selection measures are likely to yield accurate effect estimates. (Contains 1 figure.)

In this article, we review past studies comparing randomized experiments to regression discontinuity designs, mostly finding similar results, but with significant exceptions. The latter might be due to potential confounds of study characteristics with assignment method or with failure to estimate the same parameter over methods. In this study, we correct the problems by randomly assigning 588 participants to be in a randomized experiment or a regression discontinuity design in which they are otherwise treated identically, comparing results estimating both the same and different parameters. Analysis includes parametric, semiparametric, and nonparametric methods of modeling nonlinearities. Results suggest that estimates from regression discontinuity designs approximate the results of randomized experiments reasonably well but also raise the issue of what constitutes agreement between the 2 estimates. (Contains 5 figures and 4 tables.)

The effect of unreliability of measurement on propensity score (PS) adjusted treatment effects has not been previously studied. The authors report on a study simulating different degrees of unreliability in the multiple covariates that were used to estimate the PS. The simulation uses the same data as two prior studies. Shadish, Clark, and Steiner showed that a PS formed from many covariates demonstrably reduced selection bias, while Steiner, Cook, Shadish, and Clark identified the subsets of covariates from the larger set that were most effective for bias reduction. Adding different degrees of random error to these covariates in a simulation, the authors demonstrate that unreliability of measurement can degrade the ability of PSs to reduce bias. Specifically, increases in reliability only promote bias reduction, if the covariates are effective in reducing bias to begin with. Increasing or decreasing the reliability of covariates that do not effectively reduce selection bias makes no difference at all. (Contains 5 figures, 3 tables and 6 notes.)

The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known. (Contains 5 figures, 7 tables, and 2 footnotes.)

This article compares Donald Campbell's and Donald Rubin's work on causal inference in field settings on issues of epistemology, theories of cause and effect, methodology, statistics, generalization, and terminology. The two approaches are quite different but compatible, differing mostly in matters of bandwidth versus fidelity. Campbell's work demonstrates broad narrative scope that covers a wide array of concepts related to causation, with a powerful appreciation for human fallibility in making causal judgments, with a more elaborate theory of cause and generalization, and with a preference for design over analysis. Rubin's approach is a more narrow and formal quantitative analysis of effect estimation, sharing a preference for design but best known for analysis, with compelling quantitative approaches to obtaining unbiased quantitative effect estimates from nonrandomized designs and with comparatively little to say about generalization. Much could be gained by joining the emphasis on design in Campbell with the emphasis on analysis in Rubin. However, the 2 approaches also speak modestly different languages that leave some questions about their total commensurability that only continued dialogue can fully clarify.

Good quantitative evidence does not require large, aggregate group designs. The authors describe ground-breaking work in managing the conceptual and practical demands in developing meta-analytic strategies for single subject designs in an effort to add to evidence-based practice. (Contains 2 figures.)

The H. Bosch, F. Steinkamp, and E. Boller (see record 2006-08436-001) meta-analysis reaches mixed and cautious conclusions about the possibility of psychokinesis. The authors argue that, for both methodological and philosophical reasons, it is nearly impossible to draw any conclusions from this body of research. The authors do not agree that any significant effect at all, no matter how small, is fundamentally important (Bosch et al., 2006, p. 517), and they suggest that psychokinesis researchers focus either on producing larger effects or on specifying the conditions under which they would be willing to accept the null hypothesis.

The Center for Epidemiological Studies Depression scale (CES-D) was developed to assess the population prevalence of depression. The CES-D was developed and normed on an adult population. Since the CES-D's publication, various studies have both used and psychometrically assessed the scale for older adolescent populations. However, we found no report of the CES-D's psychometric properties for young adolescent populations. We conducted a confirmatory factor analysis of the CES-D with a large, young adolescent population that also included members of a number of different ethnic groups. We tested 12 models previously supported by factor analyses. Radloff's (1977) four-factor model (Depressed Affect, Positive Affect, Somatic Complaints, Interpersonal) was supported if the item "I felt that everything I did was an effort" is deleted, as this item has characteristics that are quite problematic in both the present data and in past data from adolescents. Second-order and hierarchical factor analyses fit the data well but suggested that Depressed Affect could not be distinguished from the General Depression factor. Given the high correlations among the four first-order factors, the standard scoring of the CES-D for one total score is justified.

When treatments are administered in groups, clients interact in ways that lead to violations of a key assumption of most statistical analyses-the assumption of independence of observations. The resulting dependencies, when not properly accounted for, can increase Type I errors dramatically. Of the 33 studies of group-administered treatment on the empirically supported treatments list, none appropriately analyzed their data. The current authors provide corrections that can be applied to improper analyses. After the corrections, only 12.4% to 68.2% of tests that were originally reported as significant remained significant, depending on what assumptions were made about how large the dependencies among observations really are. Of the 33 studies, 6-19 studies no longer had any significant results after correction. The authors end by providing recommendations for researchers planning group-administered treatment research.

This article builds on the previous article by Blitstein et al. (2005), which showed how external estimates of intraclass correlation can be used to improve the precision for the analysis of an existing group randomized trial. The authors extend that work to sample size estimation and power analysis for future group-randomized trials. Often this approach will allow a smaller study than would otherwise be possible without sacrificing statistical power. Such studies are needed, for example, as pilot studies to help plan for a full-scale efficacy trial, as replication studies, or in situations in which resource constraints prohibit a larger trial. The authors discuss the circumstances under which this strategy will be most helpful and the risks associated with conducting smaller studies.

This meta-analysis summarizes results from 30 randomized experiments that compare behavioral marital therapy with no-treatment control with distressed couples. Results showed that behavioral marital therapy is significantly more effective than no treatment (d=.585). Although behavioral marital therapy research studies tend to be conducted under conditions that are less clinically representative than other samples of studies, representativeness was not significantly related to outcome. However, evidence also suggested that publication bias may exist in this literature whereby small sample studies with small effects are systematically missing compared with other studies. This bias may inflate the effects of behavioral marital therapies reported in previous meta-analyses, though we also explore a number of alternative explanations for this small sample bias.

Propensity score analysis is a relatively recent statistical innovation that is useful in the analysis of data from quasi-experiments. The goal of propensity score analysis is to balance two non-equivalent groups on observed covariates to get more accurate estimates of the effects of a treatment on which the two groups differ. This article presents a general introduction to propensity score analysis, provides an example using data from a quasi-experiment compared to a benchmark randomized experiment, offers practical advice about how to do such analyses, and discusses some limitations of the approach. It also presents the first detailed instructions to appear in the literature on how to use classification tree analysis and bagging for classification trees in the construction of propensity scores. The latter two examples serve as an introduction for researchers interested in computing propensity scores using more complex classification algorithms known as ensemble methods.