Search

The Tennessee Voluntary Pre-K Program (TN-VPK) is statewide full-day program that gives priority to children from low-income families. A regression-discontinuity design with a statewide probability sample of 155 TN-VPK classrooms and 5,189 children participating across two pre-K cohorts found positive effects at kindergarten entry with the largest effects for literacy skills and the smallest for language skills. The results contribute to the growing body of regression-discontinuity studies of state and local pre-K programs and affirm the statewide generalizability of analogous prior findings from a more specialized subsample in the parent Tennessee Pre-K Study. Furthermore, the respective effect sizes compared favorably with those found in other regression-discontinuity studies of public pre-K on the same outcome measures, providing one index of the quality of the TN-VPK program.

As state-funded pre-kindergarten (pre-K) programs expand, it is critical to investigate their short- and long-term effects. This article presents the results through sixth grade of a longitudinal randomized control study of the effects of a scaled-up, state-supported pre-K program. The analytic sample includes 2,990 children from low-income families who applied to oversubscribed pre-K program sites across the state and were randomly assigned to offers of admission or a wait list control. Data through sixth grade from state education records showed that the children randomly assigned to attend pre-K had lower state achievement test scores in third through sixth grades than control children, with the strongest negative effects in sixth grade. A negative effect was also found for disciplinary infractions, attendance, and receipt of special education services, with null effects on retention. The implications of these findings for pre-K policies and practices are discussed.

Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a "convergent" approach) or an interest in exploring heterogeneity (which we term a "divergent" approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a "reductionist" meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an "integrative" approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.

Dependent effect sizes are ubiquitous in meta-analysis. Using Monte Carlo simulation, we compared the performance of 2 methods for meta-regression with dependent effect sizes--robust variance estimation (RVE) and 3-level modeling--with the standard meta-analytic method for independent effect sizes. We further compared bias-reduced linearization and jackknife estimators as small-sample adjustments for RVE and Wald-type and likelihood ratio tests for 3-level models. The bias in the slope estimates, width of the confidence intervals around those estimates, and empirical type I error and statistical power rates of the hypothesis tests from these different methods were compared for mixed-effects meta-regression analysis with one moderator either at the study or at the effect size level. All methods yielded nearly unbiased slope estimates under most scenarios, but as expected, the standard method ignoring dependency provided inflated type I error rates when testing the significance of the moderators. Robust variance estimation methods yielded not only the best results in terms of type I error rate but also the widest confidence intervals and the lowest power rates, especially when using the jackknife adjustments. Three-level models showed a promising performance with a moderate to large number of studies, especially with the likelihood ratio test, and yielded narrower confidence intervals around the slope and higher power rates than those obtained with the RVE approach. All methods performed better when the moderator was at the effect size level, the number of studies was moderate to large, and the between-studies variance was small. Our results can help meta-analysts deal with dependency in their data.

Many cognitive self-regulation (CSR) measures are related to the academic achievement of prekindergarten children and are thus of potential interest for school readiness screening and as outcome variables in intervention research aimed at improving those skills in order to facilitate learning. The objective of this study was to identify learning-related CSR measures especially suitable for such purposes by comparing the performance of promising candidates on criteria designed to assess their educational relevance for pre-K settings. A diverse set of 12 easily administered measures was selected from among those represented in research on attention, effortful control, and executive function, and applied to a large sample of pre-K children. Those measures were then compared on their ability to predict achievement and achievement gain, responsiveness to developmental change, and concurrence with teacher ratings of CSR-related classroom behavior. Four measures performed well on all those criteria: Peg Tapping, Head-Toes-Knees-Shoulders, the Kansas Reflection-Impulsivity Scale for Preschoolers, and Copy Design. Two others, Dimensional Change Card Sort and Backwards Digit Span, performed well on most of the criteria. Cross-validation with a new sample of children confirmed the initial evaluation of these measures and provided estimates of test-retest reliability.

A modification of the first stage of the standard procedure for two-stage meta-analytic structural equation modeling for use with large complex datasets is presented. This modification addresses two common problems that arise in such meta-analyses: (a) primary studies that provide multiple measures of the same construct and (b) the correlation coefficients that exhibit substantial heterogeneity, some of which obscures the relationships between the constructs of interest or undermines the comparability of the correlations across the cells. One component of this approach is a three-level random effects model capable of synthesizing a pooled correlation matrix with dependent correlation coefficients. Another component is a meta-regression that can be used to generate covariate-adjusted correlation coefficients that reduce the influence of selected unevenly distributed moderator variables. A non-technical presentation of these techniques is given, along with an illustration of the procedures with a meta-analytic dataset.

Much of the currently available evidence on the causal effects of public prekindergarten programs on school readiness outcomes comes from studies that use a regression-discontinuity design (RDD) with the age cutoff to enter a program in a given year as the basis for assignment to treatment and control conditions. Because the RDD has high internal validity when its key assumptions are met, these studies appear to provide strong evidence for the effectiveness of these programs. However, there are overlooked methodological problems in the way this design has typically been applied that have the potential to produce biased effect estimates. We describe these problems, argue that they deserve more attention from researchers using this design than they have received, and offer suggestions for improving future studies.

Prosocial behaviors or voluntary acts intended to help or promote the wellbeing of others are learned skills that accumulating research suggests have implications not only for constructively navigating interactions, relationships, and school and other group contexts, but also for life in a civil society. Yet, strikingly, there is limited knowledge of the factors that affect the degree to which prosocial behavior is learned and enhanced. The focus of this quantitative review will be to clarify the types of instructional practices and programs that show evidence of effectively prompting children and youth to act in ways that help others. The benefits of prosocial behavior to lives of children and youth have been highlighted in the last few decades, and this quantitative synthesis of the research is designed to provide evidence-based directions for supporting these constructive, caring, and helpful behaviors in interactions, relationships, schools, homes, and communities.

There are many situations where random assignment of participants to treatment and comparison conditions may be unethical or impractical. This article provides an overview of propensity score techniques that can be used for estimating treatment effects in nonrandomized quasi-experimental studies. After reviewing the logic of propensity score methods, we call attention to the importance of the strong ignorability assumption and its implications. We then discuss the importance of identifying and measuring a sufficient set of baseline covariates upon which to base the propensity scores and illustrate approaches to that task in the design of a study of recovery high schools for adolescents treated for substance abuse. One novel approach for identifying important covariates that we suggest and demonstrate is to draw on the predictor-outcome correlations compiled in meta-analyses of prospective longitudinal correlations.

Purpose: To identify, integrate, and summarize evidence from empirical studies of the language abilities of children who stutter (CWS) and children who do not stutter (CWNS). Method: Candidate studies were identified through electronic databases, the tables of contents of speech-language journals, and reference lists of relevant articles and literature reviews. The 22 included studies met the following criteria: studied both children who did and did not stutter between ages 2;0 (years;months) and 8;0, and reported norm-referenced language measures and/or measures from spontaneous language samples amenable to effect size calculation. Data were extracted using a coding manual and were assessed by application of general and specialized analytical software. Mean difference effect size was estimated using Hedges's g (Hedges, 1982). Results: Findings indicated that CWS scored significantly lower than CWNS on norm-referenced measures of overall language (Hedges's g = -0.48), receptive (Hedges's g = -0.52) and expressive vocabulary (Hedges's g = -0.41), and mean length of utterance (Hedges's g = -0.23). Conclusions: Present findings were taken to suggest that children's language abilities are potentially influential variables associated with childhood stuttering.

The objective of the proposed systematic review is to summarize the available evidence on the effects of prevention and intervention programs aimed at primary and secondary students for increasing school completion or reducing school dropout. Program effects on the closely related outcomes of school attendance (absences, truancy) will also be examined. Moreover, when accompanying dropout or attendance outcomes, effects on student engagement, academic performance, and school conduct will be considered. The primary focus of the analysis will be the comparative effectiveness of different programs and program approaches in an effort to identify those that have the largest and most reliable effects on the respective school participation outcomes, especially with regard to differences associated with treatment modality, implementation quality, and program location or setting. In addition, evidence of differential effects for students with different characteristics will be explored, e.g., in relation to age or grade, gender, race/ethnicity, and risk factors. Because of large ethnic and socioeconomic differences in graduation rates, it will be particularly important to identify programs that may be more or less effective for disadvantaged students. The ultimate objective of this systematic review is to provide school administrators and policymakers with an integrative summary of research evidence that is useful for guiding programmatic efforts to reduce school dropout and increase school completion.

"Theory as Method: Small Theories of Treatments," by Mark W. Lipsey, is one of the most influential and highly cited articles to appear in "New Directions for Evaluation." It articulated an approach in which methods for studying causation depend, in large part, on what is known about the theory underlying the program. Lipsey discussed the benefits of this approach in relation to four major issues in evaluating program effectiveness: how to base the research design on "relevant" constructs and variables; how to best "detect" outcomes subsequent to treatment; how to assure that outcomes can be "attributed" to treatment; and how to "interpret" the overall pattern and offer practical implications. This article situates Lipsey's article on small theories in the now much-broader literature on program theory, and highlights key contributions like the focus on relevant constructs and causal attributions.

Productive dialogue is informed best by multiple and diverse voices. Five seasoned evaluators, representing a range of evaluation perspectives, offer their views in two- to three-page discussant contributions. These individuals were asked to reflect and comment on the previous chapters in the spirit of critical review as a key source of evidence about methods choice. (Contains 1 table.)

This article discusses the significant contributions of Peter H. Rossi to the field of program evaluation. It was the publication of "Evaluation: A Systematic Approach" by Peter Rossi, Howard Freeman, and Sonia Rosenbaum in 1979 that most emphatically marked the point at which program evaluation had clearly consolidated as a distinct field of study with its own identity and repertoire of concepts, methods, and practices. This book was widely adopted as the primary text for courses in program evaluation taught across a spectrum of academic disciplines and quickly became the standard for the field. The evaluation textbook was not Peter's only formative influence on the development of program evaluation. Another very significant contribution was the series of seminal articles he and Huey Chen produced on the role of theory in evaluation. The main premise of these articles was that each social program embodies a theory of sorts--an action theory that reflects the assumptions inherent in the program about the nature of the social problem it addresses and the way it expects to bring about change in that problem. Chen and Rossi argued that evaluators should bring that theory to the surface and, if necessary, draw on other sources to further differentiate it. Aside from these, Peter was also a practicing evaluator who was involved in several landmark evaluation projects. Beginning with his earliest incarnation as an applied sociologist, the main theme of his scholarly career was the evaluation of social programs and related research. Most of this work focused on programs for the disadvantaged and disenfranchised. He conducted one of the most famous needs assessments of all times--a systematic survey to determine the number of homeless persons in Chicago and the circumstances that led to their condition. In his teaching, professional commentary, and evaluation research, Peter was an unstinting champion of using the strongest available methods to tackle questions about social programs. Peter was truly one of the founding fathers of program evaluation.

Outlines a few items of "unfinished business" in the profession of evaluation, focusing on outcome evaluation, also called impact evaluation. No issue related to the future of evaluation seems more important than that of finding routine methods to assess the impact of everyday practical programs on the social conditions they address. Also discusses the accumulation of knowledge in the area of program evaluation and the issue of defining programs. (SLD)

Performed a meta-analysis of 22 wilderness challenge programs to assess impact on delinquent behavior. The overall effect size for delinquency outcomes (0.18) was equivalent to a recidivism rate of 29% for program participants versus 37% for nonparticipants. Greatest reductions in delinquent behavior came with intense physical activities or therapeutic enhancements as program components. (SLD)

Summarizes findings of longitudinal study on the relationship between marijuana use and later delinquency and problem behaviors. These studies describe the magnitude of the longitudinal and cross-sectional relationships. Cross-sectional findings measured early in adolescence are strongest. Findings suggest that the nature of the relationship is likely to be one of comorbidity and not cause and effect. (Author/JDM)

Analyzes correlates of marijuana use based on 3,690 effect sizes coded from 86 prospective longitudinal studies. Summarizes findings on strength of relationships for categorizing predictor variables, and implications of these relationships. Findings are relevant for intervention programmers and policymakers since they identify characteristics of high-risk individuals. The information can help target intervention on potential mediating variables. (Author/JDM)

It is argued that, although thousands of evaluations have been conducted of social interventions, little has been done to cumulate those results to guide intervention architects. Building social intervention theory and meta analyses are suggested as ways to unify this knowledge and make it useful. (SLD)

A representative sample of studies drawn from the published program evaluation literature is examined. It is concluded that weak designs, low statistical power, ad hoc measurement, and neglect of treatment implementation and program theory characterize the state of the art in program evaluation. (Author/BS)

Three meta-analyses by C. J. Garrett (1984, 1985), P. Kaufman (1985), and W. S. Davidson and others (1984) of juvenile delinquency interventions are summarized. This systematic literature review indicates that interventions to reduce juvenile delinquency may have small, but meaningful, impacts. Promising avenues for future research are suggested. (TJH)

Evaluation research is discussed as an applied social science that uses quantitative-comparative techniques to assess the effects of planned intervention. Focus is on the generally insufficient validity of treatment effectiveness research, and the relationship between research constructs and theory and their operationalization. (TJH)

Two program goals were: (1) providing a community-based alternative for arrested juveniles who otherwise would have been referred to the juvenile justice system; and (2) reducing juvenile delinquency. Results from various measures indicated the program had little success in decreasing referrals but produced positive delinquency reduction effects. (Author/RL)