Search

This report provides a detailed account of the Regional Educational Laboratory (REL) Program's experience establishing and supporting research-practice partnerships (called "research alliances") during its 2012-17 contract cycle. The report adds to the growing literature base on researcher-practitioner partnerships by sharing how the RELs reported creating, engaging, and maintaining multiple partnerships, with the purpose of informing future collaborative efforts for researchers and practitioners and for those who wish to support research-practice partnerships. It addresses questions about: how REL research alliances fit within the broader context of research-practice partnerships; what characteristics existed among REL research alliances and how they evolved over time; and what challenges RELs reported experiencing while establishing and supporting research alliances and the strategies RELs employed to address those challenges. Finally, the paper discusses the implications of the REL research alliance experience for other networks of research-practice partnerships.

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies (Imbens and Rubin, 2015; Schochet, 2015, 2016). The estimators are derived using the building blocks of experimental designs with minimal assumptions, and are unbiased and normally distributed in large samples with simple variance estimators. The methods apply to randomized controlled trials (RCTs) and quasi-experimental designs (QEDs) with comparison groups for a wide range of designs used in social policy research. The methods have important advantages over traditional model-based impact estimation methods, such as hierarchical linear model (HLM) and robust cluster standard error (RCSE) methods, and perform well in simulations (Schochet, 2016). The free "RCT-YES" software (www.rct-yes.com) estimates and reports impacts using these design-based methods. This report discusses several key topics for estimating average treatment effects (ATEs) for multi-armed designs. The report is geared toward methodologists with a strong background in statistical theory and a good knowledge of design-based concepts for the single treatment-control group (two-group) design. The report builds on Schochet (2016), referencing key results and formulas to avoid repetition, and serves as a supplement to that report. The focus is on RCTs, although key concepts apply also to QEDs with comparison groups. The report is in three sections. Section 1 discusses how design-based ATE estimators for the two-group design need to be modified for the multi-armed design when comparing pairs of research groups to each other. Section 2 discusses multiple comparison adjustments when conducting hypothesis tests across pairwise contrasts to identify the most effective interventions. Finally, Section 3 shows that the assumptions required to identify and estimate the complier average causal effect (CACE) parameter using an instrumental variable (IV) framework become much more complex in the multi-armed context, and may not be possible in some cases. While Sections 2 and 3 are germane to multi-armed designs regardless of the impact estimation methods used for the analysis, these sections emphasize approaches that align with the non-parametric underpinnings of the design-based framework. [For related reports see: "Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027" (ED575014) and "Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation. NCEE 2017-4026" (ED575021).]

A new design-based theory has recently been developed to estimate impacts for randomized controlled trials (RCTs) and basic quasi-experimental designs (QEDs) for a wide range of designs used in social policy research (Imbens & Rubin, 2015; Schochet, 2016). These methods use the potential outcomes framework and known features of study designs to connect statistical methods to the building blocks of causal inference. They differ from model-based methods that have commonly been used in education research, including hierarchical linear model (HLM) methods and robust cluster standard error (RCSE) methods for clustered designs. In comparison to model-based methods, the design-based methods tend to make fewer assumptions about the nature of the data and also more explicitly account for known information about the experimental and sampling designs. While these theoretical differences suggest the corresponding estimates might differ, it is unclear how much of a practical difference it makes to use design-based methods versus more conventional model-based methods. This study addresses this question by re-analyzing nine past RCTs in the education area using both design- and model-based methods. The study uses real data, rather than simulated data, to better explore the differences that would arise in practice. In order to investigate the full scope of differences between the methods, the study uses data generated from different types of randomization designs commonly used in social policy research: (1) non-clustered designs in which individuals are randomized; (2) clustered designs in which groups are randomized; (3) non-blocked designs in which randomization is conducted for a single population; and (4) blocked (stratified) designs in which randomization is conducted separately within partitions of the sample. The study conducts the design-based analyses using "RCT-YES," a free software package funded by the Institute of Education Sciences (IES) that applies design-based methods to a wide range of RCT designs (www.rct-yes.com). This report focuses on two analyses that compare model- and design-based methods, both of which suggest there is little substantive difference in the results between the two methods. For both analyses, the study uses a reference model-based method that is similar to the one used in the original evaluation. In the first analysis, the study compares the reference model-based method to a design-based method with underlying assumptions that most closely align with those of the reference model-based method. In the second analysis, the report presents a sensitivity check that compares the reference model-based method to an alternative design-based method. In particular, the alternative method is based on the default settings in the" RCT-YES" software, which correspond to an alternative set of plausible assumptions. The findings from both analyses suggest that model- and design-based methods yield very similar results in terms of the magnitude of impact estimates, statistical significance of the impact estimates, and implications for policy. To contextualize the differences in impact estimates between design- and model-based methods, the report also presents a third analysis, which compares estimates from two commonly used model-based methods: (1) HLM methods; and (2) linear models with ordinary least squares (OLS) assumptions and RCSE to account for clustering. Importantly, this analysis suggests that the differences between the design- and model-based methods (with similar assumptions) are no greater than the differences that would arise between commonly used, model-based methods. The study suggests that researchers should select estimators with assumptions that best suit the goals of their study regardless of whether they use a design- or model-based approach. Moreover, researchers should consider the trade offs between different assumptions, and how these assumptions affect the interpretation of findings. Appended are: (1) Hierarchical linear model methods; and (2) Detailed description of studies and results. [For related reports see: "What Is Design-Based Causal Inference for RCTs and Why Should I Use It? NCEE 2017-4025" (ED575014)and "Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027 (ED575022).]

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies. The impact estimators are derived using the building blocks of experimental designs with minimal assumptions, and have good statistical properties. The methods apply to randomized controlled trials (RCTs) and quasi-experimental designs (QEDs) with treatment and control (comparison) groups. Importantly, design-based estimators are acceptable for What Works Clearinghouse (WWC) evidence reviews (Scher and Cole, 2017). Although the fundamental concepts that underlie design-based methods are straightforward, the literature on these methods is technical, with detailed mathematical proofs required to formalize the theory. Thus, the daunting task of wading through this literature might discourage some education researchers from using design-based methods in favor of more traditional "model-based" methods, such as hierarchical linear modeling (HLM) (Raudenbush and Bryk, 2002). This brief aims to broaden knowledge of design-based methods by providing intuition on their key concepts and how they compare to model-based methods as typically implemented. Using simple mathematical notation, the brief is geared toward education researchers with a good knowledge of evaluation designs and HLM. The discussion synthesizes Schochet (2016), omitting details for brevity and accessibility. The focus is on RCTs, although key concepts apply also to QEDs. [For related reports see: "Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation. NCEE 2017-4026" (ED575021) and "Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027" (ED575022).]

This short brief for education decision makers discusses three main factors that may contribute to a finding of no effects: failure of theory, failure of implementation, and failure of research design. It provides readers with questions to ask themselves to better understand "no effects" findings, and describes other contextual factors to consider when deciding what to do next.

The purpose of this paper is to describe best practices in designing and implementing strong quasi-experimental designs (QED) when assessing the effectiveness of policies, programs or practices. The paper first discusses the issues researchers face when choosing to conduct a QED, as opposed to a more rigorous randomized controlled trial design. Next, the paper documents four sets of best practices in designing and implementing QEDs including: (1) considering unobserved variables, (2) selecting appropriate matching strategies for creating comparison groups, (3) following general guidelines for sound research, and (4) addressing the What Works Clearinghouse (WWC) standards. The paper then presents 31 frequently-asked questions related to QEDs meeting WWC standards with reservations and discusses common pitfalls that cause studies not to meet WWC standards. Topics covered in the frequently asked questions section include: study design/group formation, outcomes, confounding factors, baseline equivalence, sample loss and power, and analytic techniques. The paper provides a detailed checklist of issues researchers should consider when designing a strong QED study, and discusses why each issue is important and how it relates to WWC standards. One appendix is included: Checklist for Quasi-Experimental Designs; Study Design Characteristics to Consider.

This report presents the statistical theory underlying the "RCT-YES" software that estimates and reports impacts for RCTs for a wide range of designs used in social policy research. The report discusses a unified, non-parametric design-based approach for impact estimation using the building blocks of the Neyman-Rubin-Holland causal inference model that underlies experimental designs. This approach differs from the more model-based impact estimation methods that are typically used in education research. The report discusses impact and variance estimation, asymptotic distributions of the estimators, hypothesis testing, the inclusion of baseline covariates to improve precision, the use of weights, subgroup analyses, baseline equivalency analyses, and estimation of the complier average causal effect parameter. A section on mathematical proofs is appended.

A study involving random assignment of all youth eligible for Job Corps to either a Job Corps program or to a control group was conducted to assess the impact of Job Corps on key participant outcomes. Participants in the study were nationwide youth eligible for Job Corps who applied for enrollment for the first time between November 16, 1994, and December 17, 1995. The study sought to determine the following:(1) how effectively Job Corps improves the employability of disadvantaged participants, (2) whether Job Corps impacts differ for youths with different baseline characteristics, and (3) how effective the residential and nonresidential components of Job Corp are. Findings over the first 4 years after random assignment include the following: (1) Job Corps provided extensive education, training, and other services to the program group and improved their educational attainment; (2) Job Corps generated positive employment and earnings impacts by the beginning of the third year after random assignment and the impacts persisted through the fourth year; (3) employment and earnings gains were found broadly across most subgroups of students; (4) the resident and nonresidential programs were each effective for the youths they served; (5) Job Corps significantly reduced youths' involvement with the criminal justice system; (6) Job Corps had small beneficial impacts on the receipt of public assistance and self-assessed health status, but no impacts on illegal drug use; and (7) Job Corps had no impacts on fertility or custodial responsibility, but it slightly promoted independent living and mobility. (The report include numerous tables and charts, 31 references, and five appendixes concerning the study methodology.) (KC)

A benefit-cost analysis of the Job Corps program compared groups randomly assigned to either enroll in the program or to constitute a control group that did not enroll. Youth who participated in the study were those found eligible for Job Corps nationwide between November 1994 and February 1996. Interviews with participants and the assignment of dollar values to costs and benefits were among the research methods used. Benefits and costs measured included the following: (1) benefits of increased output resulting from the additional productivity of Job Corps participants; (2) benefits from reduced use of other programs and services; (3) benefits from reduced crime committed by or against participants; and (4) program costs and costs of resources used by Job Corps. Benefits and costs were measured from the perspectives of society as a whole, participants, and the rest of society (non-participants in Job Corps). The study's findings indicate that Job Corps is a good investment. The benefits to society exceed the costs of the program by nearly $17,000 per participant, assuming that the observed earnings impacts do not decline rapidly as participants get older. The researchers say evidence from other studies suggests the impacts will persist without rapid decay. The study concluded that Job Corps is a valuable program whose benefits exceed costs over a wide spectrum of student groups and for several areas of society. (The report includes 28 tables and 11 figures.) (KC)

The question of whether the Job Corps's impacts on students' employment and related outcomes differ according to the characteristics of the Job Corps center attended was examined. The study sample consisted of approximately 9,400 program group members and 6,000 control group members who were randomly selected from among the nearly 81,000 applicants nationwide who applied for Job Corps services for the first time between November 17, 1994, and December 16, 1995, and were found eligible for services by February 1996. Study participants were interviewed shortly after their random assignment and 12, 30, and 48 months thereafter. Job Corps impacts were similar for contract centers and Civilian Conservation Centers. Impacts were similar in large, medium, and small centers. The beneficial impacts of the Job Corps program overall were broadly distributed throughout the country and not confined to a few regions. Impacts were similar for centers rated as high-performing, medium-performing, and low-performing centers based on the Job Corps performance measurement system. As expected, outcomes of the program group were better among the high-performing centers. However, so too were the outcomes of the control group who would have attended the high-performing centers. (Twenty-eight tables/figures are included. Eight supplementary tables are appended.) (MN)

The National Job Corps Study is based on a national random sample of all eligible applicants to Job Corps in late 1994 and 1995. The sampled youth were assigned randomly to either a program group whose members could enroll in Job Corps or a control group whose members could enroll in all other programs available to them in their communities. Findings related to delivering services indicated Job Corps centers effectively deliver the planned services called for by the program model and Job Corps provides extensive education, training, and other services. Findings related to making a difference show Job Corps substantially increases the education and training services that youths receive and improves their skills and educational attainment; Job Corps generates employment and earnings gains; employment and earnings gains are found across most groups of students; the residential and nonresidential programs are each effective for the youths they serve; Job Corps significantly reduces involvement with crime; and Job Corps has modest or no impacts on a range of other outcomes. Outcomes regarding whether Job Corps is a good investment whose benefits exceed costs are Job Corps is cost effective despite its high costs; benefits during the study period are modest; benefits should continue; and Job Corps is a good investment. (YLB)

The National Job Corps Study was undertaken to obtain the information needed to assess the Job Corps' success in providing employment assistance to disadvantaged youths aged 16 to 24 years. The study's findings related to outreach and admissions (OA) and center characteristics and practices that appear to promote positive programmatic experiences for Job Corps applicants and participants were examined by using agency records for a randomly selected sample of youth who were eligible to enroll in Job Corps. The following items had significant impacts on applicants' programmatic experiences: (1) OA counselors' outreach and screening practices; (2) OA counselors' knowledge and experience level; (3) center operator type, size, and location; and (4) the strength of centers' vocational and academic programs. The particular vocational areas offered by centers and the range of vocations offered did not significantly affect students' programmatic experiences. Limited residential living facilities were associated with shorter lengths of stay, but exceptionally good facilities did not appear to promote longer lengths of stay. (Twenty-two tables/figures are included. The following items are appended: the study methodology; results for vocational choices of Job Corps students; results for applicant typologies; and a description of the methods used to estimate impacts for different Job Corps program experiences.) (MN)

This report examines the extent to which Job Corps improves literacy and numeracy skills. Chapter I provides an overview of Job Corps and discusses key policy issues related to basic skills and study objectives. Chapter II addresses study design issues, interview response rates, and analytic methods used to estimate program impacts on literacy scores. Chapter III presents Job Corps impacts on three domains of literacy skills (prose, document, and quantitative) and provides a descriptive analysis, impact findings, and estimates of literacy for key subgroups. It reports that the typical youth Job Corps serves has lower functional literacy scores than the typical young adult, especially in the quantitative literacy domain. Job Corps' impacts on participants' functional literacy skills were positive in all three domains. Chapter IV discusses the extent to which estimated impacts on literacy skills are consistent with impact findings on key outcomes associated with literacy skills. A proposed theoretical framework is used to examine the link among educational attainment, labor market outcomes, and literacy skill scores. Empirical results on these relationships are presented: literacy levels are influenced by schooling and employment measures; and some association exists between literacy assessment scores and labor market outcomes, but a large residual variation in earnings remains. Appendixes contain 13 references, analysis of nonresponse, administration and scoring of the assessment, and supplementary tables. (YLB)

The National Job Corps Study was conducted in 1994-1996 to provide a thorough and rigorous assessment of the impacts of the Job Corps on key participant outcomes. To ensure that the study was well implemented, a study team from Mathematica Policy Research, Inc., (MPR) investigated outcome and admission (OA) procedures in each Job Corps region and developed proposed procedures for conducting random assignment tailored to each region. A four-step core random assignment process was implemented. During the sample intake period, MPR staff monitored sample buildup to ensure that the research sample was near target levels and determine whether the initial sample design parameters required adjustment. Job Corps staff implemented the random assignment procedures successfully over the 16-month sample intake period. Overall, the study had noticeable effects on key aspects of program operations but modest effects on OA counselors' activities and the composition of students coming to the program. (Fifteen tables/figures are included. The following items are appended: lists of special programs excluded from the Job Corps evaluation and data items needed for random assignment processing and monitoring; Job Corps study materials and forms; a chronology of random assignment implementation; and a list of processing steps performed by MPR before random assignment.) (MN)

The value of the output and services produced by students while enrolled in the Job Corps was estimated by analyzing data from a sample of 2 projects from each of 23 Job Corps centers. The projects were subjected to in-depth analysis based on independent-estimate and relative-productivity approaches. The following were among the key findings: (1) in 1 year, more than 1 million student-days are spent on all work projects in the Job Corps, with nearly 80% of student-days spent on vocational skills training (VST) projects and 20% spent on work experience (WE) projects; (2) students produce output worth $5.48 per hour spent on VST projects and $7.01 per hour spent on WE projects; (3) over 1 year, Job Corps students produce output worth more than $27 million while conducting non-center-serving projects, which is equivalent to $789 per student year; and (4) students working on center-serving projects reduce centers' operating costs by an estimated $280-$360 per student year, which is small when compared with the program operating costs of approximately $26,000 per student-year. (Twelve tables/figures are included. The following items are appended: a discussion of the weights used in the study; summaries of the work project studies; and standard errors of the estimates.) (MN)

This report presents results of a process analysis that describes and documents Job Corps services and operations. Chapter one provides overviews of Job Corps, the national Job Corps study, and the process analysis. Chapter two describes the administrative structure of Job Corps and presents data on the geographic distribution and characteristics of key operating components. Chapters three to nine focus on these components of Job Corps: outreach and admissions, including counseling; vocational education, including occupational exploration program, vocational skills training projects, work experience program, and advanced training; academic education; residential living and health services, including student government and leadership and behavior management; center administration related to staffing and safety/security; organization and provision of Job Corps' placement services; and Job Corps performance management system and how it affects center operations. Chapter ten offers conclusions. Appendixes include data collection design and implementation and supplementary tables. (YLB)