Search

This article estimates earnings impacts for those who completed long-term college credentials (degrees or college certificates requiring a year or more of study) and those who did not in an experimental evaluation of a federally-funded sectoral job training program. The experimental evaluation found no overall impact of the program on earnings, but we explore whether impacts vary by the long-term credential receipt. In theory, we expect that program impacts should be larger for those who earn long-term credentials. We test this theory using Analysis of Symmetrically-Predicted Endogenous Subgroups (ASPES)--an approach that leverages the experimental design to create experimentally valid treatment and control subgroups associated with some endogenous activity and estimates impacts for these subgroups (subject to assumptions required for identification). We find weak evidence that those who earned long-term credentials experienced meaningfully larger program impacts than those who did not. We posit that these differences are largely due to engagement with support services.

In experimental evaluations of policy interventions, the so-called Bloom adjustment is commonly used to estimate the impact of the treatment on the treated. It does so by rescaling the estimated impact of the intention to treat--that is, the overall treatment-control group difference in outcomes for the entire experimental sample--by the percentage of cases that took up the treatment offer. The practice of also rescaling the variance, as is common in the literature, imposes simplifying assumptions that may lead to biased variance estimates. We compare variances using the Bloom adjustment to variances that capture all the estimation error. While the difference between these variances is negligible in three experimental evaluations, we highlight three conditions that could result in larger relative bias: large impacts, large variability in compliance, and/or substantial endogeneity bias. The presence of these conditions could potentially result in a different conclusion for policy or practice. [An earlier version of this article was prepared, in collaboration with Luke Keele, with federal funding from the U.S. Department of Labor, Employment and Training Administration, under Contract Number GS10F0086J/DOLF091A21626.]

Under the banner of evidence‐based policy, evidence reviews—summaries of existing studies on an intervention—have become an established form of policy analysis. Whereas evidence reviews have typically focused on studies of an intervention as delivered, interest is growing in examining the specific elements that comprise the intervention and that contribute to its success—broadly referred to as its core components. Understanding the core components of interventions allows policymakers to identify with greater precision what works, in which contexts, and for which populations. With the aim toward advancing core components analysis for policy design, this Research Note describes and compares four synthesis approaches for identifying core components in the context of evidence reviews: the distillation and matching model, meta‐regression, framework synthesis, and qualitative comparative analysis. The article then reflects on how to advance core components analysis in evidence reviews to better inform policy design and the policymaking process. [ABSTRACT FROM AUTHOR]

In experimental evaluations of health and social programs, the role of dosage is rarely explored because researchers cannot usually randomize individuals to experience varying dosage levels. Instead, such evaluations reveal the average effects of exposure to an intervention, although program exposure may vary widely. This article compares three analytic methods that are used to examine dosage effects: the Analysis of Symmetrically Predicted Endogenous Subgroups (ASPES) method developed by Peck, instrumental variables, and propensity score matching methods. The article provides an illustration using data from the Supporting Healthy Marriage (SHM) demonstration, which rigorously tested the effectiveness of a skills-based relationship education program that had high take-up (91%) but substantial variation in exposure thereafter. To understand the relationship between program dosage level and impact magnitude, we use the ASPES method to create subgroups defined by the number of workshop hours couples attended and estimate impacts for those groups. We find that the SHM program impact on study members who received any program services is similar in magnitude to the impact on those study members who received a high or low-to-moderate dose of program services. For experimental evaluations of federal programs, the ASPES method shows promise as a way to investigate the relationship between treatment dosage and impact magnitude, a relationship crucial to designers and funders of public programs seeking to maximize program effectiveness.

Why SNAP Works: A Political History¿and Defense¿of the Food Stamp Program by Christopher Busso explores the history, evolution, politics, and effectiveness of the Supplemental Nutrition Assistance Program (SNAP) and its predecessor, the Food Stamp Program. The book discusses the coexistence of food surplus and food insecurity, the interplay between agricultural policy and food assistance policy, and the debates surrounding SNAP as a welfare program. It also examines the program's longevity and effectiveness. The book is recommended for college students, teachers, policy researchers, and the general public interested in political history and social policy. [Extracted from the article]

Several analytic strategies exist for opening up the "black box" to reveal more about what drives policy and program impacts. This article focuses on one of these strategies: the Analysis of Symmetrically-Predicted Endogenous Subgroups (ASPES). ASPES uses exogenous baseline data to identify endogenously-defined subgroups, keeping the analysis of some postrandomization choice, event, or milestone grounded in the strength of experimental design. Building on lessons from prior applications of ASPES and also adding some new analyses, this article focuses on four specific practical considerations: first-stage prediction success, assumption credibility, data availability, and sample size. Discussion implies the optimal conditions for effective application of ASPES and points to future research that can enhance the overall tool kit of "what works" analyses.

The American Journal of Evaluation's editorial, "From the Co-Editors: That's a Wrap! Closing Out Our First Volume," celebrates the successful conclusion of their first year. The issue features articles on narrowing the research-practice gap, understanding stability and change in social investment evaluation, and evaluating collective impact initiatives. Additionally, the journal includes a Method Note Section and a review of Policy Evaluation in the Era of COVID-19. The editors express gratitude to their team, reviewers, and authors for their contributions to the journal's success. [Extracted from the article]

Researchers and policy makers are increasingly dissatisfied with the "average treatment effect." Not only are they interested in learning about the overall causal effects of policy interventions, but they want to know what specifically it is about the intervention that is responsible for any observed effects. In the U.S., using experimentally-designed evaluation to capture the average treatment effect is both commonplace and preferred practice; but, as this paper argues, there are many important questions yet to be asked and answered via our body of experimental research. As a reconsideration of Peck (2003), on the tenth anniversary of its publication, this article recasts earlier work on analyzing "what works" as a call to action for evaluators and policy analysts: we can and should do better. (Contains 1 figure and 5 notes.)

AEA's Efforts on Competencies and Professionalization In 2015, AEA established a Competency Task Force (CTF) to develop evaluator competencies. From Creation to Use of Competencies The work concerning the creation, validation, and use of professional competencies is currently entering the third process step: defining how to recognize and use evaluator competencies. For example, the article by Minji Cho, Ann Marie Castleman, Haley Umans, and Mike Osiemo Mwirigi suggests how to measure evaluator competencies, noting that strengthening the rigor of validations of the 2018 competencies is an important step toward professionalization. Evaluator competencies have been a topic of keen interest in international evaluation associations and within various arms of the American Evaluation Association (AEA), including but not limited to the AEA Board, the Competencies Task Force, and Professionalization and Competencies Working Group. [Extracted from the article]

This study addresses validity issues in evaluation that stem from Ernest R. House's book, "Evaluating With Validity". The authors examine "American Journal of Evaluation" articles from 1980 to 2010 that report the results of policy and program evaluations. The authors classify these evaluations according to House's "major approaches" typology (Systems Analysis, Behavioral Objectives, Decision making, Goal-free, Professional Review, Art Criticism, Quasi-legal, and Case Study) and the types of validity (measurement, design, interpretation, use) the evaluations consider. Analyzing the intersection of evaluation type and validity type, the authors explore the status of House's standards of Truth, Beauty, and Justice in evaluation practice. (Contains 4 tables, 5 figures and 1 note.)

Background: Research on evaluation use focuses on putting evaluation recommendations into practice. Prior theoretical research proposes varied frameworks for understanding the use (or lack) of program evaluation results. Purpose: Our purpose is to create and test a single, integrated framework for understanding evaluation use. This article relies on prior theoretical research regarding categories of utilization, typologies of recommendations, and factors affecting utilization to frame an empirical study of evaluation use that then tests the integrated theory. Setting: The empirical part of the article draws on post-evaluation interviews with sixteen agencies that have engaged in evaluation research. Subjects: The agencies are mostly local non-profits, but the sample also includes a state agency, a city agency, and two university-community partnerships. All agencies had undergone a program evaluation between 2003 and 2006. Intervention: Having participated in an evaluation is the main "intervention" of interest in this article, in which we consider the relationship between evaluation use theory and empirical evidence on the topic. Research Design: A qualitative approach, our research design involved examining each of the sixteen agencies within two years of their having been evaluated. Data Collection and Analysis: Data collection included structured in-person interviews with at least one key informant in each agency. In addition, a short, closed-ended survey was administered to research participants. Interview data were analyzed using content analysis of themes and grouping agencies according to their evaluation's outcomes (favorable or not) and experiences. Survey data were analyzed with simple descriptive statistics and similarly involved a subgroup analysis, according to agencies' reported use of evaluation. Findings: Most evaluation use is conceptual, and few agencies studied actually implemented specific evaluation recommendations. Agencies perceived recommendations as changes to rules and structure, which theory and prior research suggest suppresses use. An important human factor that influenced evaluation use was minimal post-evaluation interaction with evaluators. Some long-term influence of the evaluation is evident, but only as conceptual and not as instrumental. In fact, very little instrumental use existed in this empirical assessment. Conclusions: Evidence suggests that evaluation use in practice aligns with theory, specifically emphasizing the conceptual use dimension. The proposed integrated model of prior evaluation use theories may hold value for future theoretical and empirical work. (Contains 1 figure, 3 tables and 2 footnotes.)

This article presents a creative and practical process for dealing with the problem of selection bias. Taking an algorithmic approach and capitalizing on the known treatment-associated variance in the X matrix, we propose a data transformation that allows estimating unbiased treatment effects. The approach does not call for modelling the data, based on underlying theories or assumptions about the selection process, but instead calls for using the existing variability within the data and letting the data speak. We illustrate with an application of the method to Italian Job Centres. (Contains 4 figures, 3 tables and 2 notes.)

The evaluation literature on Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) youth programs is largely absent of quantitative studies, likely in part due to the challenge of using experimental or quasi-experimental evaluation designs. This paper proposes the creative use of a national data set to overcome the problem of estimating a counterfactual for this population. In addition to discussing lessons from this approach, we describe the program and its impacts. Evidence suggests that the program under study--1n10, a local support group for LGBTQ youth in the Phoenix, AZ, metropolitan area--draws a relatively more disadvantaged group than the national average but that their levels of suicide ideation are lower. By addressing key threats to the internal validity of evaluation designs, we rule out plausible rival explanations for program impacts. (Contains 3 footnotes and 4 tables. 1n10 Survey Instrument and Index Construction Details are appended.)

This article uses propensity scores to identify subgroups of individuals most likely to experience a reduction in cash benefits because of sanctions in some of the programs that make up the National Evaluation of Welfare-to-Work Strategies. It extends program evaluation methodology by using propensity scoring to identify the subgroups of sanctioned and nonsanctioned welfare recipients. Specifically, the propensity score is used to identify the sample subset most likely to experience program sanction. In this application, the propensity score helps deal with an omitted variable problem, that of not knowing what the sanction status is in the control group (because they were not subject to the policies being tested). Findings reveal that being high sanction risk induces greater work levels and therefore higher earnings, but it also results in receiving less cash assistance so that sanctioned recipients have roughly the same net incomes as nonsanctioned ones. (Contains 3 tables and 5 notes.)

The conventional way to measure program impacts is to compute the average treatment effect; that is, the difference between a treatment group that received some intervention and a control group that did not. Recently, scholars have recognized that looking only at the average treatment effect may obscure impacts that accrue to subgroups. In an effort to inform subgroup analysis research, this article explains the challenge of treatment group heterogeneity. It then proposes using cluster analysis to identify otherwise difficult-to-identify subgroups within evaluation data. The approach maintains the integrity of the experimental evaluation design, thereby producing unbiased estimates of program impacts by subgroup. This method is applied to data from the evaluation of New York State's Child Assistance Program, a reform that intended to increase work and earnings among welfare recipients. The article interprets the substantive findings and then addresses the advantages and disadvantages of the proposed method.

Proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. The approach estimates the impact of programs on subgroups identified by a postreatment choice while maintaining the integrity of the experimental research design. (SLD)

The American Journal of Evaluation's latest issue explores various topics related to evaluation, including framing, scoping, and impact variation. The lead article discusses the theory of action framework for logic modeling, offering new perspectives on its use in evaluation. Another article conducts a comprehensive scoping review of evaluation use, identifying factors and findings that influence its implementation. The issue also includes an article on the potential for low-bias estimates of differential impact in multi-site evaluations, as well as a section on international advances in evaluation. Additionally, a book review examines evaluation in rural communities. Overall, the diverse content of this issue provides valuable insights for evaluation scholars and practitioners. [Extracted from the article]

Administered by the Office of Family Assistance in the U.S. Department of Health and Human Services' Administration for Children and Families (ACF), the Health Profession Opportunity Grants (HPOG) Program provided education and training to Temporary Assistance for Needy Families (TANF) recipients and other adults with low incomes for occupations in the healthcare field. The impact evaluation for the first cohort of grantees (HPOG 1.0) leveraged the program's implementation across many locations, using a three-armed evaluation design (including a second treatment arm) in some places, as a way to examine whether any of three selected program components, or enhancements, contributed to the program's overall impact. This article tells the story of the evaluation and draws lessons from that experience, discussing implications for future implementation of multi-armed experiments in a multi-site evaluation. [ABSTRACT FROM AUTHOR]

Prior research considers the extent to which public assistance recipients' charitable activity differs from the habits of the general population. Although receiving public assistance is negatively associated with donating money, the relationship to volunteering is unclear. In response to challenges overcoming selection bias, we conducted a multivariate cluster-based subgroup analysis to reduce bias in our claims about the ways in which public assistance receipt affects charitable activity. This innovative approach to dealing with the problem of selection bias has implications and applications across the social sciences. KEY WORDS: public assistance, welfare, charity, selection bias, cluster analysis, Much scholarship is dedicated to understanding the factors that influence one to donate time or money to charitable causes; but only a small portion of that scholarship considers those within [...]

This paper contributes to the environmental justice literature by addressing several outstanding issues in a single study. Using a cross-time data set that allows us to control for the prevalent 'chicken-and-egg' or 'which-came-first' problem, we analyze the relative importance of poverty and race/ethnicity in an analysis that includes economic costs, potential legal costs, and potential collective action. Because the most appropriate functional form is not obvious, we use several methods, including Tobit, Poisson, and ordinary least squares, on different forms of the dependent variable. In every case, controlling for the population present at the time of disamenity location and controlling the other factors mentioned, we find evidence of disproportionate collocation based on race/ethnicity, but not on poverty alone. We also find that the potential for collective action decreases the likelihood of receipt of the studied disamenities. KEY WORDS: environmental justice, discrimination, toxic release, collective action, Introduction Despite several decades of research, the issue of whether environmental discrimination exists is still unresolved. One ongoing issue in environmental justice (EJ) research is the extent to which the [...]

This research note proposes an addition to the poverty measurement debate. Motivated by dissatisfaction with the official poverty measure, which many scholars and practitioners share, we propose the use of [...]

Welfare reform has increased the importance of employment over simple eligibility determination. Research has focused on strategies for enhancing employment among welfare recipients, and this article adds to the literature by examining specifically the effects of using an assessment tool to assist case managers in targeting services to a heterogeneous client population. Analysis indicates that clients of case managers who used the screener, compared with a set of clients of case managers who did not use it, received more services and had higher program participation rates but experienced no improvement in short-term employment outcomes. This suggests that the instrument as a tool in case management is effective at identifying client needs, which is the first step to achieving more favorable employment-related outcomes in the long run., Welfare reform has changed the paradigm of the welfare system from one of determining eligibility and disbursing checks to one where employment is central. In the work-focused system, services are [...]