The Unique Challenges of Afterschool Research: A Practical Guide for Evaluators and Practitioners

Funders and policymakers are increasingly recognizing the afterschool field for its vital role in supporting the social and emotional growth and academic achievement of school-age youth. Although this recognition is welcome, it often comes with increased expectations for high-quality research demonstrating the value of programming. To satisfy these demands and make the most of funding opportunities, practitioners must develop strong partnerships with external evaluators. However, developing afterschool evaluation partnerships that work well for all parties is often far more difficult than program directors or evaluators anticipate. When research is conducted in K-12 schools, educators often bring some experience in assessment methods, and researchers often have at least a basic knowledge of pedagogy. In contrast, in the out-of-school time (OST) field, program directors with little formal research experience are frequently paired with evaluators who lack experience in OST programs. This research-practice gap, if not addressed, can translate into frustrating evaluation experiences for practitioners and evaluators alike. Program directors may finish an evaluation feeling that they did not learn anything new or that the study was entirely for the benefit of the funder. Evaluators may find themselves stymied by data collection issues and communication challenges they are unprepared to solve. The literature offers little practical guidance about developing and conducting research in OST settings, beyond instruments for possible use in evaluation. This article addresses this gap by providing candid advice for evaluators seeking to transition from K-12 to afterschool research. This advice may also help program directors and other stakeholders who want to make the research process work more effectively for them. We aim to help evaluators understand what is and is not possible (or advisable) in afterschool evaluations and to help practitioners serve as more effective partners by anticipating evaluator assumptions and other challenges that can derail a study. The authors bring a variety of experience in researching and evaluating OST programs. They have conducted mixed-method evaluation studies for general programmatic improvement as well as rigorous randomized control trials for federal agencies, including the National Science Foundation and the U.S. Department of Education. Some of them have studied community-based afterschool programs generally, while others have concentrated on specific initiatives in STEM, literacy, and social and emotional learning. Many of the afterschool programs they have researched have taken place in schools, though a few have been located in spaces such as community centers, museums, libraries, and maker labs. This article addresses a broad spectrum of research designs, from formative assessments to confirmatory analyses, in varied OST settings. In their experience, regardless of the intended audience for the report or the level of rigor in the study design, evaluators transitioning to afterschool are challenged by a common set of issues related to data collection and communication. This article addresses those challenges. First, they describe how afterschool is unique--and particularly how it is different from K-12 education. Next, they recommend ways to take those unique features into account when designing and implementing an afterschool study. The final section addresses best practices for forming and maintaining strong partnerships between evaluators and practitioners to produce results that meet the needs not only of funders but also of the program and its staff, students, and families.