Search

This report is a companion piece to WCER working paper 2018-14, "A Cyberinfrastructure for Design-Based Research: A Collaborative Design Project Proposal." That paper stressed that the educational research community would be well served by a mutually created cyberinfrastructure that encourages and supports engagement by multiple design-based researchers in working toward answers to important theory-driven research questions, moving the field toward a "bigger science" approach. This current report, prepared by the Caktus Group for the University of North Carolina School of Education, outlines a process by which a single digital tool specific to design-based research (DBR) methodologies could be built. The goal in commissioning this report was to discover the most effective and suitable system to recording and visually representing learning environment workflows, building searchable collections of well-catalogued assets, and securely sharing data and analysis with the community of peers. With that goal in mind, the Caktus Group, in collaboration with Dr. Sharon Derry, built and administered a survey to better understand the needs of the DBR community. Following the survey, and informed by its results, the Caktus Group facilitated a discovery workshop to identify design and technical requirements for the system, outline its desired architecture, and gather information for an estimate of the cost of building such a system. This report proposes a full-scale design for a workflow visualization system to support design-based research in the learning sciences. It builds on participant experiences with prototypes and incorporates outcomes of survey research and a two-day intensive design workshop led by Caktus Group that took place on July 31-August 1, 2017, in Durham, North Carolina. The work was conducted under direction of Sharon Derry. Invited participants in the workshop were Janice Anderson, Kelly Barber-Lester, Lana Minshew (all from University of North Carolina at Chapel Hill), and Vanessa Svihla (University of New Mexico). [For the companion report, "A Cyberinfrastructure for Design-Based Research: A Collaborative Design Project Proposal," see ED593634.]

Like numerous others before us, we argue that the educational research community would be well served by a mutually created cyberinfrastructure that encourages and supports engagement by multiple design-based researchers in working toward answers to important theory-driven research questions, moving our field toward a "bigger science" approach. We reach out to members of the educational research community to join with members of other relevant fields, such as computer science, to design, build, support and participate in a mutually established cyberinfrastructure and user community to support design-based research. We seed the discussion by offering a concrete initial proposal to illustrate the possibility of implementing this idea. [For the companion report, "Product Discovery Report for a Visual Data Management System. WCER Working Paper No. 2018-15," see ED593637.]

This paper examines the current state of the literature on Integrated Planning and Advising for Student Success (iPASS), an increasingly popular approach to technology-mediated advising reform. We limit our analysis to sixty relevant documents that have been released since 2010. We categorize these items into four different groups based on their aim and rigor: (1) "descriptive documents" (39 items) that describe processes and challenges of iPASS implementation, (2) "output reports" (12 items) that examine usage data to better understand iPASS implementation, (3) "correlational studies" (5 items) that examine non-causal associations between different functionalities of iPASS tools and student outcomes, and (4) "rigorous outcomes studies" (4 items) that employ experimental or quasi-experimental methods to provide causal or near-causal estimations of iPASSoriented interventions. iPASS is a relatively recent reform approach, and while few studies are available that rigorously evaluate its effects, early findings suggest that individual components of iPASS interventions may have a positive impact on student outcomes, including persistence and credit accrual. This paper also offers suggestions on how colleges can use non-causal research to support and improve iPASS. Appended to the report are figures.

According to the National Institutes of Health, "Translational research includes …the process of applying discoveries generated during research in the laboratory, and in preclinical studies, to the development of trials and studies in humans… [and] research aimed at enhancing the adoption of best practices in the community. Following this tradition, this paper defines translational research in the Social-Behavioral-Educational (SBE) sciences as the recursive, bidirectional process of integrating scientific knowledge and research-based discoveries into community and societal practices and policies. Because there is a woeful lack of understanding about the science of translation, or how this very difficult work happens. Little is actually known about the strategies, structures, and processes of translational research that influence the capability and probability of diverse researchers from distinct disciplines collaborating effectively to solve significant social, behavioral, educational, and mental health problems. Barriers such as discipline-centric terminologies, traditional methodological approaches, time constraints, and narrowly conceived and executed dissemination strategies are significant. To accelerate the transmission of knowledge across the spectrum of theory, experimentation, implementation and diffusion, an augmented translational research approach specific to the SBE sciences is needed. The specific aims of this initiative are two-fold. First, the initiative will uncover various aspects (phases and processes) of translation that are most effective in addressing a range of significant social, behavioral, educational and mental health challenges plaguing society. Second, the knowledge gained about the science of translation will be applied to ameliorate specific social, behavioral, educational, and mental health problems by targeting specific substantive issues in these areas that impede individual, familial, social, and societal health. Through advanced understanding of the mechanisms by which meaningful integration of researchers from multiple disciplines and across the basic, applied, community and policy spectrum occurs, translational researchers will provide critical core support to augment the efficient translation around targeted SBE research. Relatedly, structures developed as a product of this initiative will serve as a hub to inform significant training, practice and policy advances among researchers in the SBE sciences. [Development of this white paper was completed with partial support from the University of Nebraska-Lincoln Office of Research and Economic Development (ORED).]

Lottery-based identification strategies offer potential for generating the next generation of evidence on U.S. early education programs. Our collaborative network of five research teams applying this design in early education and methods experts has identified six challenges that need to be carefully considered in this next context: (1) available baseline covariates may not be very rich; (2) limited data on the counterfactual; (3) limited and inconsistent outcome data; (4) weakened internal validity due to attrition; (5) constrained external validity due to who competes for oversubscribed programs; and (6) difficulties answering site-level questions with child-level randomization. We offer potential solutions to these six challenges and concrete recommendations for the design of future lottery-based early education studies.

The charter school sector in the United States has grown steadily since the first charter school opened in 1992. As of the 2015-2016 school year, more than 6,800 charter schools served nearly 3 million students in forty states and the District of Columbia. Overall, research suggests that the average charter school performs about the same as nearby traditional public schools, but there is great variation in the effects of charter schools. Some charter schools are successful in boosting student achievement and others are not, which raises the question of what characteristics distinguish good charter schools from bad. This paper addresses this issue by summarizing the research on factors associated with successful charter schools. The research suggests that urban charter schools and charter schools primarily serving low-achieving and low-income students have the strongest positive impacts on student achievement. The policies most consistently found to be associated with positive charter school impacts include long school days or years, comprehensive behavioral policies with rewards and sanctions, and a mission that prioritizes boosting student achievement. In addition, moderately strong evidence suggests that high-dosage tutoring, frequent feedback and coaching for teachers, and policies promoting the use of data to guide teachers' instructional practices are positively associated with charter schools' achievement impacts.

Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not induce small-sample bias in the effect estimates. Using this result as a building block, we develop a novel approach that uses machine learning techniques to reduce the variance of the average treatment effect estimates while guaranteeing that the effect estimates remain unbiased. The framework also highlights how researchers can use data from outside the study sample to improve the precision of the treatment effect estimate by using the auxiliary data to better model the relationship between the covariates and the outcomes. We conclude with a simulation, which highlights the value of using the proposed approach.

Recent changes to the Child Care and Development Fund (CCDF) require a state's child care market price survey to: (1) be statistically valid and reliable and (2) reflect variations in the cost of child care services by geographic area, type of provider, and age of child. States may use an alternative methodology for setting payment rates--such as a cost estimation model--to take into account the cost of meeting quality requirements. This brief provides an overview on how states can adapt to meet these new requirements and also align price surveys to better support efforts to improve the access that children have to high-quality early learning programs.

The present paper, which is intended for a diverse audience of evaluation researchers, applied social scientists, and research funders, provides a broad overview of the conceptual and statistical issues involved in using multisite randomized trials to learn "about" and "from" variation in program effects across "individuals," across policy-relevant and theoretically relevant "subgroups" of individuals, and across program sites. Learning about variation in program effects involves detecting and quantifying this variation. Learning from variation in program effects involves studying the factors which predict or explain it. The paper is divided into four sections, plus a brief final discussion. The first section introduces the concepts and issues involved. Section 2 focuses on detecting and quantifying variation in effects of program "assignment," which are often referred to as effects of intent to treat (ITT). Section 3 extends the discussion to variation in effects of program "participation," which are often referred to as a complier average causal effect (CACE) or a local average treatment effect (LATE). Section 4 considers "moderators" of program effects (individual-level or site-level factors that influence the sign and magnitude of these effects) and "mediators" of program effects (individual-level or site-level "mechanisms" by which these effects are produced). Appendices include: (1) Characterizing the Impact of an Intervention on the Mean and Variance of Outcomes and Program Impacts for a Population of Individuals; (2) Derivation of the HLM [hierarchical linear model] Estimator of the Cross-Site Mean and Variance of Program Impacts.

This systematic review investigates the relevance of general pedagogical knowledge for successful teaching. It synthesises the empirical evidence of 10 769 teaching professionals and 853 452 students from primary to tertiary education in 21 countries. The meta-analysis of 20 quantitative studies revealed significant effects for teaching quality and student outcomes (Cohen's d = 0.64 and 0.26), indicating that more knowledgeable teachers achieve a three-month additional progress for students. The three themes emerging from 31 qualitative studies underline that general pedagogical knowledge is a crucial resource for teaching. Results also show that teaching requires knowledge about a range of topics, specific skills and other competences to transform knowledge into practice. Teachers need training and practical experience to acquire knowledge, which they apply according to the pedagogical situation at hand. The results allow for important conclusions for policy, practice and research.

The increasing tension between the perceived necessity of a college degree and the challenge of paying for it has led to a proliferation of financial aid policy in the U.S. and around the world. More students are receiving more aid today, and more different types of aid, than ever before. Half a century of policy experimentation has led to an equally impressive accumulation of research evidence, facilitated by methodological advances and the widespread availability of student-level administrative data. In this chapter, we present the economic rationale for financial aid, a summary of how aid works in the U.S. context, and common methodological challenges in the study of financial aid. We then review the evidence from both inside and outside the U.S. on the causal impact of a variety of financial aid policies and programs on students' college decisions, attainment, and post-college outcomes, and summarize the overarching themes with respect to margins of impact, mechanisms, and heterogeneity. We conclude with a discussion of future directions for research.

On April 11 and 12, 2014, the American Association of School Librarians (AASL) held "Causality: School Libraries and Student Success" (CLASS), an IMLS-funded national forum. Dr. Thomas Cook, one of the most influential methodologists in education research, and a five member panel of expert scholars and practitioners led 50 established and emerging school library researchers in articulating a national research agenda to investigate causal phenomena in school library instruction, resources, and services. Research in academic achievement and school librarianship has a strong foundation of over 25 correlational studies in which school librarians' activities and school library programs have been explored in relation to student learning and teacher support. The results of these studies suggest that complementary research should be conducted to establish a causal relationship between the work of effective school librarians and the creation of motivated, engaged, and agile learners. The goals of this white paper are to: (1) Capture the rich discussion emanating from the CLASS forum surrounding research and causality; (2) Propose a progression of research methods and projects that will support efforts toward theory building, exploratory research, and demonstration research; and (3) Outline mechanisms by which a community of scholars can be cultivated and nurtured toward furthering the research agenda and its activities. The following are appended: (1) Participants; (2) Further Reading; (3) Timetable; and (4) Communications Plan.

In this paper, the authors examine the relative labor market gains for first-time college students who enrolled in the North Carolina Community College System in 2002-03. The medium-term returns to diplomas, certificates, and degrees are compared with returns for students who accumulated college credits but did not graduate. The authors also investigate the returns to credit accumulation, subject field, and transfer and the early trajectories of wages for different student subgroups during the 2000s. The analysis is based on student-level administrative record data from college transcripts, Unemployment Insurance wage data, and enrollment and graduation data from the National Student Clearinghouse across 830,000 community college students between 2001 and 2010. Findings from this study confirm those from earlier work: The returns to certificates and diplomas were weak, but associate and bachelor's degrees yielded very strong returns; even small accumulations of credits had labor market value; and the returns to health sector credentials were extremely high. Returns were much higher for female students than for male students. Despite the Great Recession, analysis reveals little evidence that the returns to college decreased over the latter half of the 2000s. However, medium-term estimates likely understate the full value of college credentials, particularly bachelor's degrees. The following table is appended: Descriptive Frequencies for 2002-03 NCCCS Cohort by Highest Award Earned.

This paper reviews the new directions in institutional research on undergraduate racial/ethnic and socioeconomic diversity at the University of California, Berkeley. The use of SERU/UCUES and other web-based census surveys has made possible more detailed and extensive analysis of student diversity. Included is research on an expanded number of racial/ethnic groups and on multiracial students, the significance of the African American experience, implications of the new IPEDS racial/ethnic reporting requirements, and a closer examination of Pell Grant and first-generation college students. UCUES survey results are used to development a more comprehensive parental education and immigrant generation diversity (EID) typology that is then used to examine the interrelationships among student demographics and various facets of the undergraduate academic experience. Finally, an analysis of student accounts of the experience of diversity at Berkeley provides an example of how web-based census surveys afford new opportunities for cost-effective qualitative diversity research. (Contains 19 tables, 1 figure and 1 footnote.)

The Australian Vocational Education and Training Management Information Statistical Standard (AVETMISS) for vocational education and training (VET) providers (referred to as "the Standard") is currently being reviewed by the National Centre for Vocational Education Research (NCVER). The Standard is the framework that underpins the National VET Provider Collection. It provides consistency in data definitions, which ensures accurate data for use in national data collections, where information is compiled from many different sources. This discussion paper is the first step in the review. It provides a framework for feedback and identifies key issues for consideration. There are many issues associated with the specification of a Standard, such as what information is to be collected, from whom and how often. Decisions about implementation of the AVETMIS Standard are made by a range of different bodies and, while the focus of this paper is primarily on the information to be collected, comments on the timing and coverage are also welcome. Two appendices are included. (Contains 1 figure.)

This report provides an overview and discussion of the past decade of academic evidence on the causal effects of resources in schooling on students' outcomes. Early evidence lacked good strategies for estimating the effects of schools resources, leading many people to conclude that spending more on schools had no effect. More recent evidence using better research designs finds that resources do matter, but the range of estimates of the impacts is quite wide. The review devotes special attention to differences across the early years, primary and secondary phases. Theoretical work has indicated that interventions early in a child's life may be more productive than interventions later on. However, although there are more examples of good quality studies on primary schooling, the existing body of empirical work does not lead to a conclusive case in favour of early interventions. Three appendices present: (1) Differences in characteristics between high and low-resource schools; (2) Benchmarking and Comparing the Size and Strength of Effects Across Different Studies; and (3) An illustration of the theoretical model (one in which events in one period depend on events in previous periods) used by Nobel-laureate, James Heckman in his argument for the efficacy of early investment in children over investment at a later stage. [This work was commissioned by Ofsted as part of the "Access and Achievement in Education--20 Years On" project.]

National Centre for Vocational Education Research (NCVER) is undertaking a consultation process to determine the next set of National Research Priorities which will guide research activities in the post-compulsory education and training arena, particularly in relation to vocational education and training. This discussion paper, and the deliberations it aims to prompt, identifies a number of key themes related to priorities for future research. A section that discusses NCVER's current research and statistics is appended. (Contains 3 boxes and 5 tables.) [Funding for this paper was provided through the Department of Education, Employment and Workplace Relations.]

Community college students typically have access to a large selection of courses and programs, and therefore the student transcripts at any one college or college system tend to be very diverse. As a result, it is difficult for faculty, administrators, and researchers to understand the course-taking patterns of students in order to determine what programs of study they appear to be pursuing. Attempting to examine these patterns and then comparing them with listed program requirements would be a very time-consuming activity. The most common way of assigning a program of study to a student--picking the subject in which she has taken the most courses--is overly simple, because many programs require courses across several subjects. However, because students who have similar patterns of course-taking in terms of subjects and particular courses taken are likely to be in similar programs, clustering can be a useful way to make sense of the relevant data. Clustering allows researchers to group similar items into clusters, relying only on a measure of the similarity of those items. In this paper, we apply a clustering algorithm to the problem of understanding college transcripts, which serve as the items to be clustered. To our knowledge, this is the first effort to organize transcripts based on their course content using clustering. We base the measure of similarity on the proportion of curricular subjects that each transcript has in common with every other one. Our data are community and technical college transcripts for a cohort of students who first entered the Washington State system during the fall of the 2005-06 academic year and who had no prior postsecondary experience. We used our clustering algorithm to separately cluster liberal arts and career-technical students. We found that the algorithm did a good job of separately clustering each of these groups. The clusters roughly corresponded to programs of study, so we were able to estimate how many students were undertaking each program and what subjects students were studying within each cluster. We were also able to examine the demographics and the completion and transfer rates of the students within each cluster, in order to get an idea of what types of students were in each program of study and how successful they seemed to be in college. We found substantial variation on these dimensions as well as on the extent to which students' programs were either concentrated in a single subject or spread across several subjects. Clustering is a powerful way to understand the course-taking patterns of students and assign programs of study. It makes few prior assumptions about the data; rather, it allows the data to organize itself based on a similarity measure. It relieves the analyst of determining what the program categories should be. It has the ability to detect patterns of activity across subjects within student transcripts. Note that although we have applied this method to community college students, it is applicable at all levels of postsecondary education. We conclude that this method would be useful to researchers throughout education who are trying to understand student course-taking patterns and programs of study, and who need to organize large amounts of transcript data. Appended are: (1) Details of the Cluster Analysis; (2) Details of the Cluster Analysis; and (3) Five Clusters of Liberal Arts Student Transcripts. (Contains 10 tables and 3 footnotes.)

Although educational leaders, policy makers, and researchers have long emphasized the importance of science, technology, engineering, and mathematics (STEM) for the country's continued prosperity, increasing participation in STEM has remained a challenge for both the education and scientific communities (Pearson & Fechter, 1994; National Academies, 2007; President's Council of Advisors on Science and Technology, 2010; American Association for the Advancement of Science, 1989). A historic imbalance in STEM participation persists whereby proportionately fewer female and minority students enroll in STEM courses and seek employment in STEM professions (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 2010). Historically the subject area instruction of English Language Learners (ELLs) in math and science has received limited attention by the research community, which has instead focused on English proficiency and literacy of ELLs (Lee, 2005). Thus, the research fields of math and science education have developed independent of the research field involving ELLs. However, this trend is beginning to fade as science and math education researchers and their ELL education peers have begun to crossover to one another's domains to produce new understandings and strategies in math and science education specific to the needs of English Language Learners. For its part, the National Science Foundation's (NSF) Discovery Research K-12 (DR K-12) program is contributing to the growth of knowledge in these areas of crossover by funding ELL math and ELL science education projects that seek to understand and improve STEM learning and STEM instruction with ELLs. To investigate the contribution of the DR K-12 portfolio to the knowledge base on math and science learning among ELLs, the Community for Advancing Discovery Research in Education (CADRE) designed a study to explore the ELL work that is being conducted in the DR K-12 projects. This paper summarizes the work of this study. It begins with a description of the methodology employed, followed by a presentation of the findings, and finally a discussion of the conclusions drawn from this work. The findings are organized into discussions about the expertise held by the DR K-12 ELL researchers and research teams, the characteristics of the ELL research being conducted by the DR K-12 projects, and a comparison of the research conducted by the DR K-12 projects with published research on ELL-science education and ELL-math education. The following appendices are included: (1) Description of DR K-12 Projects; (2) Research in Science Education and ELL; and (3) Research in Math Education and ELL.

This paper provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children. To do so, the paper: (1) provides new empirical information about the values of parameters that influence the precision of impact estimates (intra-class correlations and R-squares); (2) examines the implications of planning group-randomized studies for three-level hierarchical situations, using empirical information obtained by estimating two-level hierarchical models (which under many conditions appears to not be problematic); and (3) assesses the implications of the uncertainty that exists when the design of group-randomized studies is based on estimates of intra-class correlations. Data for the paper come from two studies: the Chicago Literacy Initiative: Making Better Early Readers study (CLIMBERs) and the School Break-fast Pilot Project (SBPP). The analysis sample from CLIMBERs comprised 430 4-year old children from 47 preschool classrooms in 23 Chicago public schools. The analysis sample from the SBPP study comprised 1,151 third-graders from 233 classrooms in 111 schools in six school districts. Four appendixes are included: (1) Description of Outcome Measures; (2) Definition of the Multiplier M; (3) Complete Set of Results for Three-Level vs. Two-Level Model Comparisons: Nonstandardized Unconditional Variance Components; and (4) Proofs of the Relationship between Three-Level Models and Two-Level Models in Terms of Precision. (Contains 20 footnotes, 1 figure and 14 tables.) [This document is also sponsored by Judith Gueron Fund for methodological Innovation in Social Policy Research at MDRC, and the Jerry Lee foundation.]