Search

In the fall of 2017, ACT began providing a limited number of supports (also known as accommodations) to English Learner (EL) students in the US taking the ACT® test. The goal of the supports is to remove construct-irrelevant variance in students' scores related to limited English proficiency and allow students to more accurately demonstrate their true proficiency on the construct being measured. This paper presents a summary of previous research on the impact of providing supports to EL students and a summary of a preliminary analysis conducted by ACT based on data from states who administered the ACT statewide, captured students' EL status, and provided accommodations to EL students. ACT's future research agenda for providing validity evidence for this initiative is also discussed.

The annual Student Outcomes Survey collects information on the outcomes of two groups of students--those that have completed a qualification (graduates) and those that have completed only part of a course and then left the vocational education and training (VET) system (module completers). At the time of selecting the survey sample, insufficient information is available to identify "actual" module completers. Instead, a sample of potential module completers is drawn that includes students still in training and students who self-report that they have completed the qualification; that is, self-reported graduates. For many years these students have been counted as graduates in survey outputs. This paper examines whether: (1) self-reported graduates were eligible for the qualifications they claimed (in fact, around two-thirds were not eligible); and (2) it is possible to predict a self-reported graduate's eligibility for the claimed qualification using their personal and training characteristics. The report recommends that self-reported graduates should be assigned to the graduate or module completer category using a predictive model. This model incorporates information on the type of training provider, field of education, level of qualification, whether an apprentice or trainee, enrolment type, and method of answering the survey. The current practice of treating self-reported graduates biases the survey results. The predictive model will be used for 2012 Student Outcomes Survey reporting. The methodology will change estimates from previous surveys substantially, so these will be back cast to 2005 using the method described. Appended are: (1) Characteristics of self-reported graduates; (2) Variables fitted and their status in the final logistic model; (3) Regression results; (4) Comparison of self-reported graduates' eligibility for the claimed qualification; and (5) Comparison of survey outputs and modelled estimates. (Contains 16 tables and 1 footnote.)

Language, literacy and numeracy are necessary for greater workforce participation, productivity and social inclusion. Being able to measure the level of proficiency in these skills, and any changes in the level of skills, is important for getting a sense of how well language, literacy and numeracy programs are working. Two measurement tools used in Australia are the Adult Literacy and Life Skills (ALLS) survey and the Australian Core Skills Framework (ACSF). This paper outlines a National Centre for Vocational Education Research study which will investigate whether it is possible to map the performance levels of the ALLS to the performance levels of the ACSF in order to gain more frequent information than is currently available about the literacy and numeracy progression of adult learners against national goals. Appended are: (1) About the ALLS survey and the ACSF; and (2) Delphi method and descriptions of ALLS items. (Contains 1 table and 9 footnotes.)

This paper provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children. To do so, the paper: (1) provides new empirical information about the values of parameters that influence the precision of impact estimates (intra-class correlations and R-squares); (2) examines the implications of planning group-randomized studies for three-level hierarchical situations, using empirical information obtained by estimating two-level hierarchical models (which under many conditions appears to not be problematic); and (3) assesses the implications of the uncertainty that exists when the design of group-randomized studies is based on estimates of intra-class correlations. Data for the paper come from two studies: the Chicago Literacy Initiative: Making Better Early Readers study (CLIMBERs) and the School Break-fast Pilot Project (SBPP). The analysis sample from CLIMBERs comprised 430 4-year old children from 47 preschool classrooms in 23 Chicago public schools. The analysis sample from the SBPP study comprised 1,151 third-graders from 233 classrooms in 111 schools in six school districts. Four appendixes are included: (1) Description of Outcome Measures; (2) Definition of the Multiplier M; (3) Complete Set of Results for Three-Level vs. Two-Level Model Comparisons: Nonstandardized Unconditional Variance Components; and (4) Proofs of the Relationship between Three-Level Models and Two-Level Models in Terms of Precision. (Contains 20 footnotes, 1 figure and 14 tables.) [This document is also sponsored by Judith Gueron Fund for methodological Innovation in Social Policy Research at MDRC, and the Jerry Lee foundation.]

Beginning in 2002, The Chicago Community Trust embarked on an ambitious grant- making strategy to improve education outcomes primarily in the city of Chicago and Cook County, Illinois. Known as The Education Initiative, this effort focused on three priority grant areas: literacy, professional development, and alternative models of schools. In June, 2005 (the 3rd year of the initiative), the Trust awarded a contract to researchers at the University of Wisconsin-Madison, Wisconsin Center for Education Research (WCER) to assess the impact of the Education Initiative. During the 3rd year of the evaluation, the program officers and evaluators were especially interested in pilot testing evaluation guidelines that were developed as part of the overall evaluation of the Education Initiative. These guidelines and related evaluation reporting tools were intended to help grantees tailor projects to the foundations priorities, and help the foundation assess the impact of their diverse grant activities. The authors report on the design and pilot testing of the evaluation guidelines and reporting tools developed to support the education program of the Trust. The next section describes the impetus for developing these tools and the evaluation findings that informed their design. In the fourth section, the authors briefly describe the pilot study design and the participants. They also report findings from the pilot study. In the final section, they describe implications for use of the guidelines and tools by the Trust and other foundations. Appendices include: (1) Draft CCT Evaluation Guidelines; (2) Draft Evaluation Plan Template; and (3) Logic Model Example Adapted from BSF Grantee Model. (Contains 1 table, 7 footnotes, and 2 figures.) [This paper was written with Katie Herrem.]

The University of California's census survey of undergraduates, UCUES [University of California Undergraduate Experience Survey], presents an opportunity to measure both disciplinary and institutional differences in students' academic experience. Results from nearly 60,000 responses (38% response rate) from the 2006 administration found greater variance among majors within an institution than between equivalent majors across institutions. Cluster analysis techniques were employed to establish disciplinary patterns, with traditional distinctions between hard and soft sciences generally supported. Reporting practices called into question range from institutional comparisons that ignore academic program mix and discipline to campus performance comparisons that do not recognize pedagogical differences by academic major. More specifically, these results suggest that calls for comparable institutional performance measures, as proposed by the Spellings Commission, must take into consideration disciplinary differences in instruction. (Contains 1 table and 9 figures.) [A version of this paper was presented at the SERU Project Symposium, "Assessing the Undergraduate Experience in the Postmodern University" (Berkeley, CA, April 25, 2007).]

Roderick Watts and Omar Guessous of Georgia State University conducted exploratory research to investigate the link between math and civic engagement. Their research is based on an evaluation of the Young People's Project (YPP), a national program that recruits, trains, and deploys high school and college Math Literacy Workers for mentoring middle and elementary school students. The purpose of this study is to examine whether the Young Peoples Project, as a community-based math and social justice organization for youth, contributes to the math, academic, sociopolitical, ethnoracial identity, and positive youth development of participants. Five specific research questions addressed by this study were: (RQ #1) Program effect: Do YPP participants show more favorable outcomes than the two comparison groups on outcome indicators of interest? (RQ #2) Dosage effect: Do veterans show more favorable outcomes than novices on the outcome indicators of interest? (RQ #3) Program x dosage interaction: Is the dosage effect stronger among YPP participants than among the School Controls? In other words, is there an interaction effect between program membership and level of seniority? (RQ #4) Math--SPD link: Is there a positive relationship between math confidence and indicators of sociopolitical development? (RQ #5) Math--SPD x Program: Is this link moderated by group membership, such that YPP participants demonstrate a stronger positive relationship between math confidence and indicators of sociopolitical development than the School Controls (SC)? Participant in this cross-sectional study were Chicago-area residents in grades 8 and above who were administered a paper-and-pencil survey in a group format at the site of each YPP and After School Matters (ASM) program. YPP participants were encouraged to recruit a peer who did not participate in YPP to serve as matched comparisons. Additionally, ASM participants from three Chicago public high schools were recruited for participation in this study, to serve as an additional comparison group (N=73). There were a total of 196 participants. Each variable of interest was assessed using one or more self-report survey instruments that were answered on a numerical Likert scale. These included measures of social analysis, sense of urgency, societal involvement, cultural and ethnoracial identity. academic and math self-concept, and positive youth development. The five research questions were examined using Univariate Analysis of Covariance and Hierarchical Multiple Regression. Among the outcomes was a favorable link between YPP participation and aspirations for mathematics education, and another was evidence of sociopolitical development. The third was an exploration of other youth development outcomes of interest to the program and the researchers. Sense of Agency was positively related to math self confidence and negatively related to math self-doubt. Similarly, commitment to societal involvement activities was negatively related to math self-doubt. And in the only instance were the ethnoracial variables came into play, cultural pride reinforcement was found to positively correlate to math self confidence. The relationship between sociopolitical development and math confidence was consistently stronger among YPP participants compared to School Controls. This suggests that YPP contributed to a link between math knowledge or reasoning and to both sociopolitical agency and commitment; that is, YPP experiences are associated with the idea that "math is power." (Contains 14 tables and 9 figures.) [This Working Paper was produced by CIRCLE (The Center for Information and Research on Civic Learning and Engagement). For CIRCLE Working Paper 49, see ED494074.]

A long-held ambition for many educators and assessment experts has been to integrate summative and formative assessments so that data from external assessments used for system monitoring may also be used to shape teaching and learning in classrooms. In turn, classroom-based assessments may provide valuable data for decision makers at school and system levels. Currently there are important technical barriers to this kind of seamless integration. Nevertheless there are a number of promising developments in the field. Ongoing research and development aims at improving testing and measurement technologies, as well strengthening classroom-based formative assessment practices. Improved integration of formative and summative assessment will require investments in new testing technologies, teacher training and professional development, and further research and development.

Many urban districts have adopted interim assessments in recent years as a supplement to annual testing. Possible purposes for such tests include monitoring and assisting student progress, aligning the curriculum, and practicing for state exams, but little is known about how effectively real systems advance these purposes. The Providence Public School District (PPSD) implemented a system of quarterly assessments at every grade starting in 2004 and discontinued them in 2007. During the implementation period, authors interviewed district officials about what they were trying to accomplish, and teachers and other school personnel about their experiences with the assessments and their use of the results. Evidence of effectiveness as well as limitations were found. Uncertainty about the magnitude of positive impacts raised questions about the net value of the assessments given their considerable costs to districts, teachers, and students. The authors conclude that interim assessments are costly, and costs increase if the desired effects on student achievement require significant investments in professional development of teachers. Sound policy, therefore, should be carefully designed and evaluated. Two appendixes are included: (1) Interview Protocol; and (2) Findings from Interviews and Survey. (Contains 3 tables.)

This paper estimates the effects of state-imposed curriculum mandates on the test scores of public-school students who took the SAT in 2001. By 1998, 14 states had mandates that high-school students should take an economics course. For these states, the proportion of public-school students taking high-school economics was around twice that of states without mandates. The mandate may be interpreted as a regulation on input use in the education sector, potentially impairing the efficiency of schools. It is hypothesized that where there is a mandate, test scores should be lower. The merit of the mandate on scores can be assessed empirically. Using microlevel data on a large sample of students across the United States, this inquiry tests whether students' academic performances are higher or lower in states where there are mandates. This test offers a direct evaluation of the efficiency of the mandate. The mandate reduces test scores by as much as 0.25 standard deviations for those students who would not otherwise have enrolled. This paper sets out the theoretical explanation from the imposition of a mandate, describes the dataset, sets out the estimation method, and reports the results. (Contains 6 tables and 24 references.) (Author/WFA)

Curriculum suppliers, state departments of education, and home school leaders, are the sources used to estimate that between 248,500 and 353,500 school-aged children (K-12) were educated at home in the 1990-91 school year. This paper explains how this estimate was derived. It provides information on process, possible bias, alternate survey collection of data, and why the three methods were chosen. In the estimates reported, most adjustments to actual data were based on a survey of a sample of the membership of the Home School Legal Defense Association. In deriving figures for the first method, a survey of 16 suppliers of curricular packages yielded an estimate of between 309,000 and 353,500 K-12 children who are educated at home. For the second method, 37 state departments of education reported 82,061 children for whom parents have filed papers with state or local officials, representing a fraction of the total. For the third method, information from 25 home schooling leaders in 17 states yielded a national estimate of between 248,500 and 304,000. A brief analysis of recent trends with a conclusion is given. Appended are home school data collected by states, information from associations, and a summary of selected research reports from two state departments of education. (RR)

Higher education institutions (HEIs) have experienced increasing pressures to provide accountability data and consumer information on the quality of teaching and learning. Existing ratings and rankings of HEIs tend to neglect information on student learning outcomes. Instead, they focus on inputs, activities and research outputs, such as resources used, classes taught, and articles published. Such indicators provide no indication of the degree to which HEIs actually develop the knowledge and skills of their students. In most countries, hardly any comparable information is available on the educational quality of different programmes and institutions. In some countries, approaches to assess higher education learning outcomes have been developed, but little cross-country information is available on the characteristics of the instruments used. This paper provides an overview of experience gained in this domain across OECD and partner countries. Based on illustrative evidence collected for 18 assessment instruments, it examines conceptual, organizational and methodological aspects of existing assessments. It proposes a typology of higher education learning outcomes and reviews the ways in which these have been assessed across countries. Examples are drawn from Australia, Brazil, Mexico, the United Kingdom and the United States. (Contains 2 footnotes and 7 tables.)

As of June 1995, the federal government had granted to 28 states waivers from current federal laws and regulations, allowing states to test a wide range of welfare reform approaches. This study, conducted by the National Center for Children in Poverty, examined the evaluation plans of 21 waiver experiments in 17 states, focusing on the impact of welfare reform on family income, parenting stress behaviors, and children's access to services. For each impact study, recipients of Aid to Families with Dependent Children were randomly assigned to either an experimental group participating in the welfare reform approach or a control group. The major findings indicated that: (1) evaluators are collecting a great range of child-related data, but few variables were common among many states, reducing the chances for cross-state learning to inform national policy; (2) some waiver evaluations may lead to new information regarding children's education, particularly for states evaluating "learnfare" approaches; (3) some evaluations track potential harms to children related to family dysfunction, foster care placement, and substance abuse; (4) the waiver evaluations will not be able to demonstrate the ways welfare reform affects children's growth and development because very few variables directly measure such changes; and (5) variables in some evaluation plans related to children's services, but few related to quality services or to outcome measures indicating the effectiveness of those services, especially for child care and child health. (Appendices contain background information on selected states, research questions, and children's well-being variables. Contains 24 references.) (KB)

This study of higher education efficiency evaluation consists of five country reports from five different regions of the world: Colombia, Jordan, Malaysia, the Netherlands, and Nigeria. The reports describe the approaches to the evaluation process in each country and in the process highlight common concerns and awareness. The studies were all commissioned by the United Nations Educational, Scientific, and Cultural Organization and provide an overview of the various evaluation procedures, mechanisms and instruments currently in force in the selected countries which are intended to measure the efficiency of systems and institutions of higher education in relation to their specific economic, social and cultural roles. Each report presents one of the studies and proceeds by describing the general evaluation philosophy and methodology, the procedures at the national level and procedures in a representative sample of institutions with a conclusion analyzing the efficiency of the mechanisms at various levels. The book's concluding chapter provides an analysis of all the studies taken together. That chapter points out the striking diversity among higher education systems while arguing that the commonalities are also significant. This analysis touches on national systems, the institutions, and presents seven proposals for a future, more focused study. (JB)

This paper examines how schools choose class size and how households sort in response to those choices. Focusing on the highly liberalized Chilean education market, we develop a model in which schools are heterogeneous in an underlying productivity parameter, class size is a component of school quality, households are heterogeneous in income and hence willingness to pay for school quality, and schools are subject to a class-size cap. The model offers an explanation for two distinct empirical patterns observed among private schools that accept government vouchers: (i) There is an inverted-U relationship between class size and household income in equilibrium, which will tend to bias cross-sectional estimates of the effect of class size on student performance. (ii) Some schools at the class size cap adjust prices (or enrollments) to avoid adding another classroom, which produces stacking at enrollments that are multiples of the class size cap. This generates discontinuities in the relationship between enrollment and household characteristics at those points, violating the assumptions underlying regression-discontinuity (RD) research designs. This result suggests that caution is warranted in applying the RD approach in settings in which parents have substantial school choice and schools are free to set prices and influence their enrollments.

Over the past several years, content experts, psychometricians, and researchers have been hard at work developing, refining, and studying the digital SAT. The work is grounded in foundational best practices and advances in measurement and assessment design, with fairness for students informing all of the work done. This paper shares learnings from key research studies that have informed the design and development of the digital SAT and the current understanding of how well the assessment is working for its intended uses.

Program impact methodology--usually referred to as evaluation research--is described as it is developing in the United States. Several problems face the field of evaluation research. First, those issues grouped as "meta-scientific" include: (1) the distinction between qualitative and quantitative studies; (2) the separation of implementation and evaluation; (3) maximizing replication and criticism; and (4) evaluation research as normal rather than extraordinary science. Other problems are those of statistical issues. These include: (1) all problems related to the interrupted time-series design; (2) regression adjustments as substitutes for randomization; (3) problems with randomized experiments; (4) attrition and differential attrition; and (5) regression-discontinuity design. Political-methodological problems form a third major group of evaluation research problems, which include resistance to evaluation. It is contended that many new methodological problems will be apparent as the field moves from the laboratory into social problem evaluation. Three figures, seven graphs, and a 107-item list of references are provided. (SLD)