Search

Background/Context: The evolution of validity understandings from mid-century to now has emphasized that test validity depends on test purpose--adding consequence considerations to issues of interpretation and evidentiary warrants. Purpose: To consider the tensions created by multiple purposes for assessment and sketch briefly how we got to where we are; furthermore, to address two critically important purposes: the accountability purpose versus the learning purpose for assessment. Research Design: This is an analytic, closing commentary to this special section. Conclusions: When a test is used as an educational reform, the theory of action behind the reform should be made explicit and that theory or series of claims and assumptions is what should be examined in the validity evaluation. As to the prospect of improving the teaching profession by the use of value-added methods, I believe that this is an overly ambitious use of a potentially useful statistical tool. As these systems are being implemented, we can and should conduct validity studies designed to detect plausible shortcomings and side effects as well as intended outcomes.

Background/Context: Validity issues are often discussed in technical terms, but the context changes when measures enter broad public debate, and a wider range of interests come into play. Purpose: This article, part of a special section of TCR, considers the political dimensions of validity questions as raised by a keynote address and panel discussion originally held at Teachers College in March 2012. Research Design: This is an analytical and reflective piece, based on the author's participation in the panel and drawing on his experience and writing on the broad issue of politics and research. Conclusions: Technical expertise in the construction and interpretation of measurements is important in the new world of high-stakes and "evidence-based" education policy, but the political realities, when measures go public, make the exclusive reliance on such expertise problematic.

Background/Context: There is a deep and yawning chasm between the world of tests and testing practices as they ought to be and the actual tests and testing practices now imposed on American students, educators, and schools. That chasm of theory and practice is a function of the dominant paradigm of educational reform, with its theory of action that schools must be remade in the image and likeness of a corporation. Purpose: To explore the role, development, and implications of assessment use in the market model of education reform. Research Design: Analysis of the recent publication of the value-added measurements found in the Teacher Data Reports of the New York City Department of Education. Conclusions: Since standardized tests provide data for a "bottom line," they have been widely embraced in some circles as a basis for making "high stakes" decisions that hold individuals and schools accountable. In the market model of education reform, questions about the validity and reliability of the tests and of their use for "high stakes" decisions are dismissed as efforts to avoid accountability.

Background: Facing growing accountability pressures, Teacher Work Samples (TWSs) as a model of performance-based assessment is of growing significance in teacher education. Developed at Western Oregon University and widely adopted and adapted, proponents claim that the model is "real," "natural," "meaningful," and "helpful" (G. R. Girod, 2002). Research Questions: The study addresses three questions: (1) How do sample raters understand their responsibilities? (2) What are the underlying interactive rules and strategies used by raters to achieve their aims, and how are they employed? (3) What issues or concerns should teacher educators interested in using TWS methods address as they seek to demonstrate candidate quality and program value? Research Design: Conversation analysis, ethnomethodology. Data Collection and Analysis: Ten TWS scoring conversations conducted by four teams were recorded and analyzed to identify interactive rules and strategies. Scoring teams were composed of one tenure-track elementary teacher education faculty member and one clinical teacher education faculty member. Excerpts from a TWS case judged marginal are presented and analyzed. Findings: From the case, a set of interactive rules (tenure-track faculty speak first; the efficiency and equivalence rules; and scorers are prepared) and strategies (splitting the difference; rubric simplification; previewing scores; and rubric stretching) are identified, and implications of their use are discussed for assessment validity, fairness, content quality and coverage, meaningfulness, and cognitive complexity (R. L. Linn, E. L. Baker, & S. B. Dunbar, 1991). Conclusions: This study raises a number of concerns about the expectations for and use of Teacher Work Samples and cautions about their use for high-stakes assessment.

Assessments in English that are constructed for native English speakers may not provide valid inferences about the achievement of English language learners (ELLs). The linguistic complexity of the test items that are not related to the content of the assessment may increase the measurement error, thus reducing the reliability of the assessment. Language factors that are not relevant to the content being assessed may also be a source of construct-irrelevant variance and negatively impact the validity of the assessment. More important, the results of these tests used as the criteria for identification and classification of ELL students, particularly those at the lower end of the English proficiency spectrum, may be misleading. Caution must be exercised when the results of these tests are used for special education eligibility, particularly in placing ELL students with lower English language proficiency in the learning/reading disability category. This article discusses psychometric issues in the assessment of English language learners and examines the validity of classifying ELL students, with a focus on the possibility of misclassifying ELL students as students with learning disabilities.

Background: A large body of survey-based research asserts that the quality and strength of student-teacher relationships (STRs) predict a host of academic and nonacademic outcomes; however, advances in survey design research have led some to question existing survey instruments' psychometric soundness. Concurrently, qualitative research on STRs has identified important developmental and sociocultural variation in the ways students define, understand, and react to relationships with their teachers. The questions raised by survey methodologists, together with the conceptual elaboration of STRs, suggest that survey instruments used to assess STRs are due for a systematic review. Purpose/Research Questions: This review of survey instruments examines the strengths and shortcomings of existing measures of STRs. Specifically, we ask: How do student self-report survey instruments assess STRs? We examined the extent to which these instruments reflect current survey design principles and existing knowledge about how STRs work, particularly for adolescents. Research Design, Data Collection, and Analysis: A systematic search of peer-reviewed journal articles that (a) focused on North American middle- or high-school students, (b) linked STRs to student outcomes, and (c) used a student-report measure of STRs yielded 66 studies for which we could obtain the full instrument. Instruments were analyzed using a literature-informed protocol and an iterative process that resulted in strong inter-rater agreement. We used tables and matrices to examine patterns, themes, and outliers in our coded data. Findings: The 66 studies varied considerably with respect to how they operationalized STRs and how they addressed the validity of their instruments. Similar survey items were used to measure different constructs, and constructs with the same names were measured inconsistently across studies. Many instruments were limited by (a) items that included words with ambiguous meanings, (b) inconsistent identification of instruments' focal students and teachers across instruments, and (c) the use of negatively worded items to measure STRs' strength. Conclusions and Recommendations: If STR research is to meet its promise to guide and inform teachers' efforts to develop and sustain effective relationships with their students, the field needs to properly identify those behaviors that make a difference for different students and those that do not. The next generation of student-report STR survey instruments requires more stringent attention to construct specification and validity, as well as to item generation (specifically, language use), in order to most effectively measure and identify aspects of STRs that affect student performance and well-being.

In the present offering we challenge the presumption that the educational testing of students provides objective information about such students. This presumption largely rests on an empiricist account of science. In light of mounting criticism, however, empiricist foundationalism has given way to a social epistemology. From this standpoint, empirical data are only objective for those who share assumptions and values that are themselves without foundations. Thus, the major questions to be raised about testing are not in terms of whether test results supply the truth about those who are evaluated, but concern the utility of the tests for the full range of stakeholders. Who gains and loses as a result of testing practices, and in what ways? In the present offering we focus in particular on a range of adverse consequences. We first note the neoliberalist and individualist ideologies carried by current testing practices. We then discuss the impact on societal well-being, including the fostering of social division and distrust, the creation of hierarchies of worth, and the diminution of pluralism. We turn then to the impact of testing on the educational system, including the sacrifice of curriculum and pedagogy for the production of higher test scores, and the diminution of teacher motivation and engagement. Finally, in terms of community, there is a disregard for local needs and values, a loss in student motivation, and an increase in family tensions. We complete the paper with a discussion of possible alternatives to current testing practices, and recommendations for future policies.

Background/Context: The target of assessment validation is not an assessment but the use of an assessment for a purpose. Although the validation literature often provides examples of assessment purposes, comprehensive reviews of these purposes are rare. Additionally, assessment purposes posed for validation are generally described as discrete and static. Purpose/Objective/Research Question/Focus of Study: This paper has two goals. The first is to synthesize recent frameworks that differentiate among purposes of large-scale standardized assessments. The second is to describe the forces that shape the purposes of any particular assessment over time. The author highlights the tendency of assessment purposes toward purpose drift, where purposes evolve and differentiate from purposes originally intended. The author argues that this process is predictable enough that validation practices that ignore drift are incomplete and insufficient. Research Design: This paper is comprised of a synthesis of past validation frameworks and a case-based argument in favor of expanding these frameworks. Conclusions/Recommendations: If predictable forces cause the purposes of an assessment program to expand beyond the purposes originally validated, then conventional validation approaches proposed in the assessment literature are incomplete. Validation efforts should follow known vectors that propagate purpose drift and address the appropriateness of the purposes at the destinations of these vectors. Vectors include the aggregation of scores to support group-level inferences, the disaggregation of student results into scores for diagnostic or formative purposes, and the incentives-driven responses to raising stakes on metrics like "adequate yearly progress" and "teacher value added."

In the attempts to improve teaching and learning through the use of psychometrics and its underlying science, conditions have been set inadvertently that are unlikely to encourage and support learning and self-evaluation. (IAH)

Background/Context: For more than a century, standardized achievement tests have been a feature of American education. Throughout that time, critics of standardized tests have argued that their use has detrimental effects on students, schools, and curriculum. Despite these critiques, the number and uses of standardized tests have increased steadily. Though a great deal of research has focused on the technical design of tests, the history of individual tests, and general critiques of testing, there is little research that helps explain the continued use of standardized tests in American education despite near constant criticism. Purpose/Objective/Research Question/Focus of Study: This article develops a framework for understanding a basic paradox in the history of standardized testing in American education: the durability of standardized testing in the face of persistent criticism. Seeking to address this paradox, the article asks why tests have persisted and proliferated even though students dislike taking tests, educators believe that tests distort the learning process, and experts challenge the validity of test results. Research Design: This article involves a historical analysis of structural and cultural aspects of American education that help explain the particular uses and durability of testing. Conclusions/Recommendations: First, we identify three master critiques of standardized tests: distortion, waste, and misclassification. We find that, despite these persistent critiques, four important contextual features of the American education system help explain the continuous hold that standardized tests have had on American education: that the American education system is decentralized, avowedly meritocratic, publicly funded, and central to aspirations of upward mobility. These contextual factors, along with the historically contingent development of testing expertise, testing culture, and development of testing infrastructure, provide a framework for understanding the persistence of testing. Together, these factors create a dynamic system in which critiques of tests lead not to the elimination of testing, but to its further elaboration and evolution. [ABSTRACT FROM AUTHOR]

Background/Context: The evolution of validity understandings from mid-century to now has emphasized that test validity depends on test purpose--adding consequence considerations to issues of interpretation and evidentiary warrants. Purpose: To consider the tensions created by multiple purposes for assessment and sketch briefly how we got to where we are; furthermore, to address two critically important purposes: the accountability purpose versus the learning purpose for assessment. Research Design: This is an analytic, closing commentary to this special section. Conclusions: When a test is used as an educational reform, the theory of action behind the reform should be made explicit and that theory or series of claims and assumptions is what should be examined in the validity evaluation. As to the prospect of improving the teaching profession by the use of value-added methods, I believe that this is an overly ambitious use of a potentially useful statistical tool. As these systems are being implemented, we can and should conduct validity studies designed to detect plausible shortcomings and side effects as well as intended outcomes. [ABSTRACT FROM AUTHOR]

Background/Context: Recent U.S. policy has brought a shift in assessment use, from measurement tools to policy levers. In particular, testing has become a core part of teacher evaluation policies in many states, with test results becoming akin to a job evaluation. Purpose: To explore the notion of consequential validity m assessment use and examine the implications of using an assessment more as a policy tool than a measurement tool. Research Design: This is an essay concerning how assessment measures are used in educational policy contexts. Conclusions: Meaning is created by use, and the intense push for test-based accountability policies m the U.S. has fundamentally changed the nature of test use. The core meaning of tests and testing has accordingly been qualitatively changed. Attention to consequential validity counsels against use of tests to drive policy unless, and until, the results that process itself have been validated for their furtherance of recognized goals. [ABSTRACT FROM AUTHOR]

Background/Context: Education policymakers across the country face an urgent problem: we know there is wide disparity in teacher effectiveness, but we lack meaningful tools to identify and reward the most effective teachers or to ensure that the least effective improve or leave the classroom. Purpose: This article considers the value of the national move toward value-added measures and our current fascination with objective measurements -- a fascination that stems from our collective distrust of our teachers and ourselves, and our reluctance to make judgments about the substantive narratives we teach students. Research Design- This is an analytical and reflective piece that draws upon the author's experience serving as New York State Commissioner of Education and dean of a teacher education institution in New York City. Conclusions: Value-added measures of teachers' impact on student learning are an imperfect but important tool; however, by often refusing to take responsibility for what is worth teaching, we risk cutting off important opportunities for democratic education and ultimately impoverishing students' own ability to make meaningful judgments about their world, regardless of their test scores. [ABSTRACT FROM AUTHOR]

Background/Context: Validity issues are often discussed in technical terms, but the context changes when measures enter broad public debate, and a wider range of interests come into play. Purpose: This article, part of a special section of TCR, considers the political dimensions of validity questions as raised by a keynote address and panel discussion originally held at Teachers College in March 2012. Research Design: This is an analytical and reflective piece, based on the author's participation in the panel and drawing on his experience and writing on the broad issue of politics and research. Conclusions: Technical expertise in the construction and interpretation of measurements is important in the new world of high-stakes and "evidence-based" education policy, but the political realities, when measures go public, make the exclusive reliance on such expertise problematic. [ABSTRACT FROM AUTHOR]

Background/Context: Education policy over the past 40 years has focused on the importance of accountability in school improvement. Although much of the scholarly discourse around testing and assessment is technical and statistical, understanding of validity by a non-specialist audience is essential as long as test results drive our educational system and accountability. Purpose: To portray the roles and functions of testing in U.S. educational contexts and the challenges faced in upholding validity principles in practice. Research Design: This historical and analytical piece explores the evolution of the concept of validity, its treatment in scholarly discourse, along with challenges faced in practice, and projects future possibilities for educational assessments and their validity. Conclusions: The challenge is to imagine a fantasy (like a chimera) where technology, globalization, changes in demography, societal expectations, and individual preferences could impact new forms of tests without losing evidence of their technical quality, including validity. A brief argument is made to consider the learning of skills rather than an exclusive focus on content as the core business of assessment. [ABSTRACT FROM AUTHOR]

Background: Much is still unknown or unclear about how and where validity issues arise in high stakes testing situations in education, and ways by which we can rectify validity problems in practice and policy contexts. Purpose: This paper is the Foreword to the special issue of the Teachers College Record, When Education Measures Go Public - Stakeholder Perspectives on How and Why Validity Breaks Down. Method: The paper analyzes a recent case involving an application of the SAT to highlight tensions between validity and test score use in high stakes school accountability environments driven by the No Child Left Behind (NCLB) Act of 2001. It uses the case study as a vehicle to introduce the individual papers and authors in the section. Conclusions: There are information and power gaps among those who set societal priori- ties for using tests for high stakes purposes, those who design and conduct psychometric research on tests and testing programs, and those who could eventually face consequences of assessment misuse. These gaps could be addressed through thoughtful exchanges among key assessment stakeholders, as this special issue shows. [ABSTRACT FROM AUTHOR]

Background/Context: Testing accommodations have been widely utilized as a way of increasing the validity of content assessments for English language learner (ELL) students. However, concerns have also arisen regarding the appropriateness of accommodation use, including the accessibility and fairness of accommodations. While many states have developed ELL-specific accommodation policies and guidelines, little research has been available on how the accommodation policies are carried out in practice. Purpose/Objective/Research Question/Focus of Study: The present study investigated two states' accommodation policies, specifically for the states' respective large-scale Grade 8 math assessments, and conducted a case study to examine teachers' understanding of the policies and uses. The study aimed to identify issues to consider for an appropriate use of accommodations and provide useful information for policymakers to improve their accommodation policies. Research Design: The study utilized a qualitative method employing teacher surveys and interviews. The survey and interview instruments were developed based on previous literature and experts' feedback. The interview transcripts were coded by two researchers using a systematic coding scheme. Descriptive statistics of the responses were computed to observe trends across and within districts and schools. The results were also compared between states or between ELL and math teachers when applicable. Conclusions/Recommendations: Despite the limitation of the small sample in this study, the findings of the study offer practical implications for policymakers and educators in the use of accommodations for ELL students. The study found considerable variation reported by teachers with regard to the perception of accommodation decision makers, selection criteria, and the types of accommodations allowed in each state's math assessment. This variation raised serious concerns regarding the adequacy of the accommodation uses and the comparability of accommodated test results across schools. In this paper, we discussed a number of possible reasons for teachers' reported difficulty in keeping up with state policies: (a) lack of clear guidelines in making accommodation decisions and implementing accommodations in a standardized way, (b) lack of or limited opportunities in receiving information and communicating about accommodations among decision makers and teachers, and (c) limited resources and logistical difficulties. To support an appropriate use of accommodations for teachers, we recommend that states make efforts to provide comprehensive, operationalized guidelines for ELL accommodations, monitor the use of the guidelines, and hold regular professional meetings for ELL and content teachers. [ABSTRACT FROM AUTHOR]

Background: The Thinking Styles Inventory--developed by Sternberg and Wagner based on Sternberg's (1988, 1997) earlier theory of mental self-government--was selected for the research in order to assess thinking styles of student teachers. Another reason is that the theoretical constructs, as well as the inventory generated from the theory, have proved to be valuable to assess thinking styles of people in several studies. For example, previous research reported that students differed in their thinking styles depending on their personal characteristics (e.g., Sternberg & Grigorenko, 1995; Zhang, 1999, 2002b, 2001e). However, research found no distinct patterns between thinking styles and personal characteristics of students (e.g., Grigorenko & Sternberg, 1997; Zhang, 2002a). Little research has focused on the study of non-Western students' thinking styles. Thus, it is reasonable to expect that the thinking styles could also be identified among student teachers in Turkey since the styles are differentiated within a different country and culture. Objectives of Study: (1) What is the validity and reliability of the Thinking Styles Inventory (TSI) in assessing thinking styles (TS) among Turkish student teachers? (2) Is the structure of a factor analysis of the TSI consistent with the five dimensions postulated by the theory of mental self-government in a Turkish sample? (3) Are TS of student teachers differentiated based on such socialization variables as gender, age, educational level, type of university attended, and field of study followed? Research Design: The study reported here used quantitative methods with a survey sampling design. Subjects: The research subjects were comprised of 402 student teachers enrolled in English, mathematics, and science teaching programs at Yildiz Technical University in Istanbul, Turkey. Results: The results of factor analysis for construct validity of the inventory addressed 13 subscales under the five dimensional constructs with 104 items. Moreover, the total internal reliability of scale was a .92 reliability coefficient. Findings demonstrated that the 13 subscales had internal consistency reliabilities; Cronbach's alphas of the 13 subscales for these subjects ranged from .61 to .91, as well as item-scale correlation that ranged from .37 to .88. Test re-test reliability for external reliability of subscales was between .63 to .78. When the ANOVA findings of the research is evaluated as a whole, it might be said that the student teachers' particular thinking styles were differentiated by the socialized variables of gender, age, type of university attended, as well as field of study followed. Conclusions/Recommendations: In order to measure thinking styles of students based on a model of broad intellectual styles, the inventory might be used as an efficient instrument. Moreover, the results of this study demonstrated a diversity of thinking styles among the participants. However, further research is needed to clarify the nature of thinking styles as assessed by the inventory at different educational levels and culture to facilitate a better understanding of the thinking styles of students. [ABSTRACT FROM AUTHOR]