Search

Discussions of power and privilege, oppression, and structural inequities in classrooms can produce complex understanding and critical analysis when facilitated effectively. In this article we present the critical conversations model for facilitating conversations that open up space for discussing such issues and encourage the development of critical consciousness and reflection through experiential learning. The model provides flexible scaffolding for instructors to use to navigate their way through classroom tensions and capitalize on spontaneous learning opportunities. Theoretical underpinnings of the model are explained, followed by a description of the model. Implications for social work education and next steps are discussed.

Volume 6 of the Journal of Writing Assessment, our second as co-editors and the third as an online open-access journal, is complete. Looking back at the five articles we published, important themes emerged related to the impact of digital technologies on writing assessment and the connections between rubrics and raters.

This installment of the JWA annotated bibliography focuses on the phenomenon of machine scoring of whole essays composed by students and others. "Machine scoring" is defined as the rating of extended or essay writing by means of automated, computerized technology. Excluded is scoring of paragraph-sized free responses of the sort that occur in academic course examinations. Also excluded is software that checks only grammar, style, and spelling. Included, however, is software that provides other kinds of evaluative or diagnostic feedback along with a holistic score. While some entries in this bibliography describe, validate, and critique the ways computers "read" texts and generate scores and feedback, other sources critically examine how these results are used. The topic is timely, since the use of machine scoring of essays is rapidly growing in standardized testing, sorting of job and college applicants, admission to college, placement into and exit out of writing courses, content tests in academic courses, and value-added study of learning outcomes.

Traditionally, editors write an introduction to each issue of a journal, linking articles, identifying themes, and explaining what unifies the volume. However, because our volumes evolve organically with articles being published online as they are deemed ready through the peer-review process, that traditional approach doesn’t work. Instead, we have opted for an Afterword, an opportunity to look back, to see what links and themes have emerged in the volume.

This essay provides an overview of the research and scholarship on reliability in college writing assessment from the author's perspective as a composition and rhetoric scholar. It argues for reframing reliability by drawing on traditions from fields of college composition and educational measurement with the goal of developing a more productive discussion about reliability as we work toward a unified field of writing assessment. In making this argument, the author uses the concept of framing to argue that writing assessment scholars should develop a shared understanding of reliability. The shared understanding begins with the values—such as accuracy, consistency, fairness, responsibility, and meaningfulness—that we have in common with others, including psychometricians and measurement specialists, instead of focusing on the methods. Traditionally, reliability has been framed by statistical methods and calculations associated with positivist science although psychometric theory has moved beyond this perspective. Over time, the author argues, if we can shift the frame associated with reliability, we can develop methods to support assessments that lead to improvement of teaching and learning.

This annotated bibliography focuses on issues surrounding minorities and writing assessment, including issues associated with various ethnic groups as well as those issues associated with gender studies and with minorities in special education. In addition to reporting on minorities and testing consequences, several selections make recommendations for revising or constructing testing instruments to address some of the special issues minorities face with classroom and large-scale assessments. Several selections also pose important questions that classroom instructors, school administrators, and writing assessment specialists might ask before conducting assessments that involve minority students. Over the past 30 years, studies of SAT, ACT, and other high-stakes tests report that test scores continue to lag behind for certain minority groups. Several selections that we annotate below offer reasons for these differences and fresh insights and analyses into trends for minority test scores.

Because it is difficult, time-consuming, vexing, rewarding, and fundamental to the teaching of writing, teachers' written responses to student writing has been and continues to be a topic of great interest to those in any field where writing is assigned, responded to, and graded. Although assigning and grading writing are closely related to response, the focus for this issue of the bibliography is response. Likewise, as Mathison-Fife and O'Neill remind us, student writers receive valuable feedback and response to their writing in other forms and from other sources--peer feedback, student-teacher conferences, workshops, and so on--yet the responses that teachers write to students on their essays constitutes some of the most important formative feedback and evaluative opportunities teachers have. Not surprisingly, much of the empirical research on response is textual analysis of teachers' comments or revisions students make to their writing after reading teacher feedback. Other research includes case studies of teachers and/or students in the dialogic process of teacher response and subsequent student writing. One of the biggest questions that remains is whether teachers' response styles are consistent with the emerging themes in modern composition studies; in other words, do teachers' responses reflect such important shifts as dialogue, processes, students' rights to their own language, and so on, or is response largely negative and devoted to criticizing and correcting student prose. These and other relevant, pedagogically minded questions continue to fuel the literature on teachers' written responses to student writing.

This article reports on a study of the impact of the form of high-stakes, state-mandated writing tests on high school curricula and teaching practices. Through surveys and focus group interviews of high school English teachers, we addressed two main questions: How do different kinds of high-stakes, statewide assessments impact writing curriculum and instruction in secondary schools? and What are teachers' views of the impact of different kinds of high-stakes tests? We conducted our study in three states--California, Georgia, and Kentucky--each with a different type of writing test at the time of the study: multiple choice, timed impromptu, and portfolio, respectively. The survey results contribute to the growing body of research that indicates the forms of writing tests influence what teachers teach and how they teach it. This influence was complex with significant differences across the three states in the types of assignments, the length of assignments, number of drafts, and the amount of time allowed for assignments. Our results also indicated that the form of high-stakes writing tests also impacts teacher morale and attitudes.

In this, our third installment of the bibliography on assessment, we survey the second half of the literature on reliability and validity. The works we annotate focus primarily on the theoretical and technical definitions of reliability and validity--and in particular, on the relationship between the two concepts. We summarize psychometric scholarship that explains, defines, and theorizes reliability and validity in general and within the context of writing assessment. Later installments of the bibliography will focus on specific sorts of assessment practices and occasions, such as portfolios, placement assessments, and program assessment--all practices for which successful implementation depends on an understanding of reliability and validity.

This article re-examines the research into placement that William L. Smith did at the University of Pittsburgh during the 1980s and 1990s by situating Smith's work within the larger context of educational measurement theories, placement testing, and holistic scoring. I present the series of research studies that Smith conducted into Pitt's placement test as a case study in validation inquiry, arguing that his approach serves as a model for those who direct writing assessments. The implications of Smith's research reach beyond placement into first year composition: by approaching local writing assessment needs as Smith did, writing assessment professionals not only can create more effective assessments, but they also can contribute significantly to assessment theory.

Welcome to the first installment of an annotated bibliography on writing assessment. Over the next several issues, we will be publishing different sections of the bibliography such as Theory, Classroom Response, and Portfolios. Our intent is to provide a resource for all those who work in writing assessment (K-college) or are looking for writing assessment scholarship. In categorizing and sectioning the scholarship, we have used theories and practices of writing assessment instead of educational level because we see a need for more crossover, especially between K-12 and college, in research, theory, and practice.