Search

The Writing Mentor TM (WM) application is a Google Docs add-on designed to help students improve their writing in a principled manner and to promote their writing success in postsecondary settings. WM provides automated writing evaluation (AWE) feedback using natural language processing (NLP) methods and linguistic resources. AWE features in WM have been informed by research about postsecondary student writers often classified as developmental (Burstein et al., 2016b), and these features address a breadth of writing sub-constructs (including use of sources, claims, and evidence; topic development; coherence; and knowledge of English conventions). Through an optional entry survey, WM collects self-efficacy data about writing and English language status from users. Tool perceptions are collected from users through an optional exit survey. Informed by language arts models consistent with the Common Core State Standards Initiative and valued by the writing studies community, WM takes initial steps to integrate the reading and writing process by offering a range of textual features, including vocabulary support, intended to help users to understand unfamiliar vocabulary in coursework reading texts. This paper describes WM and provides discussion of descriptive evaluations from an Amazon Mechanical Turk (AMT) usability task situated in WM and from users-in-the-wild data. The paper concludes with a framework for developing writing feedback and analytics technology.

This study reports on validation of a writing motivation survey and its relationship with a variety of indicators of academic performance of 566 undergraduate students drawn from six US postsecondary institutions. A writing motivation survey was used to capture students’ writing goals, confidence, beliefs, and affect. Two research questions are addressed in the study: 1) What is the internal factor structure of the writing motivation survey completed by a broad range of college students? (2) Using this factor structure, how are the subconstructs of writing motivation associated with the outcomes variables of student performance and features of student writing? Our results confirmed a unidimensional structure for confidence and affect, a 3-factor structure for goals, and a 2-factor structure for beliefs. Low level but significant correlations were also identified between motivation and outcome performance measures, as well as between motivation and features of student writing. The study findings yield insights about the relationship between students’ writing motivation, college success indicators, and writing domain knowledge. These findings suggest directions for pedagogical interventions targeting both motivation and writing feature use.

Genre serves as a useful lens to investigate the range of evidence derived from automated writing evaluation (AWE). To support construct-relevant systems used for writing instruction and assessment, two investigations were conducted that focused on post-secondary writing requirements and faculty perceptions of student writing proficiency. Survey research is described from a national study and a second site study at American University, a 4-year private university in Washington, DC, to illustrate writing requirements and perceptions of writing proficiency in school and workplace settings. A mixed-methods analysis of faculty focus groups in the site study afforded more detailed discussions that were used to understand student writing support needs. Through the lens of genre, study results illustrated how the role of AWE might be expanded and aligned with instruction in four-year post-secondary institutions.

This validation study examines the WPA Outcomes Statement for First-Year Composition, a United States consensus statement for first-year post-secondary writing, as implemented in a unified instructional and assessment environment for first-year college students across three different institution types. Adapting categories of contemporary validation from Kane (2013) , we focus on four forms of evidence gathered from early and late-semester student performance (n = 153): scoring, generalization, extrapolation, and implication. With an emphasis on education policies in action, the study generates important questions that, in turn, provide a basic framework for further research into the challenges of aligning broad consensus statements with locally developed educational initiatives.

When Assessing Writing (ASW) was founded 25 years ago, conversations about fairness were very much in the air and illustrated sharp divides between teachers and educational measurement researchers. For teachers, fairness was typically associated with consistency and access. For educational measurement researchers, fairness was a technical issue: an assessment that did not identify the presence of β (the bias factor) was fair. Since its founding, ASW has continued to be a space where evolving discussions about fairness play out. In this article, we examine a selection of 73 ASW research studies published from 1994 to 2018 that use fairness as a category of evidence. In tracing the use of fairness and related terms across these research articles, our goal is to understand how the conversation about fairness has changed in the last quarter century. Following a literature review that situates fairness within generational, standards-based, and evidential scholarship, we analyze five trends in the journal: fairness as the elimination of bias; fairness as the pursuit of validity; fairness as acknowledgement of social context; fairness as legal responsibility; and fairness as ethical obligation. A tidy narrative that theoretical conceptualization of fairness has deepened over the ASW lifespan is not born out by our findings. Instead, evidence suggests that the disparate stances and methodological challenges that informed early research on fairness remain. As well, the textual record suggests that we have not developed or shared taxonomies for systematically investigating questions of fairness. In our desire to make the research we present actionable, we close by calling attention to the need for theorization of fairness, the advantages nuanced of research methods, and the benefits of non-Western perspectives.

Research problem: A construct mediated in digital environments, information communication technology (ICT) literacy is operationally defined as the ability of individuals to participate effectively in transactions that invoke illocutionary action. This study investigates ICT literacy through a simulation designed to capture that construct, to deploy the construct model to measure participant improvement of ICT literacy under experimental conditions, and to estimate the potential for expanded model development. Research questions: How might a multidisciplinary literature review inform a model for ICT literacy? How might a simulation be designed that enables sufficient construct representation for modeling? How might prepost testing simulation be designed to investigate the potential for improved command of ICT literacy? How might a regression model account for variance within the model by the addition of affective elements to a cognitive model? Literature review: Existing conceptualizations of the ICT communication environment demonstrate the need for a new communication model that is sensitive to short text messaging demands in crisis communication settings. As a result of this prefect storm of limits requiring the communicator to rely on critical thinking, awareness of context, and information integration, we designed a cognitive-affective model informed by genre theory to capture the ICT construct: A sociocognitive ability that, at its most effective, facilitates illocutionary action-to confirm and warn, to advise and ask, and to thank and request-for specific audiences of emergency responders. Methodology: A prepost design with practitioner subjects (N=50) allowed investigation of performance improvement on tasks demanding illocutionary action after training on tasks of high, moderate, and low demand. Through a model based on the independent variables character count, wordcount, and decreased time on task (X) as related to the dependent variable of an overall episode score (Y), we were able to examine the internal construct strength with and without the addition of affective independent variables. Results and discussion: Of the three prepost models used to study the impact of training, participants demonstrated statistically significant improvement on episodes of high demand on all cognitive model variables. The addition of affective variables, such as attitudes toward text messaging, allowed increased model strength on tasks of high and moderate complexity. These findings suggest that an empirical basis for the construct of ICT literacy is possible and that, under simulation conditions, practitioner improvement may be demonstrated. Practically, it appears that it is possible to train emergency responders to improve their command of ICT literacy so that those most in need of humanitarian response during a crisis may receive it. Future research focusing on communication in digital environments will undoubtedly extend these finding in terms of construct validation and deployment in crisis settings.

This study investigated the use of automated essay scoring (AES) to identify at-risk students enrolled in a first-year university writing course. An application of AES, the Criterion ® Online Writing Evaluation Service was evaluated through a methodology focusing on construct modelling, response processes, disaggregation, extrapolation, generalization, and consequence. Based on the results of our two-year study with students ( N = 1,482) at a public technological research university in the United States, we found that Criterion offered a defined writing construct congruent with established models, achieved acceptance among students and instructors, showed no statistically significant differences between ethnicity groups of sufficient sample size, correlated at acceptable levels with other writing measures, performed in a stable fashion, and enabled instructors to identify at-risk students to increase their course success.