Search

Automated scoring of student language is a complex task that requires systems to emulate complex and multi-faceted human evaluation criteria. Summary scoring brings an additional layer of complexity to automated scoring because it involves two texts of differing lengths that must be compared. In this study, we present our approach to automate summary scoring by evaluating a corpus of approximately 5,000 summaries based on 103 source texts, each summary being scored on a 4-point Likert scale for seven different evaluation criteria. We train and evaluate a series of Machine Learning models that use a combination of independent textual complexity indices from the ReaderBench framework and Deep Learning models based on the Transformer architecture in a multitask setup to predict concurrently all criteria. Our models achieve significantly lower errors than previous work using a similar dataset, with MAE ranging from 0.10-0.16 and corresponding R[superscript 2] values of up to 0.64. Our findings indicate that Longformer-based models are adequate for contextualizing longer text sequences and effectively scoring summaries according to a variety of human-defined evaluation criteria using a single Neural Network. [This paper was published in: "AIED 2022, LNCS 13355," edited by M. M. Rodrigo et al., Springer Nature Switzerland, 2022, pp. 756-761.]

Writing is essential for success in academics and everyday tasks, but the development of writing skills depends on consistent access to high-quality instruction, extended practice, and personalized feedback. To address these demands and meet students' needs, educators and researchers have turned to technology-based writing tools. Ideally, these tools integrate the core components of intelligent tutoring, including a domain model, student model, tutor model, and interface model to engage students with individualized feedback that is linked to adaptive writing instruction. However, the landscape of writing tools still has much room for improvement in terms of incorporating advanced artificial intelligence-enabled features to better approximate intelligent tutoring systems (ITSs). This chapter describes the key elements of ITS technologies and how they can be integrated to further develop ITS tools for writing. To this end, this chapter: (1) summarizes evidence-based aspects of successful ITSs and how they might be integrated into computer-based tools for writing; (2) reviews how existing systems have leveraged intelligent tutoring approaches; and (3) articulates how future technology-based writing tools could implement advanced intelligent tutoring features to better meet students' needs. The chapter concludes with the implications and future directions of intelligent tutoring for the teaching and learning of writing. [This chapter was published in: "Digital Writing Technologies in Higher Education," edited by O. Kruse et al., Springer, Cham, 2023, pp. 365-383.]

The objective in the current paper is to examine the processes of how our research team negotiated meaning using an iterative design approach as we established, developed, and refined a rubric to capture comprehension processes and strategies evident in students' verbal protocols. The overarching project comprises multiple data sets, multiple scientists across (distant) institutions, and multiple teams of discourse analysts who are tasked with scoring over 20,000 verbal protocols (i.e., think aloud, self-explanation) collected in studies conducted in the last decade. Here, we describe the iterative modifications, negotiations, and realizations while coding our first subset comprising 7,559 individual verbal protocols. Drawing upon work in design research, we describe a process through which the research team has negotiated meaning around theory-driven codes and how this work has influenced our own ways of conceptualizing comprehension research, theory, and practice. [This paper was published in: "ICLS 2021 Proceedings," ISLS, 2021, pp. 139-146.]

Research in discourse processing has provided us with a strong foundation for understanding the characteristics of text and discourse, as well as their influence on our processing and representation of texts. However, recent advances in computational techniques have allowed researchers to examine discourse processes in new ways. The purpose of the current paper is to build on prior work in this domain and describe how new methodologies that consider the multi-dimensional nature of texts can serve as a complement to the existing literature. We focus on natural language processing (NLP) methodologies, in which computers calculate information about the linguistic and semantic properties of language data. We first provide a context for the origins of computational discourse analysis through the integration of research across computer science and psychology. We then provide an overview of different NLP methodologies and describe prior work that has leveraged these techniques to advance theoretical perspectives of discourse comprehension and production. Finally, we propose new areas of research that integrate these advances with traditional research methodologies in the field. [This is the online first version of an article published in "Language and Linguistics Compass."]

Assessments of natural language can provide vast information about individuals' thoughts and cognitive process, but they often rely on time-intensive human scoring, deterring researchers from collecting these sources of data. Natural language processing (NLP) gives researchers the opportunity to implement automated textual analyses across a variety of psychological domains. NLP techniques can be used to score written language and provide instantaneous feedback. For example, the Reading Strategy Assessment Tool (RSAT) can score short constructed responses produced while students read texts for presence of various comprehension processes. Another application of NLP is automated essay scoring (AES), which has been used in high-stakes tests. AES systems are developed using machine learning techniques: a corpus of essays is scored by human raters and split into a training and test set. A model is trained to predict human ratings of essays using linguistic features derived using NLP. AES is now often as accurate as expert human raters. While AES systems simply provide scores, automated writing evaluation (AWE) systems also provide feedback on student writing. A continued challenge in both AES and AWE systems is ensuring accurate scoring across a variety of genres, prompts, topics, modes of writing, and writer characteristics. Novel approaches suggest that NLP techniques can be used not only for assessment, but also to better understand how natural language can reveal individual characteristics. For instance, the cohesion of self-explanations can be used to predict the coherence of a reader's mental representation of a text. NLP techniques have also been used to examine emotions and other psychological states from written responses, using tools such as the Linguistic Inquiry and Word Count (LIWC) and WordNet-Affect. Additionally, NLP has been used to increase student learning through implementation in a variety of intelligent tutoring systems (ITSs). When users provide open-ended responses to an ITS, NLP techniques are used to score the responses and provide automated feedback. AutoTutor is an ITS that provides instruction on technical topics such as computer literacy. After being asked a question, students engage in a dialogue of approximately 20 to 100 conversational turns with AutoTutor. The tutor uses NLP to determine how closely a student's response matches the expected answer; if the match is not sufficiently close, the tutor may pump the student for information, provide hints and prompts, or provide the answer, depending on the student's response. Another example is the Interactive Strategy Trainer for Active Reading and Thinking (iSTART), an ITS that helps high school and adult students learn and practice reading comprehension strategies. Students write self-explanations of texts as they read, and their self-explanations are assessed using a series of NLP algorithms. A pedagogical agent provides feedback based on various features of the student's response. These various applications of NLP indicate its great potential for use in learning, assessment, and exploring the relations between language and psychological processes. [This chapter was published in: "APA Handbook of Research Methods in Psychology, Second Edition: Vol. 1. Foundations, Planning, Measures, and Psychometrics," edited by H. Cooper et al., American Psychological Association, 2023, pp. 361-380.]

Writing is a crucial means of communicating with others and thus vital to success and survival in modern society. This article provides an overview of recent research and key findings about writing, including the roles of cognitive and social processes during writing, and educational research on how to improve writing proficiency. Writing processes rely on virtually all aspects of cognition (e.g., working memory, motivation, affect, self-regulation, prior knowledge, problem solving) and are naturally embedded in social contexts. Social factors include writers' objectives, audience, genre, and mode of writing. For example, the increased use of the Internet has rendered writing for informal purposes more frequent and writing mechanics (e.g., deleting, spell checking) and search for information more efficient. Research on educational interventions to improve writing points to the importance of providing students with instruction and practice using writing strategies, writing practice with feedback (e.g., instructor, automated), and collaborative writing (including peer feedback). The authors recommend that more time be devoted to writing instruction, the need for technology to support teachers and students, and a need for increased funding to support teachers in their efforts to providing writing instruction. Given the inherent complexity of writing, it is important to help students learn how to write across various situations and demands. Therefore, it is vital that students are taught how to write across various situations with varying purposes and demands. This necessitates reading many types of text genres (e.g., narrative vs. informational writing), writing frequently, and revising based on feedback. Over the past two decades, there has been a substantial increase in research on writing processes, including methods to improve writing. However, there remains a substantial need for additional experimental work to understand writing processes as well as more evidence on which types of interventions are most beneficial in helping students to improve their writing. Feedback from both "cognitive" and "sociocultural" researchers should inform future revisions of the standardized guidelines and assessments with the long-term goal of developing a clearly defined set of standards for academic excellence in writing.

With increased availability of information in modern societies, individuals are often faced with complex decisions regarding how to integrate and judge the veracity of available information. Generally, these issues have been approached using computational techniques to "detect" and "reduce" the spread of information across social media. However, few researchers have examined theoretically-grounded characteristics of misinformation. The purpose of this chapter is to examine this phenomenon through the lens of discourse processing theories, which emphasize interactions among features of the discourse, the reader, and the context. We describe a proof of concept on how dynamical systems modeling combined with computational linguistics has strong potential to reveal underlying characteristics of the spread of misinformation. Additionally, we discuss potential directions for future research, as well as implications for interventions to help students accurately process information in the modern digital age. We call for research using a combination of computational linguistics, telemetry, and dynamical systems analytics in order to better understand the temporal organization of text and the spread of misinformation. [This paper was a chapter in "Misinformation and Fake News in Education" (p159-185). 2019.]

The assessment of argumentative writing generally includes analyses of the specific linguistic and rhetorical features contained in the individual essays produced by students. However, researchers have recently proposed that an individual's ability to flexibly adapt the linguistic properties of their writing may more accurately capture their proficiency. However, the features of the task, learner, and educational context that influence this flexibility remain largely unknown. The current study extends this research by examining relations between linguistic flexibility, reading comprehension ability, and feedback in the context of an automated writing evaluation system. Students (n = 131) wrote and revised six argumentative essays in an automated writing evaluation system and were provided both summative and formative feedback on their writing. Additionally, half of the students had access to a spelling and grammar checker that provided lower-level feedback during the writing period. The results provide evidence for the supposition that skilled writers demonstrate linguistic flexibility across the argumentative essays that they produce. However, analyses also indicate that lower-level feedback (i.e., spelling and grammar feedback) have little to no impact on the properties of students' essays nor on their variability across prompts or drafts. Overall, the current study provides important insights into the role of flexibility in argumentative writing skill and develops a strong foundation on which to conduct future research and educational interventions. [This is the online first version of an article published in "Reading and Writing: An Interdisciplinary Journal."]

Background knowledge is a strong predictor of reading comprehension; yet little is known about how different types of background knowledge affect comprehension. The study investigated the impacts of both domain and topic-specific background knowledge on students' ability to comprehend and learn from science texts. High school students (n = 3650) completed two background knowledge assessments, a pretest, comprehension tasks, and a posttest, in the context of the Global, Integrated, Scenario-based Assessment (GISA) on Ecosystems. Linear mixed effects models revealed positive effects of background knowledge on comprehension and learning as well as an interactive effect of domain and topic-specific knowledge, such that readers with high domain knowledge, but low topic-specific knowledge improved most from pretest to posttest. We discuss the potential implications of these findings for educational assessments and interventions. [This paper was published in "Discourse Processes" v55 n5-6 p510-524 2018 (EJ1184439).]

The assessment of writing proficiency generally includes analyses of the specific linguistic and rhetorical features contained in the singular essays produced by students. However, researchers have recently proposed that an individual's ability to flexibly adapt the linguistic properties of their writing might more closely capture writing skill. However, the features of the task, learner, and educational context that influence this flexibility remain largely unknown. The current study extends this research by examining relations between linguistic flexibility, reading comprehension ability, and feedback in the context of an automated writing evaluation system. Students (n = 131) wrote and revised six essays in an automated writing evaluation system and were provided both summative and formative feedback on their writing. Additionally, half of the students had access to a spelling and grammar checker that provided lower-level feedback during the writing period. The results provide evidence for the fact that developing writers demonstrate linguistic flexibility across the essays that they produce. However, analyses also indicate that lower-level feedback (i.e., spelling and grammar feedback) have little to no impact on the properties of students' essays nor on their variability across prompts or drafts. Overall, the current study provides important insights into the role of flexibility in writing skill and develops a strong foundation on which to conduct future research and educational interventions. [This paper was published in: "LAK18," March 7-9, 2018, Sydney, NSW, Australia, ACM, 2018, pp. 380-388.]

As educators turn to technology to supplement classroom instruction, the integration of natural language processing (NLP) into educational technologies is vital for increasing student success. NLP involves the use of computers to analyze and respond to human language, including students' responses to a variety of assignments and tasks. While NLP is widely used to deliver students with formative feedback, it can also be used to provide educators with information about task difficulty, students' individual differences, and student performance. In this chapter, we will first provide an overview of NLP, followed by a discussion of how NLP could be used to examine the learning process across a number of time points. Finally, we consider the future applications of NLP in the learning analytics domain. [This chapter is in: "Handbook of Learning Analytics--Second edition, pp. 46-53, Society for Learning Analytics Research, Vancouver, BC.]

Automated writing evaluation (AWE) tools can facilitate teachers' analysis of and feedback on students' writing. However, increasing evidence indicates that writing instructors experience challenges in implementing AWE tools successfully. For this reason, our development of the Writing Analytics Tool (WAT) has employed a participatory approach that includes teachers throughout all stages of design. WAT aims to generate writing analytics that are both useful to teachers and actionable by students. In this report, we discuss our participatory design approach and resulting insights, including (a) teachers' expressed instructional needs and goals, and (b) design solutions that were conceptualized to address both factors. [This paper was published in: "Companion Proceedings 12th International Conference on Learning Analytics & Knowledge (LAK22)," 2022.]

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students' comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students' reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students' reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students' reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students' literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students. [This is the online version of an article published in "International Journal of Human-Computer Interaction."]

This study evaluates high school students' perceptions of automated writing feedback, and the influence of these perceptions on revising, as a function of varying modes of computer-based writing instruction. Findings indicate that students' perceptions of automated feedback accuracy, ease of use, relevance, and understandability were favorable. Immediate perceptions of feedback received on a selected essay were minimally related to how and whether students revised their essays. However, attitudes formed over multiple sessions were significantly related to revising. More importantly, the mode of instruction appeared to influence how feedback perceptions shaped revising behaviors. Students who engaged in traditional writing-based training and practice seemed to focus on their own perceived writing abilities when deciding how to revise. In contrast, students who also received strategy instruction and game-based practice attended more carefully to the perceived quality of the automated feedback. [This paper was published in: "Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting," Human Factors and Ergonomics Society, 2018, pp.2089-2093.]

This chapter explores three broad principles of user-centered design methodologies: participatory design, iteration, and usability considerations. The authors highlight the importance of considering teachers as a prominent type of ITS end user, by describing the barriers teachers face as users and their role in educational technology design. To exemplify these points, they draw upon their experiences in developing the Writing Pal--an ITS for writing strategy instruction. The chapter concludes by offering a design approach--the Design Implementation Framework (DIF)--that builds upon existing cyclical design methods but is tailored to the design of ITSs and other educational technologies. [This article was published in: R. D. Roscoe, S. D. Craig, & I. Douglas (Eds). "End-user considerations in educational technology design" (pp. 76-98). Hershey, PA: IGI Global.]

A commonly held belief among educators, researchers, and students is that high-quality texts are easier to read than low-quality texts, as they contain more engaging narrative and story-like elements. Interestingly, these assumptions have typically failed to be supported by the literature on writing. Previous research suggests that higher quality writing is typically associated with "decreased" levels of text narrativity and readability. In this study, the authors present the hypothesis that writing proficiency is associated with an individual's "flexible" use of linguistic properties, rather than simply the consistent use of a particular set of linguistic properties. To test this hypothesis, the authors leveraged both natural language processing and dynamic methodologies to capture variability in students' use of narrative style across multiple essay prompts. Forty-five high school students wrote 16 essays across 8 laboratory sessions. Natural language processing techniques were first used to calculate the narrativity of each essay. Random walk and Euclidian distance measures were then used to visualize and classify students' flexibility in narrativity across essays. The results support the hypotheses that students who were flexible in their use of narrativity also wrote essays that were rated as having higher quality, whereas inflexible writers tended to write lower quality essays. Additionally, more flexible writers performed higher than the more inflexible writers on general assessments of literacy and prior knowledge. These results are important for researchers and educators, as they indicate that the link between textual properties and writing quality may fluctuate according to the context of a given writing assignment. [This is the online first version of an article published in "Journal of Educational Psychology" (ISSN 0022-0663]. For the final published version of this article, see?EJ1115484.]

The development of strong writing skills is a critical (and somewhat obvious) goal within the classroom. Individuals across the world are now expected to reach a high level of writing proficiency to achieve success in both academic settings and the workplace (Geiser & Studley, 2001; Powell, 2009; Sharp, 2007). Unfortunately, strong writing skills are difficult to develop, as they require individuals to coordinate a number of cognitive skills and knowledge sources through the process of setting goals, solving problems, and strategically managing their memory resources (Flower & Hayes, 1980; Hayes, 1996). Given the difficulty of this process, students frequently underachieve on national assessments of writing proficiency (National Assessment of Educational Progress, 2007, 2011). The successful acquisition of this complex set of skills largely depends on the instruction and feedback provided to students as they develop. Previous research suggests that writing proficiency is best enhanced through strategy instruction (Graham & Perin, 2007), along with extended practice and individualized feedback (Kellogg & Raulerson, 2007). Importantly, curricula aimed to meet these goals can be extremely difficult to implement. This method of writing instruction demands a significant amount of time from teachers, ranging from the time required to prepare materials and instructional content to the process of reading, editing, and providing individualized feedback on students' essays. Unfortunately, teachers rarely have enough time to devote to this process, as they are faced with increasingly large class sizes and, as a consequence, have reduced time for classroom instruction and planning (The National Commission on Writing, 2004). In response to these issues surrounding effective writing pedagogy, there has been an increased effort to develop computer-based systems for writing assessment and instruction (Shermis & Burstein, 2003, 2013). These systems vary widely in their primary purposes, from the automated scoring of student essays to the provision of formative feedback or the explicit instruction of writing knowledge and strategies (Dikli, 2006; Graesser & McNamara, 2012; Roscoe, Allen, Weston, Crossley, & McNamara, 2014; Weigle, 2013; Xi, 2010). Despite the importance of writing strategy instruction and feedback, the majority of research conducted on this topic has focused on the development of computer-based systems that can provide reliable and valid scores to students' essays. However, more recently, researchers have placed a stronger emphasis on the development of computer-based systems that incorporate more instructional materials, such as formative feedback and explicit instruction on the writing process (Roscoe et al., 2011). In this chapter, we provide an overview of computer-based tools and techniques that are currently being used to support writing instruction and practice. Additionally, we discuss more recent advances in this field with suggestions for future research and development. [This chapter was published in: "Handbook for Writing Research 2016," p316-329.]

The relationship between working memory capacity and writing ability was examined via a linguistic analysis of student essays. Undergraduate students (n = 108) wrote timed, prompt-based essays and completed a battery of cognitive assessments. The surface- and discourse-level linguistic features of students' essays were then analyzed using natural language processing tools. The results indicated that WM capacity was related to surface-level, but not discourse-level features of student essays. Additionally, the results suggest that these relationships were attenuated for students with high inferencing skills, as opposed to those with lower inferencing skills. [This paper was published in the Proceedings of the 38th Annual Meeting of the Cognitive Science Society 2016, p2483-2488.]

Source-based essays are evaluated both on the quality of the writing and the content appropriate interpretation and use of source material. Hence, composing a high-quality source-based essay (an essay written based on source material) relies on skills related to both reading (the sources) and writing (the essay) skills. As such, source-based writing must involve language comprehension and production processes. The purpose of the current study is to examine the impact of reading, writing, and blended (i.e., reading and writing) strategy training on students' performance on a content-specific, source-based essay writing task. In contrast to general source-based writing tasks, content-specific source-based writing tasks are tasks wherein writers are provided the source material on which to base their essays. Undergraduate students (n = 175) were provided with strategy instruction and practice in the context of two intelligent tutoring systems, "Writing Pal" and "Interactive Strategy Training for Active Reading and Thinking" (iSTART). Results indicated that participants in the blended strategy training condition produced higher quality source-based essays than participants in the reading comprehension-only, writing-only, or control condition, with no differences observed between the latter three conditions. Further, the benefits of this blended strategy instruction remained significant regardless of prior reading and writing skills, or time on task. [This article was published in "International Journal of Artificial Intelligence in Education" p1-32 2016.]