Search

Writing researchers have suggested that students who are perceived as strong writers (i.e., those who generate texts rated as high quality) demonstrate flexibility in their writing style. While anecdotally this has been a commonly held belief among researchers and educators, there is little empirical research to support this claim. This study investigates this hypothesis by examining how students vary in their use of two linguistic features (i.e., narrativity and cohesion) across 16 prompt-based essays. Forty-five high school students wrote 16 essays across 8 sessions within an Automated Writing Evaluation (AWE) system. Natural language processing (NLP) techniques and Entropy analyses were used to calculate how rigid or flexible students were in their use of narrative and cohesive linguistic features over time and how this trait related to individual differences in literacy abilities (i.e., vocabulary knowledge and comprehension ability), prior world knowledge, and essay quality. For instance, through the unique combination of NLP and Entropy, we found that patterns of narrative flexibility (or rigidity) were related, significantly and reliably, to students' prior reading comprehension ability after 2 sessions (4 essays). Similarly, students' flexible (or rigid) use of cohesive features was reliably related to their prior reading comprehension ability after 5 sessions (10 essays). These exploratory methodologies are important for researchers and educators, as they indicate that writing flexibility is indeed a trait of strong writers and can be detected rather quickly using the combination of textual features and dynamic analyses. [There is a page numbering error on the PDF. The pages of the PDF are numbered 40-46, 38-54.]

The Writing Pal is an intelligent tutoring system designed to support writing proficiency and strategy acquisition for adolescent writers. A fundamental aspect of the instructional model is automated formative feedback that provides concrete information and strategies oriented toward student improvement. In this paper, the authors explore computational methods for expanding automated feedback by taking into account students' essay revising patterns. High school students (n = 87) used the Writing Pal to write and revise a persuasive essay each day across eight daily sessions. Analyses of the linguistic properties of original and revised essays revealed that students were more likely to implement document-level revisions focused on improving elaboration, organization, and cohesion, rather than surface word-level edits focused on incorporating bigger words or less common words. Implications and applications of essay revising data for automated feedback and essay scoring are discussed.

Game-based environments frequently afford students the opportunity to exert agency over their learning paths by making various choices within the environment. The combination of log data from these systems and dynamic methodologies may serve as a stealth means to assess how students behave (i.e., deterministic or random) within these learning environments. The current work captures variations in students' behavior patterns by employing two dynamic analyses to classify students' sequences of choices within an adaptive learning environment. Random Walk analyses and Hurst exponents were used to classify students' interaction patterns as random or deterministic. Forty high school students interacted with the game-based system, iSTART-ME, for 11-sessions (pretest, 8 training sessions, posttest, and a delayed retention test). Analyses revealed that students who interacted in a more deterministic manner also generated higher quality self-explanations during training sessions. The results point toward the potential for dynamic analyses such as random walk analyses and Hurst exponents to provide stealth assessments of students' learning behaviors while engaged within a game-based environment. [At time of submission to ERIC this article was in press with "International Journal of Artificial Intelligence in Education."]

Work in cognitive and educational psychology examines a variety of phenomena related to the learning and retrieval of information. Indeed, Alice Healy, our honoree, and her colleagues have conducted a large body of groundbreaking research on this topic. In this article we discuss how 3 learning principles (the generation effect, deliberate practice and feedback, and antidotes to disengagement) discussed in Healy, Schneider, and Bourne (2012) have influenced the design of 2 intelligent tutoring systems that attempt to incorporate principles of skill and knowledge acquisition. Specifically, this article describes iSTART-2 and the Writing Pal, which provide students with instruction and practice using comprehension and writing strategies. iSTART-2 provides students with training to use effective comprehension strategies while self-explaining complex text. The Writing Pal provides students with instruction and practice to use basic writing strategies when writing persuasive essays. Underlying these systems are the assumptions that students should be provided with initial instruction that breaks down the tasks into component skills and that deliberate practice should include active generation with meaningful feedback, all while remaining engaging. The implementation of these assumptions is complicated by the ill-defined natures of comprehension and writing and supported by the use of various natural language processing techniques. We argue that there is value in attempting to integrate empirically supported learning principles into educational activities, even when there is imperfect alignment between them. Examples from the design of iSTART-2 and Writing Pal guide this argument.

This study evaluates the use of a hierarchical classification approach to automated assessment of essays. Automated essay scoring (AES) generally relies onmachine learning techniques that compute essay scores using a set of text variables. Unlike previous studies that rely on regression models, this study computes essay scores using a hierarchical approach, analogous to an incremental algorithm for hierarchical classification. The corpus in this study consists of 1243 argumentative (persuasive) essays written on 14 different prompts, across 3 different grade levels (9th grade, 11th grade, college freshman), and four different time limits for writing or temporal conditions (untimed essays and essays written in 10, 15, and 25 minute increments). The features included in the analysis are computed using the automated tools, Coh-Metrix, the Writing Assessment Tool (WAT), and Linguistic Inquiry and Word Count (LIWC). Overall, the models developed to score all the essays in the data set report 55% exact accuracy and 92% adjacent accuracy between the predicted essay scores and the human scores. The results indicate that this is a promising approach to AES that could provide more specific feedback to writers and may be relevant to other natural language computations, such as the scoring of short answers in comprehension or knowledge assessments.

When students exhibit control and employ a strategic plan of action over a situation they are said to be demonstrating agency (Bandura, 2001). The current work is comprised of two studies designed to investigate how agency manifests within students' choice patterns and ultimately influences self-explanation quality within the game-based system iSTART-2. In Study 1, 75 college students interacted freely within iSTART-2 for 2 h. Random walk and Entropy analyses were used to quantify the amount of control demonstrated in students' choice patterns, as well as to determine the relation between variations in these patterns and self-explanation performance within iSTART-2. Overall, students who demonstrated more controlled choice patterns generated higher quality self-explanations compared to students who exhibited more disordered choice patterns. This link between performance and controlled choice patterns is hypothesized to be driven, in part, by students' experiences of agency. That is, engaging in controlled patterns should be advantageous only when doing so is a result of students' strategic planning. In Study 2, this hypothesis was tested by assigning 70 students to a choice pattern (i.e., controlled or disordered) that had been yoked to students from Study 1, thus removing students' ability to exert agency over the iSTART-2 system. Results revealed no differences in self-explanation quality between the groups assigned to controlled and disordered choice patterns. Collectively, findings from these studies support the notion that success within game-based systems is related to students' ability to exert agency over their learning paths.

The Writing Pal (W-Pal) is an intelligent tutoring system (ITS) designed to provide students with explicit writing strategy instruction and practice. W-Pal includes a suite of educational games developed to increase writing engagement and provide opportunities to practice writing strategies. In this study, first (L1) (n = 26) and second (L2) language (n = 16) students interacted with W-Pal over eight sessions. We collected students' daily self-reports of engagement, motivation, and perceptions of performance, as well as their reported game attitudes (difficulty, helpfulness for learning, and enjoyment). Results indicated that, for all students, interactions with W-Pal led to increases in writing performance and more positive attitudes towards the system (engagement, motivation, and perceived performance). For L1 students, game difficulty was a significant predictor of boredom; however, for the L2 students, game enjoyment predicted both their motivation and perceived writing improvement. Notably, the L2 students' game ratings accounted for more variance in these daily reports than did the ratings of L1 students. This study suggests that L1 and L2 students experience similar benefits offered by game-based strategy practice in an ITS. Further, the link between game attitudes and overall daily perceptions of training may be stronger for L2 students than L1 students.

Language entails many nested time scales, ranging from therelatively slow scale of cultural evolution to the rapid scale ofindividual cognition. The nested, multiscale nature oflanguage implies that even simple acts of text production,such as typing a sentence, entail complex interactionsinvolving multiple concurrent processes. As such, textproduction may have much in common with other cognitivephenomena thought to emerge from multiplicativeinteractions across temporal scales, namely those that exhibitfractal properties. We investigated the relationship betweenfractal scaling and the quality of produced text. Participants(N=131) wrote essays while their keystrokes were recorded.Fractal analyses were then performed on time series ofinterkeystroke intervals (IKIs). Results showed that fractalproperties characterizing IKIs positively predicted expertratings of essay quality, even after accounting for essaylength. The results support our hypotheses concerningmultiscale coordination and text production.

This study investigates the potential for linguistic microfeatures related to length, complexity, cohesion, relevance, topic, and rhetorical style to predict L2 writing proficiency. Computational indices were calculated by two automated text analysis tools (Coh- Metrix and the Writing Assessment Tool) and used to predict human essay ratings in a corpus of 480 independent essays written for the TOEFL. A stepwise regression analysis indicated that six linguistic microfeatures explained 60% of the variance in human scores for essays in a test set, providing an exact accuracy of 55% and an adjacent accuracy of 96%. To examine the limitations of the model, a post-hoc analysis was conducted to investigate differences in the scoring outcomes produced by the model and the human raters for essays with score differences of two or greater (N = 20). Essays scored as high by the regression model and low by human raters contained more word types and perfect tense forms compared to essays scored high by humans and low by the regression model. Essays scored high by humans but low by the regression model had greater coherence, syntactic variety, syntactic accuracy, word choices, idiomaticity, vocabulary range, and spelling accuracy as compared to essays scored high by the model but low by humans. Overall, findings from this study provide important information about how linguistic microfeatures can predict L2 essay quality for TOEFL-type exams and about the strengths and weaknesses of automatic essay scoring models.

Text cohesion is an important element of discourseprocessing. This paper presents a new approach to modeling,quantifying, and visualizing text cohesion using automatedcohesion flow indices that capture semantic links amongparagraphs. Cohesion flow is calculated by applyingCohesion Network Analysis, a combination of semanticdistances, Latent Semantic Analysis, and Latent DirichletAllocation, as well as Social Network Analysis. Experimentsperformed on 315 timed essays indicated that cohesion flowindices are significantly correlated with human ratings of textcoherence and essay quality. Visualizations of the globalcohesion indices are also included to support a more facileunderstanding of how cohesion flow impacts coherence interms of semantic dependencies between paragraphs.

We investigated the differential effects of self-explaining arefutational text, compared to thinking aloud or rereading.Undergraduate students (n = 105) read a refutational textabout natural selection and were asked to either self-explain,think-aloud, or re-read the text. Then they completed aposttest that assessed general knowledge of natural selection.Students who self-explained the refutational text subsequentlyoutperformed their peers on a test of their knowledge ofnatural selection. Additionally, the results suggest that bothinstructional and performance differences were significantlylinked to the degree of causal cohesion present withinstudents’ natural language responses to the text (i.e., selfexplanationsand think-alouds).

Using students' process data from the game-based Intelligent Tutoring System (ITS) iSTART-ME, the current study examines students' propensity to use system currency to unlock game-based features, (i.e., referred to here as "spendency"). This study examines how spendency relates to students' interaction preferences, in-system performance, and learning outcomes (i.e., self-explanation quality, comprehension). A group of 40 high school students interacted with iSTART-ME as part of an 11-session experiment (pretest, eight training sessions, posttest, and a delayed retention test). Students' spendency was negatively related to the frequency of their use of personalizable features. In addition, students' spendency was negatively related to their in-system achievements, daily learning outcomes, and performance on a transfer comprehension task, even after factoring out prior ability. The findings from this study indicate that increases in students' spendency are systematically related to their selection choices and may have a negative effect on in-system performance, immediate learning outcomes, and skill transfer outcomes. The results have particular relevance to game-based systems that incorporate currency to unlock features within games as well as to the differential tradeoffs of game features on motivation and learning.

Natural language processing (NLP) provides a powerful approach for discourse processing researchers. However, there remains a notable degree of hesitation by some researchers to consider using NLP, at least on their own. The purpose of this article is to introduce and make available a "simple" NLP (SiNLP) tool. The overarching goal of the article is to proliferate the use of NLP in discourse processing research. The article also provides an instantiation and empirical evaluation of the linguistic features measured by SiNLP to demonstrate their strength in investigating constructs of interest to the discourse processing community. Although relatively simple, the results of this analysis reveal that the tool is quite powerful, performing on par with a sophisticated text analysis tool, Coh-Metrix, on a common discourse processing task (i.e., predicting essay scores). Such a tool could prove useful to researchers interested in investigating features of language that affect discourse production and comprehension.

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students' comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students' reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students' reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students' reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students' literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students. [ABSTRACT FROM AUTHOR]