Search

The expanding domain of Artificial Intelligence (AI) offers a diverse array of educational applications and tools. However, the scholarly exploration of AI's suitability for enhancing English as a Foreign Language (EFL) instruction at the university level remains notably limited. This research gap impedes educators from fully harnessing AI's pedagogical potential. Given the inclusion of linguistic and literary disciplines in preservice EFL teacher training in Slovakia, it is increasingly imperative for educators to acquaint themselves with various AI tools, enabling the development of effective methodologies for enhancing EFL teaching and learning. Integrating AI into teacher training programs equips future EFL educators with essential skills for 21st-century classrooms and meets the evolving needs of digitally proficient students. This paper aims to provide a concise yet comprehensive overview of AI's relevance to pre-service EFL teacher training, encompassing linguistic and literary domains, by categorising six prominent AI forms: a) Natural Language Processing (NLP) Tools, b) Content Creation and Personalisation tools, c) Content Recommendation Systems, d) Emotion and Sentiment Analysers, e) Text Summarisation and Analysis tools, and f) Chatbots and Virtual Assistants. Furthermore, it highlights the research gap in AI's implementation in EFL education and emphasises the need to explore pedagogical and ethical implications while outlining future research directions to enhance our understanding of this dynamic field. [For the full proceedings, see ED652261.]

Exploring students' discourse in academic settings over time can provide valuable insight into the evolution of learner engagement and participation in online learning. In this study, we propose an analytical framework to capture topics and the temporal progression of learner discourse. We employed a Contextualized Topic Modeling technique on messages posted by undergraduates in online discussion forums from Fall 2019 to Spring 2020. We further evaluated if topics were originating from specific courses or more generally distributed across multiple courses. Our results suggested a significant increase in the number of general topics after the onset of the pandemic, suggesting emergent topics being discussed in a range of courses. In addition, using Word Mover's Distance, we examined the semantic similarity of topics in adjacent months and constructed topic chains. Our findings indicated that previously course-centric topics such as public health developed into more general discussions that emphasize inequities and healthcare during the pandemic. Furthermore, emergent topics around students' lived experiences underscored the role of discussion forums in capturing educational experiences temporally. Finally, we discuss the implications of current findings for post-pandemic higher education and the effectiveness of our framework in exploring unstructured large-scale educational data. [For the complete proceedings, see ED630829.]

Students who take an online course, such as a MOOC, use the course's discussion forum to ask questions or reach out to instructors when encountering an issue. However, reading and responding to students' questions is difficult to scale because of the time needed to consider each message. As a result, critical issues may be left unresolved, and students may lose the motivation to continue in the course. To help address this problem, we build predictive models that automatically determine the urgency of each forum post, so that these posts can be brought to instructors' attention. This paper goes beyond previous work by predicting not just a binary decision cut-off but a post's level of urgency on a 7-point scale. First, we train and cross-validate several models on an original data set of 3,503 posts from MOOCs at University of Pennsylvania. Second, to determine the generalizability of our models, we test their performance on a separate, previously published data set of 29,604 posts from MOOCs at Stanford University. While the previous work on post urgency used only one data set, we evaluated the prediction across different data sets and courses. The best-performing model was a support vector regressor trained on the Universal Sentence Encoder embeddings of the posts, achieving an RMSE of 1.1 on the training set and 1.4 on the test set. Understanding the urgency of forum posts enables instructors to focus their time more effectively and, as a result, better support student learning. [For the complete proceedings, see ED630829.]

Large language models (LLMs), such as Codex, hold great promise in enhancing programming education by automatically generating feedback for students. We investigate using LLMs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming. More concretely, given a student's buggy program, our goal is to generate feedback comprising a fixed program along with a natural language explanation describing the errors/fixes, inspired by how a human tutor would give feedback. While using LLMs is promising, the critical challenge is to ensure high precision in the generated feedback, which is imperative before deploying such technology in classrooms. The main research question we study is: "Can we develop LLMs-based feedback generation techniques with a tunable precision parameter, giving educators quality control over the feedback that students receive?" To this end, we introduce PyFiXV, our technique to generate high-precision feedback powered by Codex. The key idea behind PyFiXV is to use a novel run-time validation mechanism to decide whether the generated feedback is suitable for sharing with the student; notably, this validation mechanism also provides a precision knob to educators. We perform an extensive evaluation using two real-world datasets of Python programs with syntax errors and show the efficacy of PyFiXV in generating high-precision feedback. [For the complete proceedings, see ED630829.]

Teachers often rely on the use of a range of open-ended problems to assess students' understanding of mathematical concepts. Beyond traditional conceptions of student open-ended work, commonly in the form of textual short-answer or essay responses, the use of figures, tables, number lines, graphs, and pictographs are other examples of open-ended work common in mathematics. While recent developments in areas of natural language processing and machine learning have led to automated methods to score student open-ended work, these methods have largely been limited to textual answers. Several computer-based learning systems allow students to take pictures of hand-written work and include such images within their answers to open-ended questions. With that, however, there are few-to-no existing solutions that support the auto-scoring of student hand-written or drawn answers to questions. In this work, we build upon an existing method for auto-scoring textual student answers and explore the use of OpenAI/CLIP, a deep learning embedding method designed to represent both images and text, as well as Optical Character Recognition (OCR) to improve model performance. We evaluate the performance of our method on a dataset of student open-responses that contains both text- and image-based responses, and find a reduction of model error in the presence of images when controlling for other answer-level features. [For the complete proceedings, see ED630829.]

We propose an innovative, effective, and data-agnostic method to train a deep-neural network model with an extremely small training dataset, called VELR (Voting-based Ensemble Learning with Rejection). In educational research and practice, providing valid labels for a sufficient amount of data to be used for supervised learning can be very costly and often impractical. The shortage of training data often results in deep neural networks being overfitting. There are many methods to avoid overfitting such as data augmentation and regularization. Though, data augmentation is considerably data dependent and does not usually work well for natural language processing tasks. Moreover, regularization is often quite task specific and costly. To address this issue, we propose an ensemble of overfitting models with uncertainty-based rejection. We hypothesize that misclassification can be identified by estimating the distribution of the class-posterior probability P(y|x) as a random variable. The proposed VELR method is data independent, and it does not require changes to the model structure or the re-training of the model. Empirical studies demonstrated that VELR achieved classification accuracy of 0.7 with only 200 samples per class on the CIFAR-10 dataset, but 75% of input samples were rejected. VELR was also applied to a question generation task using a BERT language model with only 350 training data points, which resulted in generating questions that are indistinguishable from human-generated questions. The paper concludes that VELR has potential applications to a broad range of real-world problems where misclassification is very costly, which is quite common in the educational domain. [For the complete proceedings, see ED630829.]

Mobile learning or mLearning has become an essential tool in many fields in this digital era, among the ones educational training deserves special attention, that is, applied to both basic and higher education towards active, flexible, effective high-quality and continuous learning. However, despite the advances in Natural Language Processing techniques, mLearning still does not take full advantage of the great potential of the latter techniques. It is the case of the transformer language models distributed in chatbot form. Accordingly, this work presents a mLearning application based on OpenAI GPT-3 model, which enables user-friendly interactions. The proposed system was tested in two evaluation scenarios, and the experimental results endorse its appropriateness toward learning experience enhancement. [For the full proceedings, see ED639391.]

This paper delves into statistical disparities between human-written and ChatGPT-generated texts, utilizing an analysis of Shannon's equitability values, and token frequency. Our findings indicate that Shannon's equitability can potentially be a differentiating factor between texts produced by humans and those generated by ChatGPT. Additionally, we uncover substantial distinctions when studying the most frequent tokens. [For the full proceedings, see ED636095.]

The growth of Artificial Intelligence (AI) chatbots has created a great deal of discussion in the education community. While many have gravitated towards the ability of these bots to make learning more interactive, others have grave concerns that student created essays, long used as a means of assessing the subject comprehension of students, may be at risk. The bot's ability to quickly create high quality papers, sometimes complete with reference material, has led to concern that these programs will make students too reliant on their ability and not develop the critical thinking skills necessary to succeed. The rise in these applications has led to the need for the development of detection programs that are able to read the students submitted work and return an accurate estimation of if the paper is human or computer created. These detection programs use natural language processing's (NLP) ideas of perplexity, or randomness of the text, and burstiness, or the tendency for certain words and phrases to appear together, plus sophisticated algorithms to compare the essays to preexisting literature to generate an accurate estimation on the likely author of the paper. The use of these systems has been found to be highly effective in reducing plagiarism among students, however concerns have been raised about the limitations of these systems. False positives, false negatives, and cross language identification are three areas of concern amongst faculty and have led to reduced usage of the detection engines. Despite the limitations however, these systems are a valuable tool for educational institutions to maintain academic integrity and ensure that students are submitting original work. [For the full proceedings, see ED656038.]

This paper presents a reflection on the design of an Intelligent Computer-Assisted Language Learning (ICALL) 'ecosystem', integrated into an online learning environment for Spanish as a Foreign Language (SFL). The innovative dimension of the ecosystem lies in its triple focus: apart from enabling users to create and use intelligent language learning materials, it also tracks their activities in the environment and provides them insights (e.g. through knowledge clips) into Natural Language Processing (NLP), the source of ICALL's 'intelligence'. The reflective analysis is carried out by means of a case study with 32 SFL students, who work with the ecosystem in a blended writing course focused on vocabulary learning, lexical ambiguity, and Word Sense Disambiguation (WSD). Students' attitudes towards engaging in the ICALL ecosystem are gauged through a questionnaire, which revealed a statistically significant positive change in attitude after having completed the course. However, the results also show that enhanced insights into NLP and increased confidence in the computer as a learning assistant do not necessarily go hand in hand with an increased curiosity and a better user experience. [For the complete volume, "Intelligent CALL, Granular Systems and Learner Data: Short Papers from EUROCALL 2022 (30th, Reykjavik, Iceland, August 17-19, 2022)," see ED624779.]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

The process of generating challenging and appropriate distractors for multiple-choice questions is a complex and time-consuming task. Existing methods for an automated generation have limitations in proposing challenging distractors, or they fail to effectively filter out incorrect choices that closely resemble the correct answer, share synonymous meanings, or imply the same information. To overcome these challenges, we propose a comprehensive toolkit that integrates various approaches for generating distractors, including leveraging a general knowledge base and employing a T5 LLM. Additionally, we introduce a novel strategy that utilizes natural language inference to increase the accuracy of the generated distractors by removing confusing options. Our models have zero-shot capabilities and achieve good results on the DGen dataset; moreover, the models were fine-tuned and outperformed state-of-the-art methods on the considered dataset. To further extend the analysis, we introduce human annotations with scores for 100 test questions with 1085 distractors in total. The evaluations indicated that our generated options are of high quality, surpass all previous automated methods, and are on par with the ground truth of human-defined alternatives. [This paper was published in: "AIED 2024, LNAI 14830," edited by A. M. Olney et al., Springer Nature Switzerland, 2024, pp. 242-50.]

As language technologies have become more sophisticated and prevalent, there have been increasing concerns about bias in natural language processing (NLP). Such work often focuses on the effects of bias instead of sources. In contrast, this paper discusses how normative language assumptions and ideologies influence a range of automated language tools. These underlying assumptions can inform (a) grammar and tone suggestions provided by commercial products, (b) language varieties (e.g., dialects and other norms) taught by language learning technologies, (c) language patterns used by chatbots and similar applications to interact with users. These tools demonstrate considerable technological advancement but are rarely interrogated with regard to the language ideologies they intentionally or implicitly reinforce. We consider prior research on language ideologies and how they may impact (at scale) the large language models (LLMs) that underlie many automated language technologies. Specifically, this paper draws on established theoretical frameworks for understanding how humans typically perceive or judge language varieties and patterns that may differ from their own or their perceived standard. We then discuss how language ideologies can perpetuate social hierarchies and stereotypes, even within seemingly impartial automation. In doing so, we contribute to the emerging literature on how the risks of language ideologies and assumptions can be better understood and mitigated in the design, testing, and implementation of automated language technologies. [This paper was published in: "Universal Access in Human-Computer Interaction (HCII 2024) Proceedings, LNCS 14696," edited by M. Antona and C. Stephanidis, 2024, pp. 271-290.]

Multi-angle question answering models have recently been proposed that promise to perform related tasks like question generation. However, performance on related tasks has not been thoroughly studied. We investigate a leading model called Macaw on the task of multiple choice question generation and evaluate its performance on three angles that systematically reduce the complexity of the task. Our results indicate that despite the promise of generalization, Macaw performs poorly on untrained angles. Even on a trained angle, Macaw fails to generate four distinct multiple-choice options on 17% of inputs. We propose augmenting multiple-choice options by paraphrasing angle input and show this increases overall success to 97.5%. A human evaluation comparing the augmented multiple-choice questions with textbook questions on the same topic reveals that Macaw questions broadly score highly but below human questions. [This paper was published in: "Proceedings of the 3rd Workshop of the Learner Data Institute," edited by S. E. Fancsali and V. Rus, Learner Data Institute, 2022, pp. 18-23.]

Providing timely feedback is crucial in promoting academic achievement and student success. However, for multifarious reasons (e.g., limited teaching resources), feedback often arrives too late for learners to act on the feedback and improve learning. Thus, automated feedback systems have emerged to tackle educational tasks in various domains, including novice programming, short-essay writing, and open-ended questions. However, to the best of our knowledge, no previous study has investigated automated feedback generation on students' project reports. In this paper, we present a novel data-driven system, named "Insta-Reviewer," for automatically generating instant feedback on students' project reports, using state-of-the-art natural language processing (NLP) models. We also propose a framework for manually evaluating system-generated feedback. Experimental results show that feedback generated by Insta-Reviewer on real students' project reports can achieve near-human performance. Our work demonstrates the feasibility of automatic feedback generation for students' project reports while highlighting several prominent challenges for future research. [For the full proceedings, see ED623995.]

A child's ability to understand text (reading comprehension) can greatly impact both their ability to learn in the classroom and their future contributions to society. Reading comprehension draws on oral language; behavioural measures of knowledge at the word and sentence levels have been shown to be related to children's reading comprehension. In this study, we examined the impact of word and sentence level text-features on children's reading comprehension. We built a predictive model that uses natural language processing techniques to predict the question-level performance of students on reading comprehension tests. We showed that, compared to a model that used measures of student knowledge and subskills alone, a model that used features of sentence complexity, lexical surprisal, rare word use, and general context improved prediction accuracy by more than four percentage points. Our subsequent analyses revealed that these features compensate for the shortcomings of each other and work together to produce maximal performance. This provides insight into how different characteristics of the text and questions can be used to predict student performance, leading to new ideas about how text and reading comprehension interact. Our work also suggests that using a combination of text features could support the adaptation of reading materials to meet student needs. [For the full proceedings, see ED623995.]

An online peer-assessment system typically allows students to give textual feedback to their peers, with the goal of helping the peers improve their work. The amount of help that students receive is highly dependent on the quality of the reviews. Previous studies have investigated using machine learning to detect characteristics of reviews (e.g., Do they mention a problem, make a suggestion, or tell the student where to make a change?). Machine-learning approaches to peer-assessment evaluation are heavily reliant on labeled data to learn how to identify review characteristics. However, attaining reliable labels for those characteristics is always time-consuming and labor-intensive. In this study, we propose to apply pseudo-labeling, a semi-supervised learning-based strategy, to improve the recognition of reviews that detect problems in the reviewed work. This is done by utilizing a small, reliably labeled dataset along with a large unlabeled dataset to train a text classifier. The ultimate goal of this research is to show that for peer assessment evaluation, we can utilize both unlabeled and labeled datasets to obtain a robust auto-labeling system and thereby save much effort in labeling the data. [For the full proceedings, see ED623995.]

Peer assessment can be a more effective pedagogical method when reviewers provide quality feedback. But what makes feedback helpful to reviewees? Other studies have identified quality feedback as focusing on detecting problems, providing suggestions, or pointing out where changes need to be made. However, it is important to seek students' perspectives on what makes a review helpful to a reviewee. This study explores the helpfulness of feedback from students' perspectives when the feedback contained suggestions or mentioned problems or both. We applied natural language processing techniques to identify suggestions and problems mentioned in peer reviews. We also analyzed important text features that are associated with suggestions or problems detected by the peer feedback. The result showed that students are likely to find a review helpful if a suggestion is provided along with the problem mentioned in the feedback rather than simply identifying the problem. [For the full proceedings, see ED623995.]

With the rapid development of science and technology in recent years, the application areas of fuzzy logic have also gained speed. Fuzzy logic is a frequently preferred approach in the educational process, and it can be said that scientific publications on this topic have recently gained momentum in the literature. In this context, the present study aims to shed light on the research forms of fuzzy logic approach in education. The Web of Science (WoS) database was used for data collection. The search reached 374 scientific publications on the topic after various exclusion procedures. The obtained data were visualized using the mapping technique. As a result of the research, it was found that the number of publications on this topic has generally increased from past to present. The analysis revealed that the most influential countries reporting on this topic are the United States, China, and Taiwan. In addition, National Cheng Kung University, Gazi University, Cornell University, and Marmara University have published the most publications. The authors with the most publications on the fuzzy logic approach are Reyna V.F., Castillo, O. and Huang, Y.M. The analysis of the journals showed that Computer Applications in Engineering Education and International Journal of Electrical Engineering Education are the journals that contribute the most to this topic. The concepts that appeared in the clusters of co-occurrences analysis are fuzzy logic, higher education, e-learning, data mining, matlab and education. [For the full proceedings, see ED630948.]

In an Intelligent Tutoring System (ITS), problem (or question) difficulty is one of the most critical parameters, directly impacting problem design, test paper organization, result analysis, and even the fairness guarantee. However, it is very difficult to evaluate the problem difficulty by organized pre-tests or by expertise, because these solutions are labor-intensive, time-consuming, leakage-prone, or subjective in some way. Thus, it is of importance to automatically evaluate problem difficulty via information technology. To this end, we propose a novel difficulty prediction framework, named SQL-DP, for Structured Query Language (SQL) programming problems, mastering which plays a vital role in learning the database technology. In SQL-DP, semantic features of problem stems and structure features of problem answers in the form of SQL codes are both computed at first, using the NLP and the neural network techniques. Then, these features are used as the input to train a difficulty prediction model with the statistic error rates in tests as the training labels, where the whole modeling does not introduce any experts, some as knowledge labeling. Finally, with the trained model, we can automatically predict the difficulty of each SQL programming problem. Moreover, SQL programming problem answering log data of hundreds of undergraduates from Guangxi University of China are collected, and the experiments conducted on the collected log data demonstrate the propped SQL-DP framework outperforms the state-of-the-art solutions apparently. In particular, SQL-DP decreases the RMSE of difficulty prediction by at most 7.23%, compared with the best-related framework. [For the full proceedings, see ED623995.]

Mobile artifacts are the objects that increasingly surround us in life. They provide us with the opportunity to engage in activities outside the traditional context and at our own pace. In this article, we present the results of the tests of our mobile application intended for the learning of the French language with the concept of SPOC with a methodology of teaching pedagogy by project founded based on MIRTO by using the NLP tools. [For the complete proceedings, see ED639262.]

The COVID-19 epidemic had caused one of the most significant disruptions to the global education system. Many educational institutions faced sudden pressure to switch from face-to-face to online delivery of courses. The conventional classes are no longer the primary means of delivery; instead, online education and resources have become the prominent approach. With the increasing demand for supplementary course materials to fulfill the needs of each area of study, students began to use search engines and online resources that contain discussions, practical demonstrations, and tutorial videos to aid students in their studies and course work. This study addresses the underlying challenges of retrieving relevant online educational materials by introducing an intelligent agent for semantic data mining. It works as middleware infrastructure that allow context-aware data processing and mining. YouTube was used to assess the consistency of the proposed model since it returns a large number of results in its search pool. The results showed that using the extraction of topics method, the similarities scores with the proposed model provided favorable results. Furthermore, an improvement in video ranking and sorting was realized. According to the findings, using this method provided users with a more productive and reliable study experience. [For the full proceedings, see ED639633.]

Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled human evaluation of three conditions: a fine-tuned, augmented version of Macaw, instruction-tuned Bing Chat with zero-shot prompting, and human-authored questions from a college science textbook. Our results indicate that on six of seven measures tested, both LLM's performance was not significantly different from human performance. Analysis of LLM errors further suggests that Macaw and Bing Chat have different failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. For Macaw, removing error items from analysis results in performance on par with humans for all metrics; for Bing Chat, removing error items improves performance but does not reach human-level performance. [This paper was published in the "CEUR Workshop Proceedings," 2023.]

The ability to automatically assess the quality of paraphrases can be very useful for facilitating literacy skills and providing timely feedback to learners. Our aim is twofold: a) to automatically evaluate the quality of paraphrases across four dimensions: lexical similarity, syntactic similarity, semantic similarity and paraphrase quality, and b) to assess how well models trained for this task generalize. The task is modeled as a classification problem and three different methods are explored: (a) manual feature extraction combined with an Extra Trees model, (b) GloVe embeddings and a Siamese neural network, and (c) using a pre-trained BERT model fine-tuned on our task. Starting from a dataset of 1998 paraphrases from the User Language Paraphrase Corpus (ULPC), we explore how the three models trained on the ULPC dataset generalize when applied on a separate, small paraphrase corpus based on children inputs. The best out-of-the-box generalization performance is obtained by the Extra Trees model with at least 75% average F1-scores for the three similarity dimensions. We also show that the Siamese neural network and BERT models can obtain an improvement of at least 5% after fine-tuning across all dimensions.

Automated Writing Evaluation systems have been developed to help students improve their writing skills through the automated delivery of both summative and formative feedback. These systems have demonstrated strong potential in a variety of educational contexts; however, they remain limited in their personalization and scope. The purpose of the current study was to begin to address this gap by examining whether individual differences could be modeled in a source-based writing context. Undergraduate students (n=106) wrote essays in response to multiple sources and then completed an assessment of their vocabulary knowledge. Natural language processing tools were used to characterize the linguistic properties of the source-based essays at four levels: descriptive, lexical, syntax, and cohesion. Finally, machine learning models were used to predict students' vocabulary scores from these linguistic features. The models accounted for approximately 29% of the variance in vocabulary scores, suggesting that the linguistic features of source-based essays are reflective of individual differences in vocabulary knowledge. Overall, this work suggests that automated text analyses can help to understand the role of individual differences in the writing process, which may ultimately help to improve personalization in computer-based learning environments. [Paper published in: "LAK21, April 12-16, 2021, Irvine, CA, USA," ACM, 2021, pp. 620-625.]

Numerous approaches have been introduced to automate the process of text summarization, but only few can be easily adapted to multiple languages. This paper introduces a multilingual text processing pipeline integrated in the open-source "ReaderBench" framework, which can be retrofit to cover more than 50 languages. While considering the extensibility of the approach and the problem of missing labeled data for training in various languages besides English, an unsupervised algorithm was preferred to perform extractive summarization (i.e., select the most representative sentences from the original document). Specifically, two different approaches relying on text cohesion were implemented:(1) a graph-based text representation derived from Cohesion Network Analysis that extends TextRank; and (2) a class of submodular set functions. Evaluations were performed on the DUC dataset and use as baseline the implementation of TextRank from Gensim. Our results using the submodular set functions outperform the baseline. In addition, two use cases on English and Romanian languages are presented, with corresponding graphical representations for the two methods. [This paper was published in: 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) Proceedings, 2020, pp. 161-168 (ISBN 978-1-7281-7628-4).]

Text summarization is an effective reading comprehension strategy. However, summary evaluation is complex and must account for various factors including the summary and the reference text. This study examines a corpus of approximately 3,000 summaries based on 87 reference texts, with each summary being manually scored on a 4-point Likert scale. Machine learning models leveraging Natural Language Processing (NLP) techniques were trained to predict the extent to which summaries capture the main idea of the target text. The NLP models combined both domain and language independent textual complexity indices from the ReaderBench framework, as well as state-of-the-art language models and deep learning architectures to provide semantic contextualization. The models achieve low errors -- normalized MAE ranging from 0.13-0.17 with corresponding R2 values of up to 0.46. Our approach consistently outperforms baselines that use TF-IDF vectors and linear models, as well as Transfomer-based regression using BERT. These results indicate that NLP algorithms that combine linguistic and semantic indices are accurate and robust, while ensuring generalizability to a wide array of topics. [This paper was published in: A. I. Cristea and C. Troussas (Eds.), "ITS 2021: Intelligent Tutoring Systems proceedings," pp. 321-332, 2021. Springer, Cham Switzerland.]

The current study examined the extent to which the cohesion detected in readers' constructed responses to multiple documents was predictive of persuasive, source-based essay quality. Participants (N=95) completed multiple-documents reading tasks wherein they were prompted to think-aloud, self-explain, or evaluate the sources while reading a set of four texts. They were then asked to write a source-based essay based on their reading. Natural Language Processing techniques were used to automatically analyze the cohesion of the constructed responses at both within- and across-documents levels. Results indicated that within-document cohesion was "negatively" related to essay quality, whereas across-documents cohesion was positively related to essay quality. Further, these relations differed by instructional condition such that strategic instructions to either self-explain or evaluate sources seemed to promote across-text integration, compared to thinking aloud. Overall, this study indicates that the cohesion of constructed responses to text can provide insights into the coherence of the mental representations readers construct while reading multiple documents. [This paper was published in: "Proceedings of the Annual Meeting of the Cognitive Science," 2021, pp. 931-937 (ISSN- 1069-7977).]

Reading comprehension is key to knowledge acquisition and to reinforcing memory for previous information. While reading, a mental representation is constructed in the reader's mind. The mental model comprises the words in the text, the relations between the words, and inferences linking to concepts in prior knowledge. The automated model of comprehension (AMoC) simulates the construction of readers' mental representations of text by building syntactic and semantic relations between words, coupled with inferences of related concepts that rely on various automated semantic models. This paper introduces the second version of AMoC that builds upon the initial model with a revised processing pipeline in Python leveraging state-of-the-art NLP models, additional heuristics for improved representations, as well as a new radiant graph visualization of the comprehension model. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 119-123.]

Reading comprehension is key to knowledge acquisition and to reinforcing memory for previous information. While reading, a mental representation is constructed in the reader's mind. The mental model comprises the words in the text, the relations between the words, and inferences linking to concepts in prior knowledge. The automated model of comprehension (AMoC) simulates the construction of readers' mental representations of text by building syntactic and semantic relations between words, coupled with inferences of related concepts that rely on various automated semantic models. This paper introduces the second version of AMoC that builds upon the initial model with a revised processing pipeline in Python leveraging state-of-the-art NLP models, additional heuristics for improved representations, as well as a new radiant graph visualization of the comprehension model.

Today, there is a vast amount of online material for learners. However, due to the lack of prerequisite information needed to master them, a lot of time is spent in identifying the right learning content for mastering these concepts. A system that captures underlying prerequisites needed for learning different concepts can help improve the quality of learning and can save time for the learners as well. In this work, we propose an unsupervised approach, UPreG, for automatically inferring prerequisite relationships between different concepts using NLP techniques. Our approach involves extracting the concepts from unstructured texts in MOOC (Massively Open Online Courses) course descriptions, measuring semantic relatedness between the concepts and statistically inferring the prerequisite relationships between related concepts. We conducted both qualitative and quantitative studies to validate the effectiveness of our proposed approach. As there are no ground truth labels for these prerequisite relations, we conducted a user study for the evaluation of the prerequisite relations. We build the concept graph using prerequisite relations. We demonstrate few examples of the learning maps generated from the graph. The learning maps provide prerequisite information and learning paths for different concepts. [For the full proceedings, see ED615472.]

The needs for precisely estimating a student's academic performance have been emphasized with an increasing amount of attention paid to Intelligent Tutoring System (ITS). However, since labels for academic performance, such as test scores, are collected from outside of ITS, obtaining the labels is costly, leading to label-scarcity problem which brings challenge in taking machine learning approaches for academic performance prediction. To this end, inspired by the recent advancement of pre-training method in natural language processing community, we propose DPA, a transfer learning framework with Discriminative Pre-training tasks for Academic performance prediction. DPA pre-trains two models, a generator and a discriminator, and fine-tunes the discriminator on academic performance prediction. In DPA's pre-training phase, a sequence of interactions where some tokens are masked is provided to the generator which is trained to reconstruct the original sequence. Then, the discriminator takes an interaction sequence where the masked tokens are replaced by the generator's outputs, and is trained to predict the originalities of all tokens in the sequence. We conduct extensive experimental studies on a real-world dataset obtained from a multi-platform ITS application and show that DPA outperforms the previous state-of-the-art generative pre-training method with a reduction of 4.05% in mean absolute error and more robust to increased label-scarcity. [For the full proceedings, see ED615472.]

Collaborative dialogue is rich in conscious and subconscious coordination behaviours between participants. This work explores collaborative learner dialogue through theories of alignment, analysing inter-partner movement and language use with respect to our hypotheses: that they interrelate, and that they form predictors of collaboration quality and learning. In keeping with theories of alignment, we find that linguistic alignment and gestural synchrony both correlate significantly with one another in dialogue. We also find strong individual correlations of these metrics with collaboration quality. We find that linguistic and gestural alignment also correlate with learning. Through regression analysis, we find that although interconnected, these measures in combination are significant predictors of collaborative problem solving success. We contribute additional evidence to support the theory that alignment takes place across multiple levels of communication, and provide a methodological approach for analysing inter-speaker dynamics in a multimodal task based setting. Our work has implications for the teaching community, our measures can help identify poorly performing groups, lending itself to informing the design of real time intervention strategies or formative assessment for collaborative learning. [For the full proceedings, see ED615472.]

As students learn how to program, both their programming code and their understanding of it evolves over time. In this work, we present a general data-driven approach, named "Temporal-ASTNN" for modeling student learning progression in open-ended programming domains. Temporal-ASTNN combines a novel neural network model based on abstract syntactic trees (AST), named ASTNN, and Long-Short Term Memory (LSTM) model. ASTNN handles the "linguistic" nature of student programming code, while LSTM handles the "temporal" nature of student learning progression. The effectiveness of ASTNN is first compared against other models including a state-of-the-art algorithm, Code2Vec across two programming domains: iSnap and Java on the task of program classification ("correct" or "incorrect"). Then the proposed temporal-ASTNN is compared against the original ASTNN and other temporal models on a challenging task of student success early prediction. Our results show that Temporal-ASTNN can achieve the best performance with only the first 4-minute temporal data and it continues to outperform all other models with longer trajectories. [For the full proceedings, see ED615472.]

The current study explores the ability to predict argumentative claims in structurally-annotated student essays to gain insights into the role of argumentation structure in the quality of persuasive writing. Our annotation scheme specified six types of argumentative components based on the well-established Toulmin's model of argumentation. We developed feature sets consisting of word count, frequency data of key n-grams, positionality data, and other lexical, syntactic, semantic features based on both sentential and suprasentential levels. The suprasentential Random Forest model based on frequency and positionality features yielded the best results, reporting an accuracy of 0.87 and kappa of 0.73. This model will be included in an online writing assessment tool to generate feedback for student writers. [For the full proceedings, see ED615472.]

We explore how different components of an Automatic Short Answer Grading (ASAG) model affect the model's ability to generalize to questions outside of those used for training. For supervised automatic grading models, human ratings are primarily used as ground truth labels. Producing such ratings can be resource heavy, as subject matter experts spend vast amounts of time carefully rating a sample of responses. Further, it is often the case that multiple raters must come to a census before a final ground-truth rating is established. If ASAG models were developed that could generalize to out-of-sample questions, educators may be able to quickly add new questions to an auto-graded assessment without a continued manual rating process. For this project we explore various methods for producing vector representations of student responses including state-of-the-art representation methods such as Sentence-BERT as well as more traditional approaches including Word2Vec and Bag-of-words. We experiment with including previously untapped question-related information within the model input, such as the question text, question context text, scoring rubric information and a question-bundle identifier. The out-of-sample generalizability of the model is examined with both a leave-one-question-out and leave-one-bundle-out evaluation method and compared against a typical student-level cross validation. [For the full proceedings, see ED615472.]

In contrast to simple feedback, which provides students with the correct answer, elaborated feedback provides an explanation of the correct answer with respect to the student's error. Elaborated feedback is thus a challenge for AI in education systems because it requires dynamic explanations, which traditionally require logical reasoning and knowledge engineering to generate. This study presents an alternative approach that formulates elaborated feedback in terms of long-form question answering (LFQA). An off-the-shelf LFQA system was evaluated by human raters in a 2x2x2x2 ablation design that manipulated the context documents given to the LFQA model and the post-processing of model output. Results indicate that context manipulations improve performance but that postprocessing can have detrimental results. [This paper was published in: "Proceedings of the Eighth ACM Conference on Learning @ Scale," 2021, pp. 27-36.]

Educational content labeled with proper knowledge components (KCs) are particularly useful to teachers or content organizers. However, manually labeling educational content is labor intensive and error-prone. To address this challenge, prior research proposed machine learning based solutions to auto-label educational content with limited success. In this work, we significantly improve prior research by (1) expanding the input types to include KC descriptions, instructional video titles, and problem descriptions (i.e., three types of prediction task), (2) doubling the granularity of the prediction from 198 to 385 KC labels (i.e., more practical setting but much harder multinomial classification problem), (3) improving the prediction accuracies by 0.5-2.3% using Task-adaptive Pre-trained BERT, outperforming six baselines, and (4) proposing a simple evaluation measure by which we can recover 56-73% of mispredicted KC labels. All codes and data sets in the experiments are available at: https://github.com/tbs17/TAPT-BERT [This paper was published in: "AIED 2021, LNAI1 2748," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 408-19.]

Reading comprehension is essential for both knowledge acquisition and memory reinforcement. Automated modeling of the comprehension process provides insights into the efficacy of specific texts as learning tools. This paper introduces an improved version of the Automated Model of Comprehension, version 3.0 (AMoC v3.0). AMoC v3.0 is based on two theoretical models of the comprehension process, namely the Construction-Integration and the Landscape models. In addition to the lessons learned from the previous versions, AMoC v3.0 uses Transformer-based contextualized embeddings to build and update the concept graph as a simulation of reading. Besides taking into account generative language models and presenting a visual walkthrough of how the model works, AMoC v3.0 surpasses the previous version in terms of the Spearman correlations between our activation scores and the values reported in the original Landscape Model for the presented use case. Moreover, features derived from AMoC significantly differentiate between high-low cohesion texts, thus arguing for the model's capabilities to simulate different reading conditions. [This paper was published in: "AIED 2023: Artificial Intelligence in Education," edited by N. Wang et al., Springer, Switzerland, 2023, pp. 229-241, 2023.]

Self-explanation practice is an effective method to support students in better understanding complex texts. This study focuses on automatically assessing the comprehension strategies employed by readers while understanding STEM texts. Data from 3 datasets (N = 11,833) with self-explanations annotated on different comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and an overall quality score was used to train various machine learning models in both single-task and multi-task setups. Our end-to-end neural architecture considers RoBERTa as an encoder applied to the target and self-explanation texts, combined with handcrafted features for assessing text cohesion and filtering out low-quality examples. The best configuration obtained a 0.699 weighted F1-score for the overall self-explanation quality. [This paper was published in: "AIED 2023, CCIS 1831," edited by N. Wang et al., Springer Nature, 2023, pp. 695-700.]

In this investigation we propose new machine learning methods for automated scoring models that predict the vocabulary acquisition in science and social studies of second grade English language learners, based upon free-form spoken responses. We evaluate performance on an existing dataset and use transfer learning from a large pre-trained language model, reporting the influence of various objective function designs and the input-convex network design. In particular, we find that combining objective functions with varying properties, such as distance among scores, greatly improves the model reliability compared to human raters. Our models extend the current state of the art performance for assessing word definition tasks and sentence usage tasks in science and social studies, achieving excellent quadratic weighted kappa scores compared with human raters. However, human-human agreement still surpasses model-human agreement, leaving room for future improvement. Even so, our work highlights the scalability of automated vocabulary assessment of free-form spoken language tasks in early grades. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

This article describes the development of a tense and aspect identifier, an online tool designed to help learners of English by harnessing a natural language processing pipeline to automatically classify verb groups into one of 12 grammatical tenses. Currently, there is no website or application that can automatically identify tense in context, and the tense and aspect identifier fills that niche. Learners can use the tool to see how grammatical tenses are used in context. Finite verb groups are automatically identified, and the relevant words in the verb group are highlighted and colorized according to the tense identified. The latest deployed system can identify tenses in simple, compound, and complex sentences. False positive results occur when there is ellipsis of auxiliary verbs or when the tagger assigns the incorrect part-of-speech tag. The user interface of the tense identifier is a web app created using the Flask framework and deployed from the Heroku platform. The tool can be used for inductive and deductive teaching approaches, or even to check for tense consistency in a thesis. [For the complete volume, "CALL for Widening Participation: Short Papers from EUROCALL 2020 (28th, Online, August 20-21, 2020)," see ED610330.]

An intelligent textbook may be considered to be an interaction layer that lies between the text and the student, helping the student to master the content in the text. The Mobile Fact and Concept Training System (MoFaCTS) is an adaptive instructional system for simple content that has been developed into an interaction layer to mediate textbook instruction and so is being transformed into the Mobile Fact and Concept Textbook System (MoFaCTS). In this project, MoFaCTS is being completely retooled to accept texts from a textbook and to automatically create cloze sentence practice content to help the student learn the material in the text. Additional features in the prototype stage include automatically generated refutational feedback for incorrect cloze responses and a dialog system, which will trigger a short conversation by a tutor to correct conceptual misunderstandings. MoFaCTS administers this content via a web browser, providing the teacher with score reports and class management tools. Because the "optimal practice" module is interchangeable and the cloze content can come from any text, the system is highly configurable for different grade levels, populations, and academic subjects. To foster faster research progress, data export supports the DataShop transaction format, which allows quick analysis of data using the DataShop tools. [This paper was published in the CEUR workshop proceedings (Vol. 2674).]

Word vectors are widely used as input features in natural language processing (NLP) tasks. Researchers have found that word vectors often encode the biases of society, and steps have been taken towards debiasing the vectors themselves. However, little has been said about the fairness of the methods used to evaluate the quality of vectors. Analogical and word similarity tasks are commonplace, but both rely on purportedly ground truth statements about the semantic relationships between words (e.g. "man is to woman as king is to queen"). These analogies look reasonable when only taking into account the literal meanings of words, but two issues arise: (1) people don't always use words in a literal sense, and (2) the same word may be used differently by different groups of people. In this paper, we split a dataset of over 800,000 college admissions essays into quartiles based on reported household income (RHI) and train sets of word vectors on each quartile. We then test these sets of vectors on common intrinsic evaluation tasks. We find that vectors trained on the essays of higher income students encode more of each task's target semantic relationships than vectors trained on the essays of lower income students. These results hold even when controlling for word frequency. We conclude that the tasks themselves are biased towards the writing of higher income students, and we challenge the notion that there exist ground truth semantic relationships that word vectors must encode in order to be useful. [For the full proceedings, see ED607784.]

Peer assessment adds value when students provide "helpful" feedback to their peers. But, this begs the question of how we determine "helpfulness." One important aspect is whether the review detects problems in the submitted work. To recognize problem detection, researchers have employed NLP and machine-learning text classification methods. Past studies have used datasets that were narrowly focused on a small number of classes in specific academic fields. This paper reports on how well models trained on one dataset or field perform on data from classes that are unlike the classes whose data they have been trained on. Specifically we took a model developed with data from a computer science class with several programming assignments, and tried to transfer it onto an education class focused more on writing research papers. We have attempted to perform such a task on a few models including logistic regression classifier, random forest classifier, multinomial naive bayes classifier and support vector machine. We made several attempts to raise the accuracy of classification, including lemmatizing to deduct variation in data input, and active learning strategies. [For the full proceedings, see ED607784.]

Discussion forums are used to support socio-collaborative learning processes among students in online courses. However, complex forum structures and lengthy discourse require that students spend their limited time searching and filtering through posts to find those that are relevant to them rather than spending that time engaged in other meaningful learning activities (i.e., discussion). Moreover, existing adaptive systems do not accommodate individual learner needs in these contexts. In this work, we propose a multi-relational graph-based recommendation approach that mines student interaction logs to address the above problems within discussionbased socio-collaborative online courses. To account for the social aspects of learning, our approach incorporates learner modeling, social network analysis, and natural language processing techniques; it offers tailored recommendations of forum posts for learners with different types of interaction behaviors. In our experiments with small online courses, our approach outperformed competitor approaches in terms of recommendation precision while meeting expectations with respect to diversity and novelty. The results illustrate the proposed algorithm's effectiveness in predicting student preferences, suggesting its potential to increase student participation in discussion-related learning activities. [For the full proceedings, see ED607784.]

The 13th iteration of the International Conference on Educational Data Mining (EDM 2020) was originally arranged to take place in Ifrane, Morocco. Due to the SARS-CoV-2 (coronavirus) epidemic, EDM 2020, as well as most other academic conferences in 2020, had to be changed to a purely online format. To facilitate efficient transmission of presentations all paper presenters pre-recorded their presentation as a video and then hosted it on YouTube with closed-captioning (CC). The official theme of this year's conference is Improving Learning Outcomes for All Learners. The theme comprises two parts: (1) Identifying actionable learning or teaching strategies that can be used to "improve" learning outcomes, not just predict them; (2) Using EDM to promote more "equitable" learning across diverse groups of learners, and to benefit underserved communities in particular. This year's conference features three invited talks: Alina von Davier, Chief Officer at ACTNext; Abelardo Pardo, Professor and Dean of Programs (Engineering), at UniSA STEM, University of South Australia; and Kobi Gal, Associate Professor at the Department of Software and Information Systems Engineering at Ben-Gurion University of the Negev, and Reader at the School of Informatics at the University of Edinburgh.

Educational software data promises unique insights into students' study behaviors and drivers of success. While much work has been dedicated to performance prediction in massive open online courses, it is unclear if the same methods can be applied to blended courses and a deeper understanding of student strategies is often missing. We use pattern mining and models borrowed from Natural Language Processing (NLP) to understand student interactions and extract frequent strategies from a blended college course. Finegrained clickstream data is collected through Diderot, a noncommercial educational support system that spans a wide range of functionalities. We find that interaction patterns differ considerably based on the assessment type students are preparing for, and many of the extracted features can be used for reliable performance prediction. Our results suggest that the proposed hybrid NLP methods can provide valuable insights even in the low-data setting of blended courses given enough data granularity. [For the full proceedings, see ED607784.]

Learning from text is a "constructive" activity in which sentence-level information is combined by the reader to build coherent mental models. With increasingly complex texts, forming a mental model becomes challenging due to a lack of background knowledge, and limits in working memory and attention. To address this, we are taught "knowledge externalization" strategies such as active reading and diagramming. Unfortunately, paper-and-pencil approaches may not always be appropriate, and software solutions create friction through difficult input modalities, limited workflow support, and barriers between reading and diagramming. For all but the simplest text, building coherent diagrams can be tedious and difficult. We propose "Active Diagramming," an approach extending familiar active reading strategies to the task of diagram construction. Our prototype, texSketch, combines pen-and-ink interactions with natural language processing to reduce the cost of producing diagrams while maintaining the cognitive effort necessary for comprehension. Our user study finds that readers can effectively create diagrams without disrupting reading.