Search

Foreign language teaching achieves best learning outcomes when individual differences of learners are taken into account. While it is difficult for teachers to support internal differentiation in the classroom, digital tools can adaptively propose individual learning paths through activities so that students can practice with appropriately challenging exercises. But how can sufficiently varied, systematically parametrized exercises be provided to enable a system to match them to individual learner needs? We present an approach for high-variability exercise generation that transforms a single specification into a multitude of exercises varying in complexity. The approach is currently being evaluated in a randomized controlled study in regular German seventh grade English classes, facilitating a systematic empirical exploration of adaptive exercise complexity in relation to learning outcomes. [For the complete volume, "Intelligent CALL, Granular Systems and Learner Data: Short Papers from EUROCALL 2022 (30th, Reykjavik, Iceland, August 17-19, 2022)," see ED624779.]

Personalized learning stems from the idea that students benefit from instructional material tailored to their needs. Many online learning platforms purport to implement some form of personalized learning, often through on-demand tutoring or self-paced instruction, but to our knowledge none have a way to automatically explore for specific opportunities to personalize students' education nor a transparent way to identify the effects of personalization on specific groups of students. In this work we present the Automatic Personalized Learning Service (APLS). The APLS uses multi-armed bandit algorithms to recommend the most effective support to each student that requests assistance when completing their online work, and is currently used by ASSISTments, an online learning platform. The first empirical study of the APLS found that Beta-Bernoulli Thompson Sampling, a popular and effective multi-armed bandit algorithm, was only slightly more capable of selecting helpful support than randomly selecting from the relevant support options. Therefore, we also present Decision Tree Thompson Sampling (DTTS), a novel contextual multi-armed bandit algorithm that integrates the transparency and interpretability of decision trees into Thomson sampling. In simulation, DTTS overcame the challenges of recommending support within an online learning platform and was able to increase students' learning by as much as 10% more than the current algorithm used by the APLS. We demonstrate that DTTS is able to identify qualitative interactions that not only help determine the most effective support for students, but that also generalize well to new students, problems, and support content. The APLS using DTTS is now being deployed at scale within ASSISTments and is a promising tool for all educational learning platforms. [This paper was published in: "Proceedings of the Ninth ACM Conference on Learning @ Scale (L@S '22), June 1-3, 2022, New York City, NY, USA," ACM, 2022.]

Automatic short answer grading is an important research direction in the exploration of how to use artificial intelligence (AI)-based tools to improve education. Current state-of-the-art approaches use neural language models to create vectorized representations of students responses, followed by classifiers to predict the score. However, these approaches have several key limitations, including i) they use pre-trained language models that are not well-adapted to educational subject domains and/or student-generated text and ii) they almost always train one model per question, ignoring the linkage across question and result in a significant model storage problem due to the size of advanced language models. In this paper, we study the problem of automatic short answer grading for students' responses to math questions and propose a novel framework for this task. First, we use MathBERT, a variant of the popular language model BERT adapted to mathematical content, as our base model and fine-tune it on the downstream task of student response grading. Second, we use an in-context learning approach that provides scoring examples as input to the language model to provide additional context information and promote generalization to previously unseen questions. We evaluate our framework on a real-world dataset of student responses to open-ended math questions and show that our framework (often significantly) outperform existing approaches, especially for new questions that are not seen during training. [For the full proceedings, see ED623995.]

From 2020/2021 the evaluation of students' learning in primary school is no longer expressed through a decimal-based score, but through a descriptive assessment, in a more formative perspective of evaluation. In addition, the widespread use of technological tools in school has changed the teaching/learning process and it has highlighted the need to adopt new teaching methodologies. In this context, it becomes fundamental to train teachers and future teachers on the use of digital resources and innovative teaching methodologies, such as automatic formative assessment. This article presents a training module on automatic formative assessment which involved 153 university students of the Master's Degree course in Primary Education and aims to answer the following research questions: Were students able to design questions with formative assessment and interactive feedback? What were the most adopted strategies? All the forms and the questionnaire completed by students during the training course were analyzed. Even if most of the students had some difficulties in designing the interactive feedback, they really appreciated the development of the training module and the proposed methodologies. They find formative assessment very effective in several aspects. [For the full proceedings, see ED639633.]

Automatic speech recognition (ASR) has considerable potential to model aspects of classroom discourse with the goals of automated assessment, feedback, and instructional support. However, modeling student talk is besieged by numerous challenges including a lack of data for child speech, low signal to noise ratio, speech disfluencies, and multiparty chatter. This raises the question as to whether contemporary ASR systems, which are benchmarked on adult speech in idealized conditions, can be used to transcribe child speech in classroom settings. To address this question, we collected a dataset of 32 audio recordings of 30 middle-school students engaged in small group work (dyads, triads and tetrads) in authentic classroom settings. The audio was sampled, segmented, and transcribed by humans as well as three ASR engines (Google, Rev.ai, IBM Watson). Whereas all three ASRs had high word error rates, these mainly consisted of deletion errors. Further, Google successfully transcribed a greater proportion of utterances than the other two, but with more word substitutions; insertions were low across the board. ASR accuracy was robust to different speakers and recording idiosyncrasies evidenced by <5% of variance in error rates attributable to the student and recording session. We found that ASR errors had a larger negative effect on downstream natural language processing tasks at the word, phrase, and semantic levels rather than at the discourse level. Our findings indicate that ASR can be used to extract meaningful information from noisy classroom speech and might be more suitable for applications that require higher precision but are tolerant of lower recall. [For the full proceedings, see ED623995.]

Learning objectives, especially those well defined by applying Bloom's taxonomy for Cognitive Objectives, have been widely recognized as important in various teaching and learning practices. However, many educators have difficulties developing learning objectives appropriate to the levels in Bloom's taxonomy, as they need to consider the progression of learners' skills with learning content as well as dependencies between different learning objectives. To remedy this challenge, we aimed to apply state-of-the-art computational techniques to automate the classification of learning objectives based on Bloom's taxonomy. Specifically, we collected 21,380 learning objectives from 5,558 different courses at an Australian university and manually labeled them according to the six cognitive levels of Bloom's taxonomy. Based on the labeled dataset, we applied five conventional machine learning approaches (i.e., naive Bayes, logistic regression, support vector machine, random forest, and XGBoost) and one deep learning approach based on pre-trained language model BERT to construct classifiers to automatically determine a learning objective's cognitive levels. In particular, we adopted and compared two methods in constructing the classifiers, i.e., constructing multiple binary classifiers (one for each cognitive level in Bloom's taxonomy) and constructing only one multi-class multi-label classifier to simultaneously identify all the corresponding cognitive levels. Through extensive evaluations, we demonstrated that: (i) BERT-based classifiers outperformed the others in all cognitive levels (Cohen's K up to 0.93 and F1 score up to 0.95); (ii) three machine learning models -- support vector machine, random forest, and XGBoost -- delivered performance comparable to the BERT-based classifiers; and (iii) most of the binary BERT-based classifiers (5 out of 6) slightly outperformed the multi-class multi-label BERT-based classifier, suggesting that separating the characterization of different cognitive levels seemed to be a better choice than building only one model to identify all cognitive levels at one time. [For the full proceedings, see ED623995.]

Vocabulary learning that overlooks the discrimination of single sounds in English language learning could lead to communication difficulties, such as the inability to hear and speak English pronunciation. To address this issue, vocalization strategy, in which students listen to the sounds of words and memorize them aloud, is considered a useful learning strategy, and has been reported to influence performance in English language learning. Many studies have shown that metacognition promotes the use of this strategy. In this study, we developed an English vocabulary learning support system using dictation-type speech recognition technology and evaluated whether it activates learners' metacognition and promotes the use of vocalization strategies.

This mixed-methods one-shot study examines L2 writers' perceptions of using Automatic Speech Recognition (ASR) to write using the Technology Acceptance Model (TAM), based on three criteria: usefulness, ease of use, and intention to use. After receiving training on Google voice typing in Google Docs, 17 English as a Second Language (ESL) students carried out two ASR-based writing tasks over a two-hour period. After the treatment, participants filled in a TAM-informed survey and participated in semi-structured interviews to measure their perceptions based on the target criteria. Findings indicate positive perceptions of ASR as a writing tool in terms of usefulness (language learning potential) and ease of use (e.g. user-friendly voice commands). We believe that these positive perceptions might lead to an intention to continue to use ASR, suggesting that the technology has L2 pedagogical potential. [For the complete volume, "CALL and Professionalisation: Short Papers from EUROCALL 2021 (29th, Online, August 26-27, 2021)," see ED616972.]

This paper focuses on Partial and Synchronized Caption (PSC) as a tool to train L2 listening and introduces new features to facilitate speech-related difficulties. PSC is an intelligent caption that extensively processes the audio and transcript to detect and present difficult words or phrases for L2 learners. With the new features, learners can benefit from repetition and slowdowns of particular audio segments that are automatically labeled difficult. When encountering high speech rates, the system slows down the audio to the standard rate of speech. For disfluencies in speech (e.g. breached boundaries), the system generates the caption and repeats that video segment. In our experiments, intermediate L2 learners of English watched videos with different captions and functionalities, provided feedback on new PSC features, and took a series of tests. Smart repetition and slowdown components received positive learner feedback and led to significant improvement in L2 listening recognition. [For the complete volume, "CALL and Professionalisation: Short Papers from EUROCALL 2021 (29th, Online, August 26-27, 2021)," see ED616972.]

We explore how different components of an Automatic Short Answer Grading (ASAG) model affect the model's ability to generalize to questions outside of those used for training. For supervised automatic grading models, human ratings are primarily used as ground truth labels. Producing such ratings can be resource heavy, as subject matter experts spend vast amounts of time carefully rating a sample of responses. Further, it is often the case that multiple raters must come to a census before a final ground-truth rating is established. If ASAG models were developed that could generalize to out-of-sample questions, educators may be able to quickly add new questions to an auto-graded assessment without a continued manual rating process. For this project we explore various methods for producing vector representations of student responses including state-of-the-art representation methods such as Sentence-BERT as well as more traditional approaches including Word2Vec and Bag-of-words. We experiment with including previously untapped question-related information within the model input, such as the question text, question context text, scoring rubric information and a question-bundle identifier. The out-of-sample generalizability of the model is examined with both a leave-one-question-out and leave-one-bundle-out evaluation method and compared against a typical student-level cross validation. [For the full proceedings, see ED615472.]

Automated Writing Evaluation systems have been developed to help students improve their writing skills through the automated delivery of both summative and formative feedback. These systems have demonstrated strong potential in a variety of educational contexts; however, they remain limited in their personalization and scope. The purpose of the current study was to begin to address this gap by examining whether individual differences could be modeled in a source-based writing context. Undergraduate students (n=106) wrote essays in response to multiple sources and then completed an assessment of their vocabulary knowledge. Natural language processing tools were used to characterize the linguistic properties of the source-based essays at four levels: descriptive, lexical, syntax, and cohesion. Finally, machine learning models were used to predict students' vocabulary scores from these linguistic features. The models accounted for approximately 29% of the variance in vocabulary scores, suggesting that the linguistic features of source-based essays are reflective of individual differences in vocabulary knowledge. Overall, this work suggests that automated text analyses can help to understand the role of individual differences in the writing process, which may ultimately help to improve personalization in computer-based learning environments. [Paper published in: "LAK21, April 12-16, 2021, Irvine, CA, USA," ACM, 2021, pp. 620-625.]

Automatically detecting bugs in student program code is critical to enable formative feedback to help students pinpoint errors and resolve them. Deep learning models especially code2vec and ASTNN have shown great success for "large-scale" code classification. It is not clear, however, whether they can be effectively used for bug detection when the amount of labeled data is limited. In this work, we investigated the effectiveness of code2vec and ASTNN against classic machine learning models by varying the amount of labeled data from 1% up to 100%. With a few exceptions, the two deep learning models outperform the classic models. More interestingly, our results showed that when the amount of labeled data is small, code2vec is more effective, while ASTNN is more effective with more training data; for both code2vec and ASTNN, the more labeled data, the better. To further improve their effectiveness, we investigated the potential of semi-supervised learning which can leverage a large amount of unlabeled data to improve their performance. Our results showed that semi-supervised learning is indeed beneficial especially for ASTNN. [For the full proceedings, see ED615472.]

We describe a data mining pipeline to convert data from educational systems into knowledge component (KC) models. In contrast to other approaches, our approach employs and compares multiple model search methodologies (e.g., sparse factor analysis, covariance clustering) within a single pipeline. In this preliminary work, we describe our approach's results on two datasets when using 2 model search methodologies for inferring item or KCs relations (i.e., implied transfer). The first method uses item covariances which are clustered to determine related KCs, and the second method uses sparse factor analysis to derive the relationship matrix for clustering. We evaluate these methods on data from experimentally controlled practice of statistics items as well as data from the Andes physics system. We explain our plans to upgrade our pipeline to include additional methods of finding item relationships and creating domain models. We discuss advantages of improving the domain model that go beyond model fit, including the fact that models with clustered item KCs result in performance predictions transferring between KCs, enabling the learning system to be more adaptive and better able to track student knowledge. [This paper was published in: "Proceedings of the 14th International Conference on Educational Data Mining (EDM21)," International Educational Data Mining Society, 2021, pp. 672-76 (see ED615472).]