Search

Automated reading error detection has attracted a lot of interest in the area of computer-assisted language learning and auto-mated reading tutors. This paper presents preliminary experimental results on automatic detection of word-level reading errors in non-native speech. A state-of-the-art large vocabulary automatic speech recognition (ASR) system is developed to transcribe non-native speech, with performance comparable to humans in transcribing non-native read speech data. With this ASR system, we investigate the feasibility of detecting substitution, insertion and deletion errors from ASR decoding results on non-native read speech. Experimental results show that the performance of detecting substitution and insertion errors are on the low side. Several possible reasons for causing such results are discussed in this paper. Common types of reading errors occurring in non-native read speech and those that are difficult to be detected are analyzed for future investigation.

The COVID-19 pandemic has led to a dramatic increase in the use of face masks worldwide Face coverings can affect both acoustic properties of the signal as well as speech patterns and have unintended effects if the person wearing the mask attempts to use speech processing technologies In this paper we explore the impact of wearing face masks on the automated assessment of English language proficiency We use a dataset from a large-scale speaking test for which test-takers were required to wear face masks during the test administration, and we compare it to a matched control sample of test-takers who took the same test before the mask requirements were put in place We find that the two samples differ across a range of acoustic measures and also show a small but significant difference in speech patterns However, these differences do not lead to differences in human or automated scores of English language proficiency Several measures of bias showed no differences in scores between the two groups Copyright © 2020 ISCA

This study is an evaluation of the performance of automated speech scoring for speakers with documented or suspected speech impairments. Given that the use of automated scoring of open-ended spoken responses is relatively nascent and there is little research to date that includes test takers with disabilities, this small exploratory study focuses on one type of scoring technology, automatic speech scoring (the SpeechRaterSM automated scoring engine); one type of assessment, spontaneous spoken English by nonnative adults (six TOEFL iBT® test speaking items per test taker); and one category of disability, speech impairments. The results show discrepancies between human and SpeechRater scores for speakers with documented speech or hearing impairments who receive accommodations and for speakers whose responses were deferred to the scoring leader by human raters because the responses exhibited signs of a speech impairment. SpeechRater scores for these studied groups tended to be higher than the human scores. Based on a smaller subsample, the word error rate was higher for these groups relative to the control group, suggesting that the automatic speech recognition system contributed to the discrepancies between SpeechRater and human scores.

This report presents an overview of the SpeechRaterSM automated scoring engine model building and evaluation process for several item types with a focus on a low-English-proficiency test-taker population. We discuss each stage of speech scoring, including automatic speech recognition, filtering models for nonscorable responses, and scoring model building and evaluation and compare how the performance at each step differs between different item types. We conclude by discussing the effect of item type on automated scoring performance. We also give recommendations about what considerations should be taken into account when developing tests for low-proficiency English speakers to obtain reliable scores from an automatic scoring engine.

For the last 5 years, we have developed and maintained RSMTool – an open-source tool for evaluating NLP systems that automatically score written and spoken responses. RSMTool is designed to be cross-disciplinary, borrowing heavily from NLP, machine learning, and educational measurement. Its cross-disciplinary nature has required us to learn a user-centered development approach in terms of both design and implementation. We share some of these lessons in this paper.