Search

This paper describes the development and evaluation of Interaction Competence Elicitor (ICE), a spoken dialog system (SDS) for the delivery of a paired oral discussion task in the context of language assessment. The purpose of ICE is to sustain a topic-specific conversation with a test taker in order to elicit discourse that can be later judged to assess the test taker's oral language ability, including interactional competence. The development of ICE is reported in detail to provide guidance for future developers of similar systems. The performance of ICE is evaluated on two aspects: (1) by analyzing system errors that occur at different stages in the natural language processing (NLP) pipeline in terms of both their preventability and their impact on the downstream stages of the pipeline; and (2) by analyzing questionnaire and semistructured interview data to establish the test takers' experience with the system. Findings suggest that ICE was robust in 90% of the dialog turns it produced, and test takers noted both positive and negative aspects of communicating with the system as opposed to a human interlocutor. We conclude that this prototype system lays important groundwork for the development and use of specialized SDSs in the assessment of oral communication, which includes interactional competence.

The construct of oral ability is multifaceted, but due to technological and practical constraints, the majority of computer-delivered speaking assessments are designed to measure only certain aspects of this ability. Most notably, interactional competence, which we define as the ability to actively structure dialogic speech in real time, is often not assessed. Innovations in technology, namely, computermediated video, make it possible for test takers in different locations to see and speak with others in real time and may make it achievable for computer-based tests to assess more aspects of oral communication, including interactional competence. This report gives the findings from a study that explored to what extent computer-mediated video, namely, Skype, could function in conjunction with a platform designed to present four innovative speaking tasks that could conceivably assess a broad construct of oral ability. The overarching goals of this project were twofold. First, we aimed to determine (a) the degree to which current computer video-mediated technology can be used effectively to deliver assessments remotely and (b) the extent to which participants felt that four specific tasks could assess speaking ability by means of this technology. The speaking tasks included giving short responses to an interlocutor's questions, summarizing a proposal, defending a position in a group discussion, and giving a prepared presentation and responding to questions from other participants. Two data collections were conducted: one with all 72 participants located in the United States and one with all 74 participants located in China. The findings provide preliminary evidence that the stability of computer video-mediated technology varies considerably, with technical disruptions being relatively few in the U.S. trial but very frequent in the China context. Moreover, the findings suggest that participants viewed the tasks as generally representing interactive speaking activities that they encounter in the oral language use domain, affording them the opportunity to demonstrate their oral abilities, and that these tasks can be effectively completed in a computer video-mediated environment when technology cooperates.

A challenge of large-scale oral communication assessments is to feasibly assess a broad construct that includes interactional competence. One possible approach in addressing this challenge is to use a spoken dialog system (SDS), with the computer acting as a peer to elicit a ratable speech sample. With this aim, an SDS was built and four trained human raters assessed the discourse elicited from 40 test takers that completed a paired oral task with both a human and a computer partner. The test takers were evaluated based on the analytic operational oral communication rating scales which included interactional competence, fluency, pronunciation, and grammar/vocabulary. Repeated-measures ANOVA indicated that fluency, pronunciation, and grammar and vocabulary were scored similarly across the two conditions, while interactional competence was scored substantially higher in the human partner condition. A g-study indicated that the computer partner was more reliable in assessing interactional competence, and rater questionnaire and interview data suggested the computer provided a more standardized assessment. Conversely, raters generally favored the human partner, in part because of its perceived authenticity and naturalness.

Various approaches help determine students' need for language support courses. Some programs use standardized language proficiency test scores, which are available from the admissions process; others use local placement tests, which are typically designed to be aligned with local language needs and ESL course curricula. Using local placement tests may be more defensible, but sufficient resources to develop, administer, and score valid assessments are often not available. Another approach is using standardized test scores to screen out students with a high likelihood of passing the local placement test, followed by local placement tests for students who need further evaluation. This approach has potential for placements with limited use of resources, but little research has been conducted on its effectiveness. This study aimed to determine the extent to which appropriate cut scores for a standardized test (TOEFL iBT Speaking) could be used to identify students who would have a high likelihood of passing the local English Placement Test of Oral Communication (EPT OC), which was designed to determine the need for taking an oral communication course. Teacher judgments of 136 students placed in the oral communication class indicated that only five did not need it. A TOEFL iBT score of 22 was found to be a reasonable cut score for exemption from the oral communication class.

Responding to the challenge of delivering valid assessments during the unprecedented times of the COVID-19 pandemic was certainly not easy. However, the efforts will undoubtedly lead to developments in the field of language assessment. This special issue closes with a brief summary of the commentaries it contains and raises some unanswered questions that need to be addressed moving forward.

The COVID-19 pandemic led to rapid innovation among language assessment developers who had to adapt their testing processes to address various restrictions from governments and institutions. This overview introduces some of the challenges and opportunities test developers grappled with during this period as well as introduces the six commentaries provided in the special issue. These commentaries were written by testing teams in various higher education contexts who were tasked with modifying their assessments in the rapidly evolving situation associated with the pandemic. The assessment contexts described in this special issue range from relatively small-scale local placement to international large-scale.

As our understanding of the construct of oral communication (OC) has evolved, so have the possibilities of computer technology undertaking the delivery of tests that measure this ability. It is paramount to understand to what extent such developments lead to accurate, comprehensive, and useful assessment of OC. In this paper, we discuss five models of technology-delivered OC assessment that have appeared in the past three decades. We evaluate these models in terms of how well their respective methods aid in assessing OC. To achieve this aim, we use a framework which takes into account a contemporary view of OC ability, including the call for incorporating English as a lingua franca (ELF) considerations into English language assessment. The evaluation of the five models suggests strengths and weaknesses of each that should be considered when determining which is used for a particular purpose.

The COVID-19 pandemic pushed the Iowa State University administration to impose face coverings and other safety regulations across the university campus. To comply with these safety measures, several changes to the English Placement Test of Oral Communication (EPT OC) had to be made. This paper discusses how the EPT OC was adapted to comply with the safety regulations mandated by the university administration and best ensure the safety of students and raters. It also describes the different considerations that were made prior to implementing these adaptations, and our thoughts while administering and rating the test. The paper concludes with what might be learned from administering the OC test during the pandemic.

A growing number of stakeholders argue for the use of second language (L2) speaking assessments that measure the ability to orally communicate in real time. A Web-based virtual environment (VE) that allows live voice communication among individuals may have potential for aiding in delivering such assessments. While off-the-shelf voice communication VEs, such as Second Life, have been used for language learning and to a much lesser extent, language assessment, a very limited number of such environments have been designed with the explicit aim of aiding in assessing L2 oral communication. Moreover, test developers who desire to build such a VE have little theoretical basis to guide them. This conceptual article fills this void by discussing some of the theoretical issues that were considered when the researchers built a VE to help deliver L2 oral communication assessments.

This study describes the development process and examines the construct validity of an English placement test of oral communication (EPT OC) developed at a Midwestern university in the United States. This test includes a one‐on‐one oral interview and paired discussion task, and test performance is judged on an analytic rating scale. A confirmatory factor analysis conducted on the ratings of 338 students who took the initial fully operational EPT OC revealed the test structure was represented by a correlated four‐factor model with interactional competence, fluency, pronunciation/comprehensibility, and grammar/vocabulary as sub‐constructs, in line with its targeted theoretical framework. Both tasks were effective in measuring the targeted sub‐constructs, but the sub‐constructs were not sufficiently distinct from each other to completely justify a four‐factor model. The findings provide some support for the proposed interpretations of the EPT OC test scores but indicate the need for some modifications to the assessment, such as more thorough rater training and/or revised rating scales to better distinguish the targeted sub‐constructs. [ABSTRACT FROM AUTHOR]

Concerns about the need for assessing multidialectal listening skills for global contexts are becoming increasingly prevalent. However, the inclusion of multiple accents on listening assessments may threaten test fairness because it is not practical to include every accent that may be encountered in the language use domain on these tests. Given this dilemma, this study aimed to determine the extent to which accent strength and familiarity affect comprehension and to provide a defensible direction for assessing multidialectal listening comprehension. A strength of accent scale was developed, and one US, four Australian, and four British English speakers of English were selected based on a judgment of their strength of accent. Next, TOEFL test takers (N = 21,726) were randomly assigned to listen to a common lecture given by one of the nine selected speakers, and respond to six comprehension items and a survey designed to assess their familiarity with various accents. The results suggest that strength of accent and familiarity do affect listening comprehension, and these factors affect comprehension even with quite light accents.

Multiple-choice formats remain a popular design for assessing listening comprehension, yet no consensus has been reached on how multiple-choice formats should be employed. Some researchers argue that test takers must be provided with a preview of the items prior to the input (Buck, 1995; Sherman, 1997); others argue that a preview may decrease the authenticity of the task by changing the way input is processed (Hughes, 2003). Using stratified random sampling techniques, more and less proficient Japanese university English learners (N = 206) were assigned one of three test conditions: preview of question stem and answer options (n = 67), preview of question stem only (n = 70), and no preview (n = 69). A two-way ANOVA, with test condition and listening proficiency level as independent variables and score on the multiple-choice listening test as the dependent variable, indicated that the amount of item preview affected test scores but did not affect high and low proficiency students' scores differently. Item-level analysis identified items that were harder or easier than expected for one or more of the conditions, and the researchers posit three possible sources for these unexpected findings: 1) frequency of options in the input, 2) location of item focus, and 3) presence of organizational markers.

This article reports on a study which aimed to determine the effect of strength of accent on listening comprehension of interactive lectures. Test takers (N = 21,726) listened to an interactive lecture given by one of nine speakers and responded to six comprehension items. The test taker responses were analyzed with the Rasch computer program WINSTEPS to investigate the relative difficulty of the items associated with the nine versions of the interactive lectures. Results indicated that comprehension of interactive lectures was diminished with quite light accents, as has been found with monologic lectures.

Studies that use structural equation modeling (SEM) techniques are increasingly encountered in the language assessment literature. This popularity has created the need for a set of guidelines that can indicate what should be included in a research report and make it possible for research consumers to judge the appropriateness of the interpretations made from a reported study. This article attempts to fill this void by providing a set of reporting guidelines appropriate for language assessment researchers.

The purpose of this study was to determine the extent to which performance on the TOEFL iBT speaking section is associated with other indicators of Japanese university students' abilities to communicate orally in an academic English environment and to determine which components of oral ability for these tasks are best assessed by TOEFL iBT. To achieve this aim, TOEFL iBT speaking scores were compared to performances on a group oral discussion, picture and graph description, and prepared oral presentation tasks, and their component scores of pronunciation, fluency, grammar/vocabulary, interactional competence, descriptive skill, delivery skill, and question answering. Participants were Japanese university students (N = 222), who were English majors in a Japanese university. Pearson product-moment correlations, corrected for attenuation, between scores on the speaking section of TOEFL iBT and the three university tasks indicated strong relationships between the TOEFL iBT speaking scores and the three university tasks and high or moderate correlations between the TOEFL iBT speaking scores and the components of oral ability. For the components of oral ability, pronunciation, fluency, and vocabulary/grammar were highly associated with TOEFL iBT speaking scores while interactional competence, descriptive skill, and delivery skill were moderately associated with TOEFL iBT speaking scores. The findings suggest that TOEFL iBT speaking scores are good overall indicators of academic oral ability and that they are better measures of pronunciation, fluency and vocabulary/grammar than they are of interactional competence, descriptive skill, and presentation delivery skill.

For decades, research has indicated that preadolescents' self-concept is comprised of subject-specific academic factors, a general academic factor, and several nonacademic factors. More recently, there have been some indications that academic self-concept might further be differentiated into competence and affect factors, at least for some preadolescent ethnic groups. This study examines the extent to which a 6-factor model of academic self-concept, in which competence and affect are distinct, is tenable for White, Hispanic, and Native American preadolescents. A total of 313 White, 331 Hispanic, and 224 Native American preadolescents from the southwestern United States were administered a modified version of the SDQI. A confirmatory factor analysis (CFA) indicated that the 6-factor model of academic self-concept was tenable. A multiple-groups CFA suggested that the 6-factor model was largely group invariant between White and Hispanic and White and Native American preadolescents. However, important differences in the structures were identified. (Contains 1 table, 1 figure, and 2 notes.)

Test takers should have a voice in testing practices (Mathew, 2004). However, when incorporating their input, systematic processes to ensure the validity of testing practices must be followed. Such processes allow for test development to be a more democratic process (Shohamy, 2001), without sacrificing the value of the resulting inferences made from the test scores. This article describes a case study that incorporated the desires of test takers to change the procedures of a group oral discussion test in a university English as a Foreign Language program. A study was designed to determine the extent to which the proposed changes would threaten the validity of the testing process. Specifically, the procedures for the group oral were altered to investigate the effect of interlocutor familiarity. Students were randomly assigned to class-familiar (n?=?146) and class-unfamiliar (n?=?159) groups to identify to what extent group familiarity affected test takers' scores in the four assessed categories: pronunciation, fluency, lexis and grammar, and communication skills. For the two groups, no statistically significant difference in scores was found, and score reliability estimates were similar. The implications of the findings are addressed in terms of recommendations for using stakeholder input in the assessment design process.

Computer-based testing (CBT) to assess second language ability has undergone remarkable development since Garret (1991) described its purpose as "the computerized administration of conventional tests" in "The Modern Language Journal." For instance, CBT has made possible the delivery of more authentic tests than traditional paper-and-pencil tests. CBT has also made it possible to more reliably, practically, and almost instantaneously score essays, oral speech samples, and other types of test responses. Unfortunately, however, due to a number of unresolved problems, CBT has failed to realize its anticipated potential. CBT has limited usability because systems that ensure test and score security have yet to be developed. Computer-adaptive testing, one of the most promising areas of CBT has not met expectations because of failure to solve problems with the statistical techniques on which it is based and the lack of resources necessary to implement it in most assessment contexts. In spite of these and other limitations, given the growing capability of CBT to deliver more authentic tests than paper-and-pencil, its use for assessing second language ability will undoubtedly continue to expand.

The second language group oral is a test of second language speaking proficiency, in which a group of three or more English language learners discuss an assigned topic without interaction with interlocutors. Concerns expressed about the extent to which test takers' personal characteristics affect the scores of others in the group have limited its attractiveness. This study investigates the degree to which assertive and non-assertive test takers' scores are affected by the levels of assertiveness of their group members. The sample of test takers was Japanese first year university students who were studying English in Japan. The students took the revised NEO-PI-R (Costa & McCrae, 1992; Shimanoka et al., 2002), a group oral test, and PhonePass SET-10 (Ordinate, 2004). Two separate MANCOVA analyses were conducted, one designed to determine the extent to which assertive test takers' scores are affected by the levels of assertiveness of group members (N = 112), and one designed to determine the extent to which non-assertive test takers' scores are affected by the levels of assertiveness of group members (N = 113). The analyses indicated that assertive test takers were assigned higher scores than expected when grouped with only non-assertive test takers and lower scores than expected when grouped with only assertive test takers, while the study failed to find an effect for grouping based on assertiveness for non-assertive test takers' scores. The findings of the study suggest that when the group oral is used, rater-training sessions should include guidance on how to evaluate a test taker in the context of the group in which the test taker is assessed and assign scores that are not based on a comparison of proficiencies of group members. (Contains 2 tables, 4 figures and 4 footnotes.)

Over the past decade, listening comprehension tests have been converting to computer-based tests that include visual input. However, little research is available to suggest how test takers engage with different types of visuals on such tests. The present study compared a series of still images to video in academic computer-based tests to determine how test takers engage with these two test modes. The study, which employed observations, retrospective reports and interviews, used data from university-level non-native speakers of English. The findings suggest that test takers engage differently with these two modes of delivery. Specifically, while test takers engaged minimally and similarly with the still images, there was wide variation in the ways and degree to which they engaged with the video stimulus. Implications of the study are that computer-based tests of listening comprehension could include still images with only minimally altering the construct that is measured by audio-only listening tests, but the utilization of video in such computer-based tests may require a rethinking of the listening construct. The following are appended: (1) Questions asked during input stoppages; (2) Interview Questions after Each Test; and (3) Interview Questions at the End of Both Tests. (Contains 2 tables and 1 figure.)

When testing English language learners (ELLs) in subject matter areas, construct irrelevant variance could result from English, the language in which the test is presented. Differential item functioning (DIF) techniques have been used to determine if items are operating differently for population subgroups and might therefore be appropriate for such an investigation. In the study presented here, ELL and non-ELL 8th-grade students' scores on National Assessment of Educational Progress (NAEP) math word problems were compared. After the non-ELLs were found to outperform the ELLs, two independent techniques for detecting DIF--an item response theory approach and the Mantel-Haenszel approach--were employed to determine if a plausible explanation for the score difference could be attributed to DIF against ELLs. The results showed that only 1 of 10 original NAEP items and none of 10 linguistically simplified items were found to exhibit DIF against ELLs, suggesting that the score difference between the groups could not be attributed to DIF against ELLs. (Contains 4 tables.)

FACETS many-facet Rasch analysis software was utilized to look at two consecutive administrations of a large-scale second language oral assessment in the form of a peer group discussion task with Japanese English-major university students. Results are discussed. (Author/VWL)