Search

Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long and ranged in proficiency levels. The raters scored each sample on fluency, vocabulary, grammar, and comprehensibility using 7-point semantic differential scales. Nonverbal behavior was extracted using an automated machine learning software called iMotions, and data was analyzed with ordinal mixed effects regression. Results showed that attentional variance predicted fluency, vocabulary, and grammar scores, but only when accounting for proficiency. Higher standard deviations of attention corresponded with lower scores for the lower-proficiency group, but not the mid/higher-proficiency group. Comprehensibility scores were only predicted by mean valence when proficiency was an interaction term. Higher mean valence, or positive emotional behavior, corresponded with higher scores in the lower-proficiency group, but not the mid/higher-proficiency group. Effect sizes for these predictors were quite small, with small amounts of variance explained. These results have implications for construct representation and test fairness. [ABSTRACT FROM AUTHOR]

Integrated group discussion (IGD) tasks represent a new type of integrated speaking task that requires students to communicate with their peers after comprehending and using provided source materials. With its potential to enhance effective communication and information literacy, IGD stands at the core of language education. Although the IGD task type has become increasingly popular, its construct is still unexplored. In this study, 124 Hong Kong undergraduate students completed an IGD task and a group discussion (GD) task (i.e., without source materials) in Putonghua. Four factors, speech and expression, content with source use, interaction, and language usage, were extracted to represent IGD performance. Two indicators of GD performance (linguistic expression and engagement in interaction) significantly predicted IGD performance, explaining 21.9 % of its total variance. The large proportion of unexplained IGD performance variance suggests that IGD tasks may be worth implementing. The theoretical and practical implications of the findings are discussed in details. [ABSTRACT FROM AUTHOR]

This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the many-facet Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice. [ABSTRACT FROM AUTHOR]

This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the many-facet Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence rating behaviour. Finally, we elaborate on implications for both research and practice. [ABSTRACT FROM AUTHOR]

Using a multidimensional analysis (MDA), this study aimed to increase the understanding of international teaching assistants’ (ITAs’) instructional language use by analyzing co-occurring linguistic features during their mock-teaching presentations in ITA assessment. With this purpose, the study examined the effects of ITAs’ proficiency levels, disciplines, and first-language backgrounds on their discourse patterns. The data comprised a corpus of 165 prospective ITAs’ mock-teaching presentations in ITA assessment. Based on the two dimensions of the twenty-six linguistic features found prevalent in university classroom talk (Csomay, 2005), an additive MDA was performed. Overall, the mock-teaching presentations of more proficient ITAs involved more contextual, directive, and personalized framing discourse features, with interaction effects for the ITAs’ disciplines and first-language backgrounds. The study discusses the implications for ITA training and assessment and directions for future studies. This article was published open access under a CC BY licence: https://creativecommons.org/licences/by/4.0.

The teaching and assessment of interactive speaking skills is a key aim in many English-language programs, and the right assessment rubric is a key component of any effective course. There is no one-size-fits-all approach and rubrics need to be validated, feedback collected, and revisions made. This paper reports on a small-scale, exploratory study undertaken to develop and improve a rubric for assessing interactive discussion skills. Utilizing many-facet Rasch measurement (MFRM), the paper reports on an analysis of the rubric originally used on the course. Based on these findings, the rubric was revised and subject to a second round of analysis. The findings indicate that the revisions led to considerably improved rubric function, due primarily to a reduced number of scale points and more clearly defined rubric categories. Finally, the paper suggests a number of recommendations for rubric crea- tion that can be applied to a range of assessment contexts. [ABSTRACT FROM AUTHOR]

Learner and L2 user corpora are increasingly valued in language testing and assessment as they can inform test design, revision, and validation. This paper illustrates the benefits of using an L2 corpus to explore patterns of epistemic stance marking in computer-mediated speaking tests with no live human interlocutor. Drawing on the British Council-Lancaster Aptis Corpus – comprising over 630,000 words of L2 speech – we explored the frequency of epistemic stance markers (adverbial, adjectival and verbal) across proficiency levels and speaking task types. The analysis revealed that epistemic stance was prevalent in test-taker discourse and that frequency was influenced by L2 proficiency and task type. The findings demonstrate that computer-mediated speaking tests can elicit expressions of epistemic stance in a comparable way to tests which involve human-human interaction. Implications are drawn for examiner training, test preparation, and an enriched understanding of the elements of pragmatic competence that can be elicited in computer-mediated speaking assessment. [ABSTRACT FROM AUTHOR]

Speaking assessment is believed to be difficult in its expansion and execution. Thus, it is a challenge for teachers in preparing students for speaking tests. This study's purpose was to identify whether topic familiarity affects speaking performance among students who were preparing to sit for the high-stakes Malaysian University English Test speaking test. The study aimed to investigate the validity and reliability of the speaking assessment measures and, subsequently, to examine the speaking tests' item difficulty measures differences according to topic familiarity level. Data were collected from 40 non-native speakers of English among Malaysian Form Six pre-university students who were preparing for their MUET test. The researcher conducted two practice speaking tests, which used retired papers of the MUET CEFR-aligned speaking tests, to the 40 participants who were grouped into 10 speaking test groups. The practice speaking tests were video and audio-recorded. Topic familiarity was measured using self-report questionnaires. In the second phase, each of the seven appointed raters scored all 40 students' responses in two speaking tests consisting of two speaking tasks assessed across four criteria: task fulfillment, language, communicative ability, and group discussion. Many-facet Rasch measurement results revealed significant differences translating to significant influence of topic familiarity on speaking performance. The present study's results not only confirmed the significance of topic familiarity in the preparation for speaking assessments, but also highlighted the need for formal teaching on potential topic themes that are commonly encountered in such assessments, particularly those that are at the post-secondary level. These findings imply importance in designing speaking tests taking into account test-taker topic familiarity. [ABSTRACT FROM AUTHOR]

The contributions to this Special Issue employ conversation analysis to illustrate how detailed analysis of language use can lead to the identification of assessable features of second/foreign language Interactional Competence (L2 IC) and the development of institutional testing instruments and practices. L2 IC has been the focus of much research at the intersection of social interaction and second language acquisition. It has also been treated as a construct in the field of language assessment. However, scholars in each research branch have just begun to collaborate systematically. This Special Issue furthers this collaboration, connecting research on L2 IC in diverse learning contexts with practical questions regarding the assessment of individual learners. It adopts a dialogic 'full paper–commenting paper' structure: Four empirical papers are each paired with invited commentaries that provide critical discussion and a complementary view of the topics the full papers address. The final discussion papers take a broader perspective on the complex nature of L2 IC and assessment and propose ways to productively move forward. Besides introducing the notion of L2 IC and each individual contribution, this introductory article explains the rationale behind the Special Issue in relation to current research. [ABSTRACT FROM AUTHOR]

This classroom-based project included the development of clear and concise instructions, an analytic rubric as well as practice and testing materials to assess students’ speaking abilities. The speaking tasks consisted of three parts: reading aloud, describing a picture, and giving an opinion. As an extension of an ongoing departmental project originally focusing on second-year pharmacy students, the researchers have incorporated eleven first-year student groups from all departments. Six of the groups were offered the module face-to-face, while the other five received instruction, practice tasks, and feedback online. The aim of the current project was to improve confidence and production of logical arguments by implementing a formative speaking assessment module that provides students with opportunities for advancement through succinct instruction, insight into strengths and weaknesses, self-evaluation and instructor feedback. Research questions included: 1) To what extent did the use of the assessment tool improve first-year students’ comprehensibility in English? 2) To what extent did classroom instruction help improve student confidence? This study took a mixed-method approach where data from student grades and surveys were collected and analysed. To obtain a baseline of students’ speaking abilities prior to any formal instruction and gain insight into the assessment process, one class was chosen by each instructor and asked to record themselves describing a picture. A comparison of the baseline assessment and finals resulted in improvement following the introduction of rubrics and explicit instruction. The remaining two aspects also showed a distinct rise both in grades and comprehensibility between midterms and finals. Survey results showed that across departments, students’ confidence in their abilities to express themselves in English increased over the semester. Future research will focus on redesigning and evaluating the speaking assessment rubrics both for first and second years.

Given the prolonged pandemic situation that forced schools to run online in both synchronous and asynchronous modes has impacted some changes in either teaching practice or conducting the assessment. With the limited timetable provided to each school subject, there is a significant adjustment that the teachers should take to make the teaching and learning meaningful, not to mention in assessing students’ learning. In the English as a Foreign Language (EFL) setting, assessing speaking is another challenge the teachers encounter specifically during online teaching. Consequently, English teachers are required to have specific strategies to manage this assessment still. Electronic portfolio-based assessment has been shown to serve both as an effective learning tool and assessment tool. Hence, this study was aimed to investigate the use of the Flipgrid as the electronic portfolio in assessing speaking. The qualitative design was applied in this study with students' scores, teacher’s field notes, and questionnaires as the instruments to collect data. This study was expected to describe the washback of the Flipgrid as electronic portfolios in speaking assessment. BAHASA INDONESIA ABSTRAK Situasi pandemi yang berkelanjutan telah memaksa setiap sekolah untuk melaksanakan pembelajaran daring baik secara synchronous maupun asynchronous. Hal ini berdampak pada berubahnya praktik mengajar dan penilaian dalam pembelajaran. Dengan jadwal pembelajaran yang terbatas untuk setiap mata pelajaran, guru harus dapat melakukan penyesuaian yang signifikan untuk membuat proses kegiatan belajar mengajar menjadi berarti, begitu pula dengan penilaian hasil belajar siswa. Pada pembelajaran bahasa Inggris sebagai bahasa asing, penilaian keterampilan berbicara menjadi tantangan tersendiri bagi para guru khususnya pada saat pembelajaran daring. Sebagai konsekuensinya, setiap guru bahasa Inggris harus memiliki strategi tertentu untuk dapat mengelola penilaian dalam proses pembelajaran. Penilaian berbasis portofolio elektronik telah menunjukkan bahwa penilaian ini digunakan sebagai alat pembelajaran dan penilaian yang efektif. Oleh karena itu, penelitian ini bertujuan untuk menelaah penggunaan Flipgrid sebagai portofolio elektronik pada penilaian keterampilan berbicara dalam pembelajaran Bahasa Inggris. Pendekatan yang digunakan dalam penelitian ini adalah pendekatan kualitatif dengan menggunakan nilai siswa, catatan lapangan guru, dan kuesioner sebagai instrumen untuk mengumpulkan data. Penelitian ini diharapkan dapat mendeskripsikan dampak penggunaan Flipgrid sebagai portofolio elektronik pada penilaian keterampilan berbicara dalam pembelajaran Bahasa Inggris.

Assessment of speaking skills differs from other language skills. The methods of conducting speaking skills are included in the English Assessment Module developed by the researchers from the National University of Malaysia (UKM). The module is designed to cater postgraduates (adult learners) through Online Distance Learning (ODL) program. ODL has become essential in education setting for adult learners as they could manage their learning whilst juggling their career and personal commitments. The covid-19 pandemic has become the starting point for postgraduates to conduct the course remotely according to their preferences of learning. The revolution of education allows adult learners to pursue their Masters. The module includes various types of English language assessments in the context of speaking English as a Second Language (ESL). Hence, eight adult learners were interviewed to give their perceptions towards this newly designed module. This qualitative research method employed a set of semi-structured interview questions and open-ended questions via Google Meet. The findings showed that English Assessment Module; Speaking Assessment is well equipped with comprehensive information and guidelines on how the speaking assessments should be conducted in ESL context. Generally, the findings indicate that the English Assessment Module; Speaking Assessment is suitable for ODL postgraduates (adult learners). This paper also includes the participants' suggestions to further improve the module. The findings could benefit the online distance course designers in designing better module for adult learners and enable the learners to learn effectively at their own convenience. [ABSTRACT FROM AUTHOR]

In response to the calls for making key stakeholders' perspectives relevant in the test validation process, the study discussed in this report sought test‐taker feedback as part of collecting validity evidence and supporting the ongoing field testing efforts of the new TOEFL ITP® Speaking section. Specifically, I aimed to investigate the extent to which test takers' perceptions of the newly proposed ITP test tasks are in agreement with the intended characteristics and qualities of the tasks. In addition, I opted to gather insights into whether the speaking tasks are perceived as acceptable by its prospective test takers and also to identify any unwarranted challenges posed for them in completing the tasks. A two‐part questionnaire was thus administered during field testing of the new speaking section, and resulting data were analyzed both quantitatively and qualitatively. Findings emerging from the questionnaire data suggest that test‐taker perceptions can be used to provide support to corroborate the intended (or hypothesized) properties of the tasks, while pointing to several areas for further monitoring and improvement. [ABSTRACT FROM AUTHOR]

As one of the fundamental skills of the 21st century, critical thinking (CT) is a topic of considerable interest within the domain of assessing English for academic purposes (EAP). Recent literature on EAP instruction and assessment indicates that EAP has evolved beyond a strict focus on language improvement to encompass discourse competence and broader academic literacy development, with CT playing an important role in academic communication. A fundamental consideration in educational assessment is the construct that defines the knowledge, skills, or abilities to be assessed. However, CT has not received due attention in the research literature on EAP speaking assessment, thus raising a legitimate concern about its underrepresentation in the academic speaking construct. This study draws on Macqueen's (2022) distinction between theoretical, stated, perceived, and operationalized assessment constructs, with an aim to describe the EAP speaking assessment construct by including the concept of CT. By establishing alignment between what the literature indicates (theoretical), what assessment requires (stated), and how people understand assessment (perceived), a CT assessment framework is proposed to reflect how test takers experience it (operationalized). Our study contributes to a more nuanced conceptualization of CT in the context of EAP speaking, with implications for EAP speaking test development. [ABSTRACT FROM AUTHOR]

Abstract This study explored two assessment approaches to oral performances: analytical complexity, accuracy, and fluency (CAF) indices and human raters’ evaluations. CAF indices are frequently used in second-language speaking (L2) research; however, because tasks are communicative and goal-oriented, the degree to which students achieve such communicative goals must also be included. By incorporating human ratings of monologue organization and perceived CAF into speaking assessments, researchers can better understand the relationship between the analytical CAF indices and human ratings of a monologue task. The participants consisted of 48 English as a Foreign Language (EFL) students in a Japanese university. Their oral performances of 2-min opinion-based monologues were audio-recorded and then transcribed and analyzed using CAF measures. In addition, 11 human raters evaluated the same recordings in terms of the following criteria: topic organization, complexity, accuracy, and fluency. These ratings were then analyzed using the many-facet Rasch measurement (MFRM). Multiple linear regression results showed that fluency accounted for a significant amount of the human ratings, but other measures (lexis, complexity, accuracy) explained only a small portion of the variance. This study concluded with implications regarding L2 testing in speaking assessments.

In speaking assessment, many factors such as characteristics of test-takers, test tasks, rating bias, etc. may affect the speaking performance of test-takers. Besides, the stability of raters’ rating of a speaking test might pose a threat to its reliability, validity, and fairness, which calls for longitudinal construct validation of the speaking test. This study explores the construct validity of PRETCO-Oral through analysis of data retrieved from various sources, including longitudinal ratings of performances of testtakers across four occasions, and perceptions of the construct of PRETCO-Oral from both raters and test-takers. The results indicate that raters’ ratings keep stable and the PRETCO-Oral assessment is equipped with longitudinal reliability; tasks of Interpretation and Presentation represent a large amount of variance of the construct, while those of Reading Aloud and Question and Answer seem to be construct-underrepresented, as evidenced via analyzing the data collected from perceptions of raters and test-takers upon the test construct. Finally, factors that threaten the construct representation are also discussed. [ABSTRACT FROM AUTHOR]

Given the increasing awareness of oral communication in this era of globalized collaborative learning trends, there is an imminent need to inform language educators of ways in which the under-researched oral interactional strategies are related to first language (L1) teaching. However, no consensus has yet been reached on the relationship between interactional strategy use and oral language proficiency. This study investigates the effect of oral interactional strategy use on group discussion performance in L1 Chinese for Primary 5 students (N = 140) in Hong Kong. Based on ANOVA and regression analyses of the data on group discussion performance, five strategies have been identified: expressing actively, asking for opinion, expressing attitude, giving clarification and non-verbal language. They all significantly predicted students' group discussion performance, with overall strategies explaining 55.5% of total variation of the performance, where higher-proficiency students tended to use more strategies that enable comprehension and elaboration in the group discussions. The patterns of strategy use among students with different levels of discussion performance have also been identified. Implications of the findings are discussed with reference to the roles individuals play in the overall performance of group discussion. [ABSTRACT FROM AUTHOR]

Second-language (L2) testing researchers have explored the relationship between speakers' overall speaking ability, reflected by holistic scores, and the speakers' performance on speaking subcomponents, reflected by analytic scores (e.g., McNamara, 1990; Sato, 2011). These research studies have advanced applied linguists' understanding of how raters view the components of effective speaking skills, but the authors of the studies either used analytic composite scores, instead of true holistic ratings, or ran regression analyses with highly correlated subscores, which is problematic. To address these issues, 10 experienced ITA raters rated the speaking of 127 international teaching assistant (ITA) candidates using a four-component analytic rubric. In addition, holistic ratings were provided for the 127 test takers from a separate (earlier) scoring by two experienced ITA raters. The two types of scores differentiated examinees in similar ways. The variability observed in students' holistic scores was reflected in their analytic scores. However, among the four analytic subscales, examinees' scores on Lexical and Grammatical Competence had the greatest differentiating power. Its scores indicated with a high level of accuracy who passed the test and who did not. The paper discusses the components contributing to ITAs' L2 oral speaking proficiency, and reviews pedagogical implications. [ABSTRACT FROM AUTHOR]

This article describes Schema Association Training (SAT), an enrichment programme designed to develop students' confidence and ability to sustain discussions during speaking assessments. The author observed that many students had difficulties introducing relevant ideas to prolong discussions during Spoken Interaction, an assessment component of the Singapore-Cambridge GCE O-Level English language oral communication exam. Notably, there appears to be a tendency for students to compartmentalize the content knowledge they possess, often resulting in missed opportunities to surface relevant ideas that could possibly add depth to discussions. This raises a need for classroom preparatory work to move towards encouraging learners to make meaningful connections between related topics. SAT was thus implemented to achieve this. During SAT, students first go through an individual Spoken Interaction exercise which is audio recorded. This audio recording is subsequently utilized during an individual reflective conversation with the teacher. Finally, students consolidate their learning via an online learning module. This article describes the steps involved in carrying out SAT, and also presents preliminary findings that suggest the potential benefits of SAT in helping learners elaborate better during speaking assessments. [ABSTRACT FROM AUTHOR]

This paper reports an exploratory and descriptive qualitative study on speaking assessment approaches in a teacher education program at a Colombian university. The study aimed to explore how four in-service English language teachers approach the assessment of students’ speaking skill. The data were gathered through classroom observations, interviews, and documentary analysis. Results revealed teachers’ preference for summative assessment practices to determine students’ progress regarding speaking. As a conclusion, teacher professional development in terms of language assessment may be seen as an alternative to develop significant assessment processes where students, teachers, and the institution can be benefited.

In 2016, Euroexam International launched a complex and ambitious project to redesign the rating process and practice of its subjectively scored papers. As part of the project, a series of level-specific performance checklists were developed for speaking and writing. This paper reports on the standard setting and internal validation procedures for B2 speaking in English as a foreign language (EFL) as executed in Spring, 2020. The participants of the study were 8 EFL teachers and oral examiners from international backgrounds with varying degrees of expertise in language testing. The method adopted for standard setting was the Body of Work method. The participants were allocated into either of two groups working independently on a random sample of 32 audio-recorded live speaking paper performances in a counterbalanced design. The study found that (a) a level-specific 30-item checklist for EFL speaking at B2 can adequately replace the operational rating scales for pass/fail decisions; (b) the granular nature of the checklist renders it more capable of covering the targeted content areas; and (c) the explicit checklist statements support fairness, transparency and accountability.

Corpus studies have highlighted the role of multiword units in naturally occurring language. Speech theories, too, have underlined the linkage between such formulaic sequences- collocations in particular- and speech production. Few studies, however, have focused their attention on examining collocations in speaking assessment, especially in high-stakes tests. This study investigated the most frequently used collocational patterns and their discriminating power across three groups of participants with band scores 6, 7, and 8 respectively. To collect data, a corpus entailing 60 IELTS speaking samples, 20 samples from each band score, and approximately 110,000 words was gleaned. The results revealed that L1 (adjective + noun) and L7 (verb + noun) were the most frequently used types of lexical collocations, and G8 (verb + preposition) was the most frequently used grammatical collocation. The study also found that L1(adjective + verb), L5 (noun of noun), L8 (phrasal verb and adverb), L9 (noun and phrasal verb), and L10 (phrasal verb and noun) were the five types of lexical collocations with the most discriminating power across the band scores. Given the grammatical collocations, G4 (preposition + noun) and G5 (adjective + preposition) had the power to differentiate across the three band scores.

This study explored two assessment approaches to oral performances: analytical complexity, accuracy, and fluency (CAF) indices and human raters' evaluations. CAF indices are frequently used in second-language speaking (L2) research; however, because tasks are communicative and goal-oriented, the degree to which students achieve such communicative goals must also be included. By incorporating human ratings of monologue organization and perceived CAF into speaking assessments, researchers can better understand the relationship between the analytical CAF indices and human ratings of a monologue task. The participants consisted of 48 English as a Foreign Language (EFL) students in a Japanese university. Their oral performances of 2-min opinion-based monologues were audio-recorded and then transcribed and analyzed using CAF measures. In addition, 11 human raters evaluated the same recordings in terms of the following criteria: topic organization, complexity, accuracy, and fluency. These ratings were then analyzed using the many-facet Rasch measurement (MFRM). Multiple linear regression results showed that fluency accounted for a significant amount of the human ratings, but other measures (lexis, complexity, accuracy) explained only a small portion of the variance. This study concluded with implications regarding L2 testing in speaking assessments. [ABSTRACT FROM AUTHOR]

This research presents the development of an online speaking test of English for students at the end of primary and beginning of secondary school education in state schools in Uruguay. Following the success of the Plan Ceibal one computer-tablet per child initiative, there was a drive to further utilize technology to improve the language ability of students, particularly in speaking, where the majority of students are at CEFR levels pre-A1 and A1. The national concern over a lack of spoken communicative skills amongst students led to a decision to develop a new speaking test, specifically tailored to local needs. This paper provides an overview of the speaking test development and validation project designed with the following objectives in mind: to establish, track, and report annually learners' achievements against the Common European Framework of Reference for Languages (CEFR) targeting CEFR levels pre-A1 to A2, to inform teaching and learning, and to promote speaking practice in classrooms. Results of a three-phase mixed-methods study involving small-scale and large-scale trials with learners and examiners as well as a CEFR-linking exercise with expert panelists will be reported. Different sources of evidence will be brought together to build a validity argument for the test. The paper will also focus on some of the challenges involved in assessing young learners and discuss how design decisions, local knowledge and expertise, and technological innovations can be used to address such challenges with implications for other similar test development projects. [ABSTRACT FROM AUTHOR]

Amid a global pandemic, while schools in many parts of the world were closed to adhere to quarantine orders, schools in Japan resumed face-to-face classes after only a month of closure with strict adherence to COVID-19 guidelines and standard operating procedures (SOP). This study examined how speaking assessments were administered face-to-face for Grade 5 and 6 elementary school students prior to and after introducing the Common European Framework of Reference (CEFR) and amid a global pandemic between April to October 2020. The paper also reports the challenges and strategies employed in carrying out the speaking assessments following the CEFR while adhering to the SOP. The study employed a qualitative research method that utilised semi-structured interviews to elicit information from four teachers who taught in eight schools within Niigata City, Japan. Findings suggest that prior to the implementation of CEFR, not all teachers carried out speaking assessments. However, the implementation of CEFR emphasised the need to teach speaking and carry out speaking assessments. The CEFR also served as guidance for the teachers in preparing the assessment scoring rubrics. The results also showed that the speaking assessments were implemented individually instead of in groups before the pandemic and the presence of the masks, which increased the student's anxiety and affected their performance. However, the teachers employed various strategies to overcome the challenges by modifying the assessment tasks and utilising web conferencing technology. [ABSTRACT FROM AUTHOR]

The present study explores the test examiners' perspectives on the role and qualitative aspects of the current localized speaking assessment framework used in Vietnam. A case study with two experienced test examiner-cum-English lecturers was conducted. Inductive content analysis was used to analyze the qualitative data findings obtained from individual semi-structured interviews. Drawbacks, merits, and standardization issues of the current localized speaking assessment frameworks, i.e., the Vietnamese Standardized Test of English Proficiency (VSTEP), were also discussed, especially in comparison to internationally recognized examinations and frameworks such as the International English Language Testing System (IELTS) and the Certificate in Advanced English (CAE) as well as the Common European Framework of Reference for Languages (CEFR). The study informed both English educators and policymakers to improve localized speaking assessment to suit the local teaching needs while still meeting the requirements of widely accepted international proficiency tests. [ABSTRACT FROM AUTHOR]

The elicited imitation task (EIT), in which language learners listen to a series of spoken sentences and repeat each one verbatim, is a commonly used measure of language proficiency in second language acquisition research. The TOEFL® Essentials™ test includes an EIT as a holistic measure of speaking proficiency, referred to as the "Listen and Repeat" task type. In this report, we describe the design considerations that informed the development of the EIT for TOEFL Essentials. We also report the results of a series of investigations conducted during the prototyping and pilot phases of test development, which were undertaken with the goal of confirming task design specifications, evaluating scoring performance, and obtaining initial validity evidence to support score interpretation and use of the EIT in the TOEFL Essentials test. We found that task design variables generally performed as expected. The length of input sentence was strongly associated with performance (Pearson r =.88), consistent with the construct measured by the EIT, while other task variables not directly related to the EIT construct did not impact performance (e.g., graphics, speaker accent, and response time). Scorers drawn from TOEFL iBT test raters were able to score responses consistently with over 98% exact or adjacent interrater agreement on a 6‐point scale, and scores on the pilot version of the EIT were highly reliable (Cronbach's α =.93 on the 15‐item pilot version). Correlations between EIT scores and other measures were generally as expected: Correlations with other speaking tasks were high (.78–.84) and slightly to somewhat lower for other language measures (.73 for writing,.68 for listening, and.57 for reading). Correlation with an independent measure of holistic language proficiency (C‐test) was moderately high (.69), as expected. We discuss the study findings in terms of the TOEFL Essentials test validity argument and point out limitations to the current results along with future research needs. Overall, we believe that the findings provide initial support to warrant the use of the EIT as operationalized in the TOEFL Essentials test. Report Number: RR‐21‐24 [ABSTRACT FROM AUTHOR]

Abstract Two assumptions of speaking proficiency tests are that the speech produced is spontaneous and the the scores on those tests predict what examinees can do in real-world communicative situations. Therefore, when examinees memorize scripts for their oral responses, the validity of the score interpretation is threatened. While the American Council on the Teaching of Foreign Languages (ACTFL) Proficiency Guidelines identify rehearsed content as a major hindrance to interviewees being rated above Novice High, many examinees still prepare for speaking tests by memorizing and rehearsing scripts hoping these "performances" are awarded higher scores. To investigate this phenomenon, researchers screened 300 previously rated Oral Proficiency Interview-computer (OPIc) tests and found 39 examinees who had at least one response that had been tagged as rehearsed. Each examinee’s responses were then transcribed, and the spontaneous and rehearsed tasks were compared. Temporal fluency articulation rates differed significantly between the spontaneous and rehearsed segments; however, the strongest evidence of memorization lay in the transcriptions and the patterns that emerged within and across interviews. Test developers, therefore, need to be vigilant in creating scoring guidelines for rehearsed content.

Human raters and machine scoring systems potentially have complementary strengths in evaluating language ability; specifically, it has been suggested that automated systems might be used to make consistent measurements of specific linguistic phenomena, whilst humans evaluate more global aspects of performance. We report on an empirical study that explored the possibility of combining human and machine scores using responses from the speaking section of the TOEFL iBT® test. Human raters awarded scores for three sub-constructs: delivery, language use and topic development. The SpeechRaterSM automated scoring system produced scores for delivery and language use. Composite scores computed from three different combinations of human and automated analytic scores were equally or more reliable than human holistic scores, probably due to the inclusion of multiple observations in composite scores. However, composite scores calculated solely from human analytic scores showed the highest reliability and reliability steadily decreased as more machine scores replaced human scores. [ABSTRACT FROM AUTHOR]

Copyright of Profile: Issues in Teachers' Professional Development is the property of Universidad Nacional de Colombia, Facultad de Ciencias Humanas and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Speaking can be said as one of the four basic skills in English. In every English textbook, there must be a speaking assessment. This research aims to analyze the five types of speaking assessments in the textbook “When English Rings a Bell for Grade VII”. The textbook analysis in this study considered five types in speaking assessment there are: 1. Imitative speaking tasks, 2. Intensive speaking tasks, 3. Responsive speaking tasks, 4. Interactive speaking tasks and, 5. Extensive speaking tasks. This study includes the design of descriptive qualitative. This study using a method of content analysis. The collection of the data was done through procedures: 1) looking, 2) observing, 3)analyzing the data based on Brown’s theory. The result of the study there are five types of speaking tasks in that textbook, but it has a different rate of the types. The textbook not only has a speaking task, it also has another skill of English there are reading, listening, and writing.

This paper provides a narrative review of empirical research on the assessment of speaking proficiency published in selected journals in the field of language assessment. A total of 104 published articles on speaking assessment were collected and systematically analyzed within an argument-based validation framework (Chapelle et al., 2008). We examined how the published research is represented in the six inferences of this framework, the topics that were covered by each article, and the research methods that were employed in collecting the backings to support the assumptions underlying each inference. Our analysis results revealed that: (a) most of the collected articles could be categorized into the three inferences of evaluation, generalization, and explanation; (b) the topics most frequently explored by speaking assessment researchers included the constructs of speaking ability, rater effects, and factors that affect spoken performance, among others; (c) quantitative methods were more frequently employed to interrogate the inferences of evaluation and generalization whereas qualitative methods were more frequently utilized to investigate the explanation inference. The paper concludes with a discussion of the implications of this study in relation to gaining a more nuanced understanding of task- or domain-specific speaking abilities, understanding speaking assessment in classroom contexts, and strengthening the interfaces between speaking assessment, and teaching and learning practices.

Abstract Background Although teachers of English are required to assess students’ speaking proficiency in the Common European Framework of Reference for Languages (CEFR), their ability to rate is seldom evaluated. The application of descriptors in the assessment of English speaking on CEFR in the context of English as a foreign language has not often been investigated, either. Methods The present study first introduced a form of rater standardization training. Two trained raters then assessed the speaking proficiency of 100 learners by means of actual corpus data. The study then compared their rating results to evaluate inter-rater reliability. Next, ten samples of exact/adjacent agreement between Raters 1 and 2 were rated by six teachers of English in tertiary education. Two of them had attended rater standardization training with Raters 1 and 2, while the other four had not received any relevant training. Results The two raters agreed exactly in 44% of cases. The rating results between the two trained raters were closely correlated (ρ = .893). Cross-tabulation showed that in one third of the samples, Rater 2 scored higher than Rater 1 and they agreed more often at the higher levels. The better rating performance of Teachers 1 and 2 suggested that rater standardization training may have helped enhance their performance. The unsatisfactory proportion of correctly assigned levels in teachers’ ratings overall was probably due to the high input of subjective judgment based on vague CEFR descriptors. Conclusions Regarding assessment, it is shown that the attendance of rater standardization training is of help in assessing learners’ speaking proficiency in CEFR. This study provides a model for assessing data from spoken learner corpora, which adds an important attribute to future studies of learner corpora. The paper also raises doubts about teachers’ ability to evaluate students’ speaking proficiency in CEFR. As CEFR has been widely adopted in the relevant fields of English language teaching and assessment, it is suggested that the rating training framework established in this study, which uses learner corpus data, be offered to (prospective) teachers of English in tertiary education.

In 2016, Euroexam International launched a complex and ambitious project to redesign the rating process and practice of its subjectively scored papers. As part of the project, a series of level-specific performance checklists were developed for speaking and writing. This paper reports on the standard setting and internal validation procedures for B2 speaking in English as a foreign language (EFL) as executed in Spring, 2020. The participants of the study were 8 EFL teachers and oral examiners from international backgrounds with varying degrees of expertise in language testing. The method adopted for standard setting was the Body of Work method. The participants were allocated into either of two groups working independently on a random sample of 32 audio-recorded live speaking paper performances in a counterbalanced design. The study found that (a) a level-specific 30-item checklist for EFL speaking at B2 can adequately replace the operational rating scales for pass/fail decisions; (b) the granular nature of the checklist renders it more capable of covering the targeted content areas; and (c) the explicit checklist statements support fairness, transparency and accountability. [ABSTRACT FROM AUTHOR]

This mixed methods study examined holistic, analytic, and part marking models (MMs) in terms of their measurement properties and impact on candidate CEFR classifications in a semi-direct online speaking test. Speaking performances of 240 candidates were first marked holistically and by part (phase 1). On the basis of phase 1 findings—which suggested stronger measurement properties for the part MM—phase 2 focused on a comparison of part and analytic MMs. Speaking performances of 400 candidates were rated analytically and by part during that phase. Raters provided open comments on their marking experiences. Results suggested a significant impact of MM; approximately 30% and 50% of candidates in phases 1 and 2 respectively were awarded different (adjacent) CEFR levels depending on the choice of MM used to assign scores. There was a trend of higher CEFR levels with the holistic MM and lower CEFR levels with the part MM. Although strong correlations were found between all pairings of MMs, further analyses revealed important differences. The part MM was shown to display superior measurement qualities particularly in allowing raters to make finer distinctions between different speaking ability levels. These findings have implications for the scoring validity of speaking tests. [ABSTRACT FROM AUTHOR]

This paper provides a narrative review of empirical research on the assessment of speaking proficiency published in selected journals in the field of language assessment. A total of 104 published articles on speaking assessment were collected and systematically analyzed within an argument-based validation framework (Chapelle et al., 2008). We examined how the published research is represented in the six inferences of this framework, the topics that were covered by each article, and the research methods that were employed in collecting the backings to support the assumptions underlying each inference. Our analysis results revealed that: (a) most of the collected articles could be categorized into the three inferences of evaluation, generalization , and explanation ; (b) the topics most frequently explored by speaking assessment researchers included the constructs of speaking ability, rater effects, and factors that affect spoken performance, among others; (c) quantitative methods were more frequently employed to interrogate the inferences of evaluation and generalization whereas qualitative methods were more frequently utilized to investigate the explanation inference. The paper concludes with a discussion of the implications of this study in relation to gaining a more nuanced understanding of task- or domain-specific speaking abilities, understanding speaking assessment in classroom contexts, and strengthening the interfaces between speaking assessment, and teaching and learning practices. [ABSTRACT FROM AUTHOR]

Integrated speaking tasks requiring test takers to read and/or listen to stimulus texts and to incorporate their content into oral performances are now used in large-scale, high-stakes tests, including the TOEFL iBT. These tasks require test takers to identify, select, and combine relevant source text information to recognize key relationships between source text ideas, and to organize and transform information. Despite being central to evaluations of validity, relationships between stimulus content, task demands, and the oral discourse produced by test takers are yet to be empirically scrutinized to an adequate degree. In this study, we focus on a TOEFL iBT reading–listening–speaking task, applying discourse analytic measures developed by Frost, Elder and Wigglesworth (2012) to 120 oral performances to examine (a) the integration of source text ideas by test takers across three proficiency levels, and (b) the appropriateness of content-related criteria in the TOEFL integrated speaking rubric. We then combine analyses of these aspects of performances with a qualitative analysis of the generic structure and semantic profiles of stimulus texts to explore relationships between stimulus text properties and oral performances. Findings suggest that the extent to which content-related rating scale criteria distinguish between proficiency levels is contingent on stimulus text properties, with important implications for construct definitions and task design. [ABSTRACT FROM AUTHOR]

Taking into account the significant literature in Language Teaching Assessment, this article aims at understanding and discussing, based on studies about reliability (BROWN; BAILEY, 1984; BROWN, 2004; SCHOFFEN, 2009) and speaking assessment (BACHMAN, 1990; SCARAMUCCI, 2000; BORGES-ALMEIDA, 2009; QUEVEDO-CAMARGO, 2014), how English language teachers working in a language institute grade their students’ oral performance, considering the adoption (or not) of previously established criteria. A qualitative-interpretivist study was conducted and the data was collected by means of semi-structured interviews with five teachers. In face of the discrepancies among the students’ oral assessment grades, what are the factors that influence and reinforce such discrepancies? To what extent speaking assessment grading mirrors students’ real performance? Where there are different groups and different teachers, how to ensure speaking assessment reliability of students at the same reference level? The findings indicate the need to adopt previously established criteria as, due to excessive subjectivity, oral tests do not seem to measure the students’ expected level of oral performance, nor mirror language use reality.

In the realm of language assessment, the speaking abilities of second language (L2) speakers in communicative contexts have garnered increasing attention, as English serves as a lingua franca in our globalized world. In oral communication, suprasegmental features play a crucial role in shaping interactions. However, our comprehension of how proficient L2 speakers employ these suprasegmentals, particularly in the context of assessment, has been limited. This study investigated the utilization of suprasegmental features by L2 speakers, namely rate, pause, prominence, and pitch, to construct interactions and examined their connection to paired performance in a high-stakes oral assessment. This assessment is evaluated based on five criteria: Sensitivity to Situation, Language Use, Content Delivery, Discourse Management, and Engaging with Interaction. The study analyzed twenty-five paired interactions from the Cambridge English Language Assessment. The results, derived from mixed-effect modeling and stepwise regression analyses, highlight that rate and pitch measures are significant predictors of overall speaking scores. Furthermore, in-depth analysis identified specific suprasegmental features that contributed to the scores of each criterion: Sensitivity to Situation (rate), Language Use (pitch), Content Delivery (pitch), and Engaging with Interaction (rate). The findings of this study hold significant implications for language learning, speaking assessment, and pedagogy. [ABSTRACT FROM AUTHOR]

The research on the second language (L2) speaking assessment has been predominantly concerned with formal proficiency tests. However, with a growing interest in learning-oriented assessment (LOA), more researchers are interested in learner involvement in speaking assessment (Blanche & Merino, 1989; Luoma, 2004; Ocarson, 1989). Self- and peer-assessment of speaking, where learners evaluate the performance of their own and their peers, can be an exemplar of assessment with its primary purpose on learning (Chen, 2006, 2008; Ibersson, 2012; Saito, 2008; Topping & Ehly, 1998). Depending on the purpose, self- and peer-assessment can take various forms: a questionnaire on general speaking ability, a learning log for metacognitive reflection, or a classroom activity where learners use the same rating criteria as their teachers (Bachman & Palmer, 1989; Chen, 2008, Rivers, 2001).

As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. In the present study we seek first to estimate the variance attributable to rater background and attitudinal variables on novice raters' assessments of L2 spoken English. Second, we examine the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, we examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters' score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters' own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics. [ABSTRACT FROM AUTHOR]

This study investigated the impact of different lengths of pre-task planning time on performance in a test of second language speaking ability for university admission. In the study, 47 Turkish-speaking learners of English took a test of English language speaking ability. The participants were divided into two groups according to their language proficiency, which was estimated through a paper-based English placement test. They each completed four monologue tasks: two picture-based narrative tasks and two description tasks. In a balanced design, each test taker was allowed a different length of planning time before responding to each of the four tasks. The four planning conditions were 30 seconds, 1 minute, 5 minutes, and 10 minutes. Trained raters awarded scores to the test takers using an analytic rating scale and a context-specific, binary-choice rating scale, designed specifically for the study. The results of the rater scores were analysed by using a multifaceted Rasch measurement. The impact of pre-task planning on test scores was found to be influenced by four variables: the rating scale; the task type that test takers completed; the length of planning time provided; and the test takers' levels of proficiency in the second language. Increases in scores were larger on the picture-based narrative tasks than on the two description tasks. The results also revealed a relationship between proficiency and pre-task planning, whereby statistical significance was only reached for the increases in the scores of the lowest-level test takers. Regarding the amount of planning time, the 5-minute planning condition led to the largest overall increases in scores. The research findings offer contributions to the study of pre-task planning and will be of particular interest to institutions seeking to assess the speaking ability of prospective students in English-medium educational environments. [ABSTRACT FROM AUTHOR]

This study was primarily aimed at developing an English-speaking proficiency test and analytic rubrics designed to measure speaking proficiency of Malaysian undergraduates. On the basis of Littlewood's Methodological Framework and Long's Interaction Hypothesis, the researchers derived three speaking tasks from four sources: (a) syllabus of the English language courses at the relevant university, (b) Kathleen Bardovi-Harlig's operationalizing conversation speech acts, (c) IELTS part B speaking test, and (d) task B speaking section of Malaysian University English Test (MUET). A total of 96 undergraduates with four levels of the language proficiency (i.e., low performers, intermediate performers, upperintermediate performers, and high performers) from a public university in Malaysia voluntarily participated in the study. While two TESOL experts were invited to validate the content of the tasks and the rubrics, two raters rated students' test scores. Construct validity was established through known-group validity (construct validity) for a known-group comparison of the task performance at the three difficulty levels namely, elementary, intermediate and advanced. The test scores, having good internal consistency (a= .89) and inter-rater reliability (ICC= .84), yielded speaking proficiency descriptors. This result showed that the test is reliable and valid to diagnose speaking proficiency of Malaysian undergraduates in pursuit of improvement. [ABSTRACT FROM AUTHOR]

The ability to interact with others has gained recognition as part of the L2 speaking construct in the assessment literature and in high- and low-stakes speaking assessments. This paper first presents a review of the literature on interactional competence (IC) in L2 learning and assessment. It then discusses a particular feature - producing responses contingent on previous speakers' contributions - that emerged as a de facto construct feature of IC, oriented to both candidates and examiners within the school-based group speaking assessment in the Hong Kong Diploma of Secondary Education (HKDSE) English Language Examination. Previous studies have, similarly, argued for the importance of responding to or linking one's own talk to previous speakers' contributions as a way of demonstrating comprehension of co-participants' talk. However, what counts as such a response has yet to be explored systematically. This paper presents a conversation analytic study of the candidate discourse in the assessed group interactions, identifying three conversational actions through which student-candidates construct contingent responses to co-participants. The thick description about the nature of contingent responses lays the groundwork for further empirical investigations on the relevance of this IC feature and its proficiency implications. [ABSTRACT FROM AUTHOR]

We investigated how young language learners process their responses on and perceive a computer-mediated, timed speaking test. Twenty 8-, 9-, and 10-year-old non-native English-speaking children (NNSs) and eight same-aged, native English-speaking children (NSs) completed seven computerized sample TOEFL® Primary™ speaking test tasks. We investigated the children’s attentional foci on different test components (e.g., prompts, pictures, and a countdown timer) by means of their eye movements. We associated the children’s eye-movement indices (visit counts and fixation durations) with spoken performance. The children provided qualitative data (interviews; picture-drawings) on their test experiences as well. Results indicated a clear contrast between NNSs and NSs in terms of speech production (large score differences) as expected. More interestingly, the groups’ eye-movement patterns differed. NNSs tended to fixate longer on and looked more frequently at the countdown timer than their NS peers, who were more likely to look at content features, that is, onscreen pictures meant to help with building up speech. Specifically, the NNSs’ fixations on timers were likely to co-occur with hesitation phenomena (e.g., hemming; pausing; silence). We discuss (a) the potential effects of test-specific features on children’s performance and (b) child-appropriate test accommodations and practices. [ABSTRACT FROM AUTHOR]