Search

This quasi-experimental study aimed to determine the relationship between (1) oral language ability and emotions represented by facial emotions, and (2) modality of assessment (audios versus videos) and sentiments embedded in each modality. Sixty university students watched and/or listened to four selected audio-visual stimuli and orally answered follow-up comprehension questions. One stimulus was designed to evoke happiness while the other, sadness. Participants' facial emotions during the answering were measured using the "FaceReader" technology. In addition, four trained raters assessed the responses of the participants. An analysis of the "FaceReader" data showed that there were significant main and interaction effects of sentiment and modality on participants' facial emotional expression. Notably, there was a significant difference in the amount of facial emotions evoked by (1) the happy vs. sad sentiment videos and (2) video vs. audio modalities. In contrast, sentiments embedded in the stimuli and modalities had no significant effect on the measured speaking performance of the participants. Nevertheless, we found a number of significant correlations between the participants' test scores and some of their facial emotions evoked by the stimuli. Implications of these findings for the assessment of oral communication are discussed.

This study reviewed conceptualizations and operationalizations of second language (L2) listening constructs. A total of 157 peer-reviewed papers published in 19 journals in applied linguistics were coded for (1) publication year, author, source title, location, language, and reliability and (2) listening subskills, cognitive processes, attributes, and listening functions potentially measured or investigated. Only 39 publications (24.84%) provided theoretical definitions for listening constructs, 38 of which were general or had a narrow construct coverage. Listening functions such as discriminative, empathetic, and analytical listening were largely unattended to in construct conceptualization in the studies. In addition, we identified 24 subskills, 27 cognitive processes, and 54 listening attributes (total = 105) operationalized in the studies. We developed a multilayered framework to categorize these features. The subskills and cognitive processes were categorized into five principal groups each (10 groups total), while the attributes were divided into three main groups. This multicomponential framework will be useful in construct delineation and operationalization in L2 listening assessment and teaching. Finally, limitations of the extant research and future directions for research and development in L2 listening assessment are discussed.

Studies have shown that test-takers tend to use keyword-matching strategies when taking listening tests. Keyword-matching involves matching content words in the written modality (test items) against those heard in the audio text. However, no research has investigated the effect of such keywords in listening tests, or the impact of gazing upon these keywords on listening test scores. Thus, this study examined whether test-takers' performance on a listening test can be explained by their gaze behaviors across three types of content words in the written modality: nouns, verbs, and adjectives. Using eye-tracking technology, this study measured the gaze behavior of 66 listening test-takers during reading content words in test item stems. Using linear mixed effect model, binary probit regression, and multinomial logistic regression, we found that test-takers' performance was predicted by gaze behavioral measures on content words. Among the content words, fixating on nouns in written test items had the most significant role in predicting test performance, followed by adjectives and verbs. By shedding light on how keywords in test items are attended to by test-takers and the relationship between keyword-matching and listening test performance, this study has provided significant evidence for the overwhelming role of reading in listening tests. Implications for test score interpretation are discussed.

Construct validity and building validity arguments are some of the main challenges facing the language assessment community. The notion of construct validity and validity arguments arose from research in psychological assessment and developed into the gold standard of validation/validity research in language assessment. At a theoretical level, construct validity and validity arguments conflate the scientific reasoning in assessment and policy matters of ethics. Thus, a test validator is expected to simultaneously serve the role of conducting scientific research and examining the consequential basis of assessments. I contend that validity investigations should be decoupled from the ethical and social aspects of assessment. In addition, the near-exclusive focus of empirical construct validity research on cognitive processing has not resulted in sufficient accuracy and replicability in predicting test takers' performance in real language use domains. Accordingly, I underscore the significance of prediction in validation, in contrast to explanation, and propose that the question to ask might not so much be about what a test measures as what type of methods and tools can better generate language use profiles. Finally, I suggest that interdisciplinary alliances with cognitive and computational neuroscience and artificial intelligence (AI) fields should be forged to meet the demands of language assessment in the 21st century.

With the advent of new technologies, assessment research has adopted technology-based methods to investigate test validity. This study investigated the neurocognitive processes involved in an academic listening comprehension test, using a biometric technique called functional near-infrared spectroscopy (fNIRS). Sixteen right-handed university students completed two tasks: (1) a linguistic task that involved listening to a mini-lecture (i.e., Listening condition) and answering of questions (i.e., Questions condition) and (2) a non-linguistic task that involved listening to a variety of natural sounds and animal vocalizations (i.e., Sounds condition). The hemodynamic activity in three left brain regions was measured: the inferior frontal gyrus (IFG), dorsomedial prefrontal cortex (dmPFC), and posterior middle temporal gyrus (pMTG). The Listening condition induced higher activity in the IFG and pMTG than the Sounds condition. Although not statistically significant, the activity in the dmPFC was higher during the Listening condition than in the Sounds conditions. The IFG was also significantly more active during the Listening condition than in the Questions condition. Although a significant gender difference was observed in listening comprehension test scores, there was no difference in brain activity (across the IFG, dmPFC, and pMTG) between male and female participants. The implications for test validity are discussed.

This study aimed to investigate the test-taking strategies needed for successful completion of a lecture-based listening test by employing self-reported test-taking strategy use, actual strategy use measured via eye-tracking, and test scores. In this study, participants' gaze behavior (measured by fixation and visit duration and frequency) were recorded while they completed two listening tests of three stages each: pre-listening, in which participants (n = 66) previewed question stems; while-listening, in which participants simultaneously listened to the recording and filled in their answers; and post-listening, in which they had time to review their answers and make necessary amendments. Following the listening tests, participants filled up a posttest questionnaire that asked about their strategy use in each of the three stages. Rasch measurement, "t"-test, and path analysis were performed on test scores, questionnaire results, and gaze patterns. Results suggest that gaze measures (visit duration and fixation frequency) predicted participants' final test performance, while self-reports had moderate predicting power. The findings of this study have implications for the cognitive validity of listening tests, listening test design and pedagogical approaches in building listening competence.

Automated writing evaluation (AWE) systems are developed based on interdisciplinary research and technological advances such as natural language processing, computer sciences, and latent semantic analysis. Despite a steady increase in research publications in this area, the results of AWE investigations are often mixed, and their validity may be questionable. To yield a deeper understanding of the validity of AWE systems, we conducted a systematic review of the empirical AWE research. Using Scopus, we identified 105 published papers on AWE scoring systems and coded them within an argument-based validation framework. The major findings are: (i) AWE scoring research had a rising trend, but was heterogeneous in terms of the language environments, ecological settings, and educational level; (ii) a disproportionate number of studies were carried out on each validity inference, with the evaluation inference receiving the most research attention, and the domain description inference being the neglected one, and (iii) most studies adopted quantitative methods and yielded positive results that backed each inference, while some studies also presented counterevidence. Lack of research on the domain description (i.e., the correspondence between the AWE systems and real-life writing tasks) combined with the heterogeneous contexts indicated that construct representation in the AWE scoring field needs extensive investigation. Implications and directions for future research are also discussed.

This paper investigates the relationship between secondary school students' reading metacognition and summary writing skills--which represent a part of their academic language skills--within a network of psychological and sociocultural factors consisting of positivity, tolerance and respect, resilience, and culture and cooperation. We used data from the 2018 OECD's Programme for International Student Assessment (PISA) database (n = 612,004) from 79 countries. We conducted a series of confirmatory factor analyses (CFA) to validate the variance and covariance structure of the tests and questionnaires used. Next, we performed structural equation modeling (SEM) to investigate the effect of the predictors on reading metacognition and summary writing skills. A significant relationship was found between sociocultural factors, reading metacognition, and summary writing skills. We propose that it is essential to consider how a student's background, experiences, and school curriculum determine how these indicators present themselves in each student and affect their academic language performance.

Digital storytelling (DST) is a novel approach that uses modern computer technology to amplify language learning and teaching. The present study aims to review how the published DST research utilizes visuals and audio to influence the learning environment and engage adolescent and adult language learners. This was measured through their improvement in the four main language skills: reading, writing, listening, and speaking. A total 71 journal papers were identified using the Scopus database. The papers studied both first and second language learning and were coded in full-text screening for the research topics and methods adopted, theories or frameworks adopted, outcomes across the language skills, and reliability investigation of the studies. The results showed a range of research used in the studies, with 39.7% of the total studies using a mixed number of methods. The theories adopted in these studies were limited to components of DST, age group, and the type of study. Most studies neither tested nor mentioned the use of the three theoretical variables mentioned above. Notably, a majority of the studies reported positive outcomes when DST was used in the learning environment. However, not all claims were supported with evidence. Lastly, only a handful of the studies reviewed reported reliability, highlighting a lack of verification of the precision of the measurement instruments used. Implications of these findings and recommendations for designing DST and language learning research in the future will be discussed.

The fear of public speaking is a prevalent phobia that has a damaging impact on the lives of many phobic patients. One method to treat this phobia is the use of virtual reality (VR). A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol was used to report how the publications that have examined the role of VR in treating public speaking anxiety were identified. A meta-analysis of 92 non-review publications published by January 15, 2021 was conducted. In this meta-analysis, the effectiveness of the treatment of public speaking anxiety refers to the degree of reduction in the participants' public speaking anxiety from pre-test to post-test. This meta-analysis consisted of an examination of the homogeneity of the studies (the I.sup.2 indexes), publication bias (Kendall's tau and Egger's regression values) and an estimation of the grand effect size for all studies. The three major findings of this meta-analysis are: (1) Overall, VR had a statistically significant effect on reducing public speaking anxiety, which suggests that VR is a useful and promising therapeutic tool for the treatment of public speaking anxiety; (2) Studies that found VR to be effective in the treatment of public speaking anxiety conducted an average of approximately six VR sessions, with each session lasting around 37 minutes; and (3) VR is statistically as effective as other treatment methods such as cognitive behavioral therapy. Therefore, rather than completely replacing other treatment methods, VR should be used to complement other treatment methods to compensate for some of their disadvantages., Author(s): Mei Hui Lim [sup.1] , Vahid Aryadoust [sup.2] , Gianluca Esposito [sup.1] [sup.3] Author Affiliations: (1) grid.59025.3b, 0000 0001 2224 0361, Nanyang Technological University, , Singapore, Singapore (2) grid.59025.3b, [...]

Work motivation has profound benefits ranging from increased development to performance for employees and companies. The present nationwide survey adopted the Multidimensional Work Motivation Questionnaire (Gagné et al., 2015) and developed a Work Motivation Scale to accurately measure the construct amongst three groups of respondents working in the Training and Adult Education Sector in Singapore. Using the Rasch-Andrich rating scale model and structural equation modeling (SEM), we investigated the psychometric features of the dimensions of the instrument and its measurement invariance, respectively. We found evidence supporting the fit of the dimensions to the Rasch model and also a good level of partial structural invariance for the majority of the parameters in the SEM model. The present study extends existing self-determination theory literature by measuring work motivation in the multicultural context of Singapore and emphasises the importance of invariance measurement. The findings lend support for the continued use of the Work Motivation Scale to measure the antecedents and outcomes of work motivation., Author(s): Zan Chen [sup.1] , Vahid Aryadoust [sup.2] Author Affiliations: (1) grid.443365.3, 0000 0004 0388 6484, Institute for Adult Learning, Singapore University of Social Sciences, , Singapore, Singapore (2) grid.59025.3b, [...]

This is the first study to investigate the effects of test methods (while-listening performance and post-listening performance) and gender on measured listening ability and brain activation under test conditions. Functional near-infrared spectroscopy (fNIRS) was used to examine three brain regions associated with listening comprehension: the inferior frontal gyrus and posterior middle temporal gyrus, which subserve bottom-up processing in comprehension, and the dorsomedial prefrontal cortex, which mediates top-down processing. A Rasch model reliability analysis showed that listeners were homogeneous in their listening ability. Additionally, there were no significant differences in test scores across test methods and genders. The fNIRS data, however, revealed significantly different activation of the investigated brain regions across test methods, genders, and listening abilities. Together, these findings indicated that the listening test was not sensitive to differences in the neurocognitive processes underlying listening comprehension under test conditions. The implications of these findings for assessing listening and suggestions for future research are discussed.

A fundamental requirement of language assessments which is underresearched in computerized assessments is impartiality (fairness) or equal treatment of test takers regardless of background. The present study aimed to evaluate fairness in the Pearson Test of English (PTE) Academic Reading test, which is a computerized reading assessment, by investigating differential item functioning (DIF) across Indo-European (IE) and Non-Indo-European (NIE) language families. Previous research has shown that similarities between readers' mother tongue and the second language being learned can advantage some test takers. To test this hypothesis, we analyzed data from 783 international test takers who took the PTE Academic test, using the partial credit model in Rasch measurement. We examined two main types of DIF: uniform DIF (UDIF), which occurs when an item consistently gives a particular group of test takers an advantage across all levels of ability, and non-uniform DIF (NUDIF), which occurs when the performance of test takers varies across the ability continuum. The results showed no statistically significant UDIF (p > 0.05), but identified 3 NUDIF items out of 10 items across the language families. A mother tongue advantage was not observed. Similarity in test takers' level of computer and Internet skills, test preparation, and language policies could contribute to the finding of no UDIF. Post-hoc content analysis of items suggested that the decrease of mother tongue advantage for IE groups in high-proficiency groups and lucky guesses of low-ability groups may have contributed to the emergence of NUDIF items. Lastly, recommendations for investigating social and contextual factors are proposed.

This study sought to examine research trends in computer-assisted language learning (CALL) using a retrospective scientometric approach. Scopus was used to search for relevant publications on the topic and generate a dataset consisting of 3,697 studies published in 11 journals between 1977 and 2020. A document co-citation analysis method was adopted to identify the main research clusters in the dataset. The impact of each publication on the field was measured by using the burst index and the betweenness centrality and the content of influential publications was closely analysed to determine the focus of each cluster and the key themes of the studies in focus. Overall, we identified seven major clusters. We further found that leveraging synchronous computer-mediated communication and negotiated interaction, multimedia, telecollaboration or e-mail exchanges, blogs, digital games, Wikis and podcasts to support language learning was probably beneficial for language learning. Varying degrees of support were found in various studies for each of these technologies. Stronger support was found for synchronous computer-mediated communication and negotiated interaction, multimedia, telecollaboration or e-mail exchanges and digital games and weaker support was found for blogs, Wikis, and podcasts. The limitations the supporting studies listed were also considered inconsequential. On the other hand, while there was strong support for blogs, Wikis and podcasts, some major drawbacks were observed. The findings of the study would be helpful for teachers and instructors who want to decide whether to use technology in the classroom for instructional purposes. Additionally, researchers and graduate students who need to identify a research topic for their thesis or dissertation may find the results of the study useful for them, too.

This study evaluated the validity of the Michigan English Test (MET) Listening Section by investigating its underlying factor structure and the replicability of its factor structure across multiple test forms. Data from 3255 test takers across four forms of the MET Listening Section were used. To investigate the factor structure, each form was fitted with four Bayesian confirmatory factor analysis (CFA) models: (1) a three correlated-factor model, (2) a bi-factor model, (3) a higher-order factor model, and (4) a single general-factor model. In addition, a four-pronged heuristic comprising construct delineation, construct operationalization, factor structure analysis, and congruence coefficient was developed to examine the replicability of factor structures across the test forms. Results from the CFA models showed that the test forms were unidimensional and the four-pronged heuristic indicated that the test construct was consistently operationalized across forms. Furthermore, the congruence coefficient indicated that the factor structure representing listening was highly similar and replicable across test forms. In sum, the construct of the MET Listening Section did not comprise divisible subskills. Yet, the unidimensional factor structure of the test was replicable across the test forms.

The aim of this study was to investigate how test methods affect listening test takers' performance and cognitive load. Test methods were defined and operationalized as while-listening performance (WLP) and post-listening performance (PLP) formats. To achieve the goal of the study, we examined test takers' (N = 80) brain activity patterns (measured by functional near-infrared spectroscopy (fNIRS)), gaze behaviors (measured by eye-tracking), and listening performance (measured by test scores) across the two test methods. We found that the test takers displayed lower activity levels across brain regions supporting comprehension during the WLP tests relative to the PLP tests. Additionally, the gaze behavioral patterns exhibited during the WLP tests suggested that the test takers adopted keyword matching and "shallow listening." Together, the neuroimaging and gaze behavioral data indicated that the WLP tests imposed a lower cognitive load on the test takers than the PLP tests. However, the test takers performed better with higher test scores for one of two WLP tests compared with the PLP tests. By incorporating eye-tracking and neuroimaging in this exploration, this study has advanced the current knowledge on cognitive load and the impact imposed by different listening test methods. To advance our knowledge of test validity, other researchers could adopt our research protocol and focus on extending the test method framework used in this study.

Over the past decades, the application of Rasch measurement in language assessment has gradually increased. In the present study, we coded 215 papers using Rasch measurement published in 21 applied linguistics journals for multiple features. We found that seven Rasch models and 23 software packages were adopted in these papers, with many-facet Rasch measurement (n = 100) and Facets (n = 113) being the most frequently used Rasch model and software, respectively. Significant differences were detected between the number of papers that applied Rasch measurement to different language skills and components, with writing (n = 63) and grammar (n = 12) being the most and least frequently investigated, respectively. In addition, significant differences were found between the number of papers reporting person separation (n = 73, not reported: n = 142) and item separation (n = 59, not reported: n = 156) and those that did not. An alarming finding was how few papers reported unidimensionality check (n = 57 vs 158) and local independence (n = 19 vs 196). Finally, a multilayer network analysis revealed that research involving Rasch measurement has created two major discrete communities of practice (clusters), which can be characterized by features such as language skills, the Rasch models used, and the reporting of item reliability/separation vs person reliability/separation. Cluster 1 was accordingly labelled the production and performance cluster, whereas cluster 2 was labelled the perception and language elements cluster. Guidelines and recommendations for analyzing unidimensionality, local independence, data-to-model fit, and reliability in Rasch model analysis are proposed.

Eye tracking technology has become an increasingly popular methodology in language studies. Using data from 27 journals in language sciences indexed in the Social Science Citation Index and/or Scopus, we conducted an in-depth scientometric analysis of 341 research publications together with their 14,866 references between 1994 and 2018. We identified a number of countries, researchers, universities, and institutes with large numbers of publications in eye tracking research in language studies. We further discovered a mixed multitude of connected research trends that have shaped the nature and development of eye tracking research. Specifically, a document co-citation analysis revealed a number of major research clusters, their key topics, connections, and bursts (sudden citation surges). For example, the foci of clusters #0 through #5 were found to be perceptual learning, regressive eye movement(s), attributive adjective(s), stereotypical gender, discourse processing, and bilingual adult(s). The content of all the major clusters was closely examined and synthesized in the form of an in-depth review. Finally, we grounded the findings within a data-driven theory of scientific revolution and discussed how the observed patterns have contributed to the emergence of new trends. As the first scientometric investigation of eye tracking research in language studies, the present study offers several implications for future research that are discussed.

This study investigates the underlying structure of the listening test of the Singapore-Cambridge General Certificate of Education (GCE) exam, comparing the fit of five cognitive diagnostic assessment models comprising the deterministic input noisy "and" gate (DINA), generalized DINA (G-DINA), deterministic input noisy "or" gate (DINO), higher-order DINA (HO-DINA), and the reduced reparameterized unified model (RRUM). Through model-comparisons, a nine-subskill RRUM model was found to possess the optimal fit. This study shows that students' listening test performance depends on an array of test-specific facets, such as the ability to eliminate distractors in multiple-choice questions alongside listening-specific subskills such as the ability to make inferences. The validated list of the listening subskills can be employed as a useful guideline to prepare students for the GCE listening test at schools.

I analyzed a corpus of the international English language testing system (IELTS) comprising 256 listening sections (1996–2021). The primary objective of the study was to gain insights into the assumptions made by test designers regarding the real-life contexts that test-takers will encounter. Overall, 15 superordinate topic areas and 300 subtopics were identified in the corpus. There was relatively more diversity in topic coverage before 2000. However, the test did not incorporate texts that would address sociocultural matters related to local or international contexts. Additionally, English-as-L2 accents were virtually unrepresented with only three samples from 514 speakers, potentially suggesting a racialized perspective on the listening construct. I argue that it is possible that this way of testing promotes and normalizes test designers' ideologies, while overlooking the importance of the diversity of domains that test-takers will encounter in daily life. I discuss the potential construction of test-takers' identity through exposure to these topics and suggest that test designers may consider reevaluating topic and accent coverage in the test to improve fairness and equity in the test. Finally, I provide ideas on how the use of generative artificial intelligence (AI) can enhance quality of language assessments. [ABSTRACT FROM AUTHOR]

The aim of the present study is two-fold. First, it uses eye-tracking to investigate the dynamics of item reading, both in multiple choice and matching items, before and during two hearings of listening passages in a computerized while-listening performance (WLP) test. Second, it investigates answer changing during the two hearings, which include four rounds of item reading taking place during: pre-listening in hearing 1, while-listening in hearing 1, pre-listening in hearing 2, and while-listening in hearing 2. The listening test was completed by 28 secondary school students in different sessions. Using time series, cross-correlation functions, and multivariate data analyses, we found that listeners tended to quickly skim the test items, distractors, and answers during pre-listening in hearing 1 and pre-listening in hearing 2. By contrast, during while-listening in hearing 1 and while-listening in hearing 2, significantly more attention was paid to the written stems, distractors, and options. The increment in attention to the written stems, distractors, and options was greater for the matching items and interactions between item format and item reading were also detected. Additionally, we observed a mixed answer changing pattern (i.e., incorrect-to-correct and correct-to-incorrect), although the dominant pattern for both item formats (67%) was wrong-to-correct. Implications of the findings for language research are discussed.

This article proposes an integrated cognitive theory of reading and listening that draws on a maximalist account of comprehension and emphasizes the role of bottom-up and top-down processing. The theoretical framework draws on the findings of previous research and integrates them into a coherent and plausible narrative to explain and predict the comprehension of written and auditory inputs. The theory is accompanied by a model that schematically represents the fundamental components of the theory and the comprehension mechanisms described. The theory further highlights the role of perception and word recognition (underresearched in reading research), situation models (missing in listening research), mental imagery (missing in both streams), and inferencing. The robustness of the theory is discussed in light of the principles of scientific theories adopted from Popper (1959).

To investigate the reliability of L2 listening tests and explore potential factors affecting the reliability, a reliability generalization (RG) meta-analysis was conducted in the present study. A total number of 122 alpha coefficients of L2 listening tests from 92 published articles were collected and submitted to a linear mixed effects RG analysis. The papers were coded based on a coding scheme consisting of 16 variables classified into three categories: study features, test features, and statistical results. The results showed an average reliability of 0.818 (95% CI: 0.803 to 0.833), with 40% of reliability estimates falling below the lower bound of CI. The presence of publication bias and heterogeneity was found in the reliability of L2 listening tests, indicating that low reliability coefficients were likely omitted from some published studies. In addition, two factors predicting the reliability of L2 listening tests were the number of items and test type (standardized and researcher- or teacher-designed tests). The study also found that reliability is not a moderator of the relationship between L2 listening scores and theoretically relevant constructs. Reliability induction was identified in reporting the reliability of L2 listening tests, too. Implications for researchers and teachers are discussed. [ABSTRACT FROM AUTHOR]

Previous research on providing feedback on public speaking has investigated the effectiveness of feedback sources, namely teacher feedback, peer feedback, and self-feedback, in enhancing public speaking competence, predominantly individually. However, how these sources of feedback can be collectively harnessed to optimize learner engagement and public speaking performance still warrants further investigation. Adopting a pre- and post-test quasi-experimental design, this study randomly assigned four classes to four feedback conditions: Group 1 received teacher feedback, Group 2 self-feedback and teacher feedback, Group 3 peer and teacher feedback, and Group 4 feedback from all three sources. Both student engagement, measured using the Public Speaking Feedback Engagement Scale (PSFES), and their public speaking performance ratings, assessed using the Public Speaking Competency Instrument (PSCI), were validated using Rasch analysis. The inferential statistics revealed that Group 3 showed significant improvements across nearly all three dimensions of engagement, whereas Group 2 experienced significant declines in all dimensions of engagement except behavioral engagement. Group 3 demonstrated significantly greater engagement gain compared to Groups 2 and 4, indicating the synergistic effect of peer and teacher feedback in contrast to the limited impact of self-feedback. Additionally, all groups demonstrated significant improvements except for Group 2, which showed significantly lower improvement compared to Group 4. The following correlation analysis identified a significant correlation between the gain of students' behavioral engagement and the gain of public speaking performance, whereas such association was absent between cognitive or emotional engagement and public speaking competence. This study suggests that peer feedback should be preceded by group discussion and supplemented with teacher feedback in classes for enhancing the teacher–student dialog, while self-feedback should be conducted after class to improve student engagement and public speaking performance. [ABSTRACT FROM AUTHOR]

In this study, we provide a scientometric analysis of 43,685 studies published in 51 quartile-1 journals in the field of applied linguistics (1970–2022). Scientometric analysis uses citation records to quantitatively compute networks of cited works and map out how published works have been cited. We adapted a multi-stage scientometric method consisting of database identification, dataset generation, document co-citation analysis, research cluster identification, and cluster characterization. A number of major research clusters were identified and a high degree of interconnectedness in terms of theoretical base was observed between the clusters. The pre-2000 publications had a conspicuous focus on theories derived from language use, which might be said had set the tone for the maturation of the field. By contrast, the clusters that emerged from the 2000s showed more specificity and granularity in focus and scope, suggesting the beginning of a research era with more specialized directions. Despite this trend, we identified influential publications which received several spikes in citations in different eras, indicating their continued temporal and thematic relevance in different clusters. In addition, we found evidence of inter-cluster cross-pollinations. We discuss how each cluster should be characterized in terms of its knowledge base and knowledge front. Highly cited works form the knowledge base of a cluster while novel works form the knowledge fronts of a cluster. Future directions are mentioned and highlighted. [ABSTRACT FROM AUTHOR]

The formulation of the construct of communicative competence (CC) was the consequence of the perceived "inappropriateness" of the theory of linguistic competence and performance. To obtain a comprehensive understanding of second language (L2) CC, a systematic review of 85 studies was conducted to assess how empirical studies have defined and operationalized the construct in the context of L2 learning and assessment. Four main themes emerged from the papers: (1) beliefs and perceptions about CC, (2) expansion of CC, (3) mixed specifications of CC, and (4) tests and measures of CC and communicative language ability (CLA). The analysis of these themes foregrounded several significant findings. First, the construct of L2 CC was significantly more prevalent in the body of research compared to L2 CLA. Second, CC has been perceived as a multifarious construct that has been researched from varied perspectives. It was found that older studies were more consistent with traditional approaches to defining CC and CLA, while the construct shifted focus to technology and self-appraisals in more recent studies. Third, there is no consensus amongst the reviewed studies about tests and methods of operationalization of CC, suggesting that the evidence provided is sample-specific and non-replicable. Importantly, it was found that over the years, CC has undergone a gradual evolution. With the multimodal and intercultural turn, CC has branched into new concepts namely intercultural and multimodal communicative competencies. Pertaining to these concepts, new notions such as telecollaboration, digital literacies, and multiliteracies have emerged. CC has also been, for long, analogous to performance, proficiency, social rules of language use, rules of appropriateness, willingness to communicate, self-perceptions of CC, and the goal of being a native speaker, which can add to the confusion surrounding the construct. The implications of the present research synthesis are two-fold. It becomes imperative to adapt world language classrooms to the rising trend in intercultural and multimodal communicative competencies. In addition, further replicable investigations should focus on developing optimal methods of operationalization that are in line with the new contemporary theoretical frameworks of language in the age of digital technologies and artificial intelligence. [ABSTRACT FROM AUTHOR]

We present a thematic review and analysis of the variables affecting language learning motivation (LLM) (2008–2022). The second-language motivational self-system (L2MSS) model was found to be the most applied construct in measuring LLM. Complex systems theory was also another method gaining prominence in LLM research to explain the interactions between micro- and macro-structures surrounding the learner in influencing motivation. Other factors such as socioeconomic status, dialogism and anagnorisis were also identified as variables relating to LLM. For instance, research on dialogism and dialogue has indicated the role of conversation in shaping identity, motivation, and meaning for learners. However, our review found that much of the focus in LLM research has been on the L2MSS learning or teaching experience, while daily living has been largely neglected. We further conducted a reliability generalization meta-analysis. Our analysis found an average reliability of 0.84 (CI = 0.816–0.856), with 34% of reliability coefficients falling below the lower bound of CI. A meta-regression analysis revealed that 16% of the variance in the reliability coefficients was predicted by the number of items in the instruments. Questionnaires with an internal consistency below the lower bound of 0.816 had an average of 4.14 items, while the rest had an average of 5.71 items. We further found significant publication bias. Based on our findings, we suggest areas for future research in LLM. [ABSTRACT FROM AUTHOR]

The purpose of the present study was twofold: (a) it examined the relationship between peer-rated likeability and peer-rated oral presentation skills of 96 student presenters enrolled in a science communication course, and (b) it investigated the relationship between student raters' severity in rating presenters' likeability and their severity in evaluating presenters' skills. Students delivered an academic presentation and then changed roles to rate their peers' performance and likeability, using an 18-item oral presentation scale and a 10-item likeability questionnaire, respectively. Many-facet Rasch measurement was used to validate the data, and structural equation modeling (SEM) was used to examine the research questions. At an aggregate level, likeability explained 19.5% of the variance of the oral presentation ratings and 8.4% of rater severity. At an item-level, multiple cause-effect relationships were detected, with the likeability items explaining 6-30% of the variance in the oral presentation items. Implications of the study are discussed.

This study adapts Levels 1 and 2 of Kirkpatrick's model of training evaluation to evaluate learning outcomes of an English as a second language (ESL) paragraph writing course offered by a major Asian university. The study uses a combination of surveys and writing tests administered at the beginning and end of the course. The survey evaluated changes in students' perception of their skills, attitude, and knowledge (SAK), and the writing tests measured their writing ability. Rasch measurement was applied to examine the psychometric validity of the instruments. The measured abilities were successively subjected to path modeling to evaluate Levels 1 and 2 of the model. The students reported that the module was enjoyable and useful. In addition, their self-perceived level of skills and knowledge developed across time alongside their writing scores but their attitude remained unchanged. Limitations of Kirkpatrick's model as well as lack of solid frameworks for evaluating educational effectiveness in applied linguistics are discussed.