Search

Background: Mid-semester evaluations are gaining traction as a means to gather evaluation data for formative purposes. However, it is not clear if course coordinators who conduct these evaluations are adequately equipped with evaluative knowledge and skills to guide them through their evaluative processes. Objectives: This study is a meta-evaluation of course coordinators' mid-semester evaluative practices in a Higher Education institution language centre and the extent to which their evaluative practices are attuned to RUFDATA for its preparatory design framework in the evaluative process and Utilization-Focused Evaluation (UFE) for its sets of principles with regards to uses and users and how these impact on the design and use of the evaluation. Research design & subjects: Data was gathered using meta evaluative mixed methods from three groups of participants consisting of course coordinators, academics who taught on their courses and their learners. Results: The results imply that whilst most course coordinators who participated in the study conducted mid-semester evaluations and that their evaluative practices contained some elements from the frameworks, it is problematic to align their evaluative practices and processes against these tools due to the challenges that they encountered in their evaluative processes. Conclusions: As important agents of evaluations in mid semester evaluations, it should be noted that course coordinators are not trained evaluators and conducting evaluations does not constitute a major component of their professional duties and roles. This study has implications for higher education policies and practices, commissioners, and users of evaluations.

When forensic interviewers reject children's "Don't know" responses, either by repeating questions or pressuring the children to provide different responses, children may change their subsequent responses. The primary objective of the current study was to examine interviewer reactions following preschool-aged alleged abuse victims' "Don't know" responses in 114 forensic interviews and the children's responses to these rejections. Interviewer reactions were dichotomously coded as either interviewer acceptance (i.e., transitioning to the next logical question or formulating questions focusing on previously mentioned details) or interviewer rejection (i.e., repeating questions or making negative remarks about recall ability). The results showed that the interviewers accepted the children's "Don't know" responses 75.3% of the time and rejected them 24.7% of the time. When interviewers rejected the children's "Don't know" responses, 75.9% of the subsequent responses contradicted the children's initial responses. These results suggest that interviewer rejections following preschool-aged children's "Don't know" responses may be suggestive.

This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the many-facet Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence rating behaviour. Finally, we elaborate on implications for both research and practice.

Researchers seeking to engage in projects related to Tribal communities and their citizens, lands, and non-human relatives are responsible for understanding and abiding by each Tribal nation's research laws and review processes. Few studies, however, have described the many diverse forms of Tribal research review systems across the United States (US). This study provides one of the most comprehensive examinations of research review processes administered by Tribal Research Review Boards (TRRBs) in the US. Through a systematic analysis, we consider TRRBs' online presence, online documentation, and themes across documents, for five entity types: Tribal nations and Tribal consortiums, Tribal colleges and universities, Tribal health organizations, Indian Health Services, and other Tribal organizations. Results include an assessment of online presence for 98 potential TRRBs, identification of 118 publicly available online documents, and analysis of 41 themes across four document types: Tribal research codes and TRRBs' guidelines, applications, and post-approval documents. Altogether, this research provides a macro-level analysis of the most common types of TRRBs in the US in an effort to increase researchers' understanding of these important processes as they prepare to ethically engage Tribal communities in research. These results aim to empower Tribal decision makers as they align their TRRBs' online visibility and documentation with community priorities and strengthen their protections for the rights and wellbeing of their citizens and community. Ultimately, by expanding our knowledge of TRRBs across the US, this contribution seeks to uphold Tribal sovereignty in research and promote ethical approaches to research with Indigenous communities.

This study's purpose is to extend our understanding of school leadership for student learning by identifying high-leverage teacher evaluation practices that improve teaching. A partnership with a state education agency administered a teacher questionnaire regarding evaluation practices multiple times in one semester, then linked teacher responses to their next within-semester observation score. Broadly, teachers reported on scoring practices, the facilitation of post-observation conferences, feedback characteristics, and post-conference supports for evaluation-informed professional learning. Fixed effect regressions effectively compare observation scores and teacher-reported evaluation practices within the same teacher or teacher-by-evaluator pairing over four months while controlling for month-to-month influences on performance. The methods remove several serious confounders plausibly affecting related estimates in prior work. The analysis identifies six high-leverage teacher-reported evaluation practices, most of which apply to post-conference practices linking evaluation to professional learning. The evidence refines the academic understanding of leadership for student learning and implies that leadership preparation and in-service programs might emphasize the six high-leverage evaluation practices to promote active use among practicing leaders. Policymakers might ensure that aspiring and in-service leaders can develop these practices and that there are strong links between teacher evaluation and professional learning systems for school leaders to use.

Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained Transformer (GPT)-4, the biggest LLM so far. This pre-registered study uses a "human-out-of-the-loop" approach to evaluate GPT-4's capability in title/abstract screening, full-text review and data extraction across various literature types and languages. Although GPT-4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full-text literature using highly reliable prompts, GPT-4's performance was more robust, reaching "human-like" levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance.

The impostor phenomenon is a psychological construct referring to a range of negative emotions associated with a person's perception of their own "fraudulent competence" in a field or of their lack of skills necessary to be successful in that field. Anecdotal evidence suggests that many practicing evaluators have experienced impostor feelings, but lack a framework in which to understand their experiences and the forums in which to discuss them. This paper summarizes the literature on the impostor phenomenon, applies it to the field of evaluation, and describes the results of an empirical, quantitatively focused study which included open-ended qualitative questions exploring impostorism in 323 practicing evaluators. The results suggest that impostor phenomenon in evaluators consists of three constructs: Discount, Luck, and Fake. Qualitative data analysis suggests differential coping strategies for men and women. Thematic analysis guided the development of a set of proposed solutions to help lessen the phenomenon's detrimental effects for evaluators.

People with intellectual disabilities (IDs) have experienced involuntary and inhumane research practices. Consequently, researchers have shifted towards "excluding" those with IDs; caregivers, teachers, or peers compose study samples, dominating a space they "indirectly" experience. Researcher bias regarding intellectual capacity has resulted in a unique research gap that overlooks insights from the population the research is intended to benefit. People with ID are interested in research participation and have a right to be included in decision-making that impacts them. Emancipatory inquiry allows school psychology scholars to center student voices while also promoting social justice. Emancipatory inquiry empowers the "subjects" of social inquiry by producing knowledge that directly benefits disenfranchised populations. Emancipatory inquiry aligns with the social justice frame of DisCrit, which compels us to privilege the voices of marginalized populations and recognize that many of the gains for disabled populations have largely occurred because of the benefits they afford White, able-bodied, middle-class citizens. Using Emancipatory inquiry, school psychology scholars can prepare trainees and future researchers to conduct ethically sound research, prioritize first-person voices of those with ID, provide socially valid services to students and their families, and move school psychology closer to partnership with the disabled community.

The field of evaluation has experienced greater professionalization in the areas of evaluator education and training. Growth in these areas included sensitivity to issues of diversity, with efforts to attract and retain evaluators of color. Currently, there is limited scholarship on navigating a world with more opportunity but still dealing with being Black. Motivated by our identities as Black evaluators, we explored the education, training and socialization concerns experienced by Black evaluators and the advice they would offer. Through content analysis of semi-structured interviews and the utilization of Critical Race Theory as a framework, we garnered advice to and from Black evaluators on how to (a) navigate the politics in evaluation; (b) engage in reflection and reflexivity; (c) network and collaborate; (d) mentor and bring others along the way; and (e) develop as a professional evaluator. We offer a synthesis of these findings in the discussion and articulate further implications.

Anxiety is a multifaceted force that can negatively impact the ability of evaluators to succeed in practice. In the evaluation literature, discussions concerning anxiety have primarily been limited to strategies to reduce stakeholder anxiety to encourage positive and productive working relationships with evaluators. This study was among the first to explicitly recognize and empirically assess the anxiety "evaluators" feel during practice. Using a mixed-methods design, a random sample of 109 American Evaluation Association members was surveyed. There was overwhelming agreement (96.30%) that evaluator anxiety is a commonly experienced phenomenon, often citing reasons such as lack of experience, imposter syndrome, unreasonable scope, and difficulties with stakeholder interactions as key contributors to increased anxiety. Evaluator anxiety varied across stages of the evaluation process, with some stages eliciting more anxiety than others. Further discussion and training concerning how to reduce evaluator anxiety could be useful in the preparation of evaluators.

This study aimed to develop and validate the Evaluator Competencies Assessment Tool (ECAT) Cultural Competencies Subscale, which measures cultural competence among evaluators. By addressing the scarcity of validated tools in this area, the study offers professionals a valuable resource to assess their strengths and areas for improvement. The subscale comprises 11 items rated on a seven-point Likert scale, derived from the American Evaluation Association (AEA) Evaluator Competencies. Validation involved a survey of 116 AEA members, employing multiple validity procedures. The subscale demonstrated excellent internal consistency ([alpha] =0.96) and significant correlation with the Cultural Competence of Program Evaluators scale, confirming its convergent validity. However, the subscale's structural validity yielded mixed results, indicating the need for further exploration. Moreover, the demographic analysis highlighted underrepresentation of Asian and younger members. Overall, the ECAT Cultural Competencies Subscale shows promise for assessing cultural competency, but refinement of its factor structure and additional research are needed.

Given the ever-evolving role of school counselors and increasing demands placed on them in today's sociopolitical climate, proficiency in program evaluation is more important than ever as school counselors advocate for their role and for comprehensive school counseling program delivery in schools. This article presents a reexamination of the purpose and importance of program evaluation and the development of an evaluator identity. This quantitative study investigated which factors contribute to school counselors' identity as a program evaluator using self-report survey data on levels of self-efficacy, confidence, perceived importance of program evaluation, and interest in program evaluation. It also assessed the overall significance of each factor for engaging in program evaluation behaviors, as operationalized through data-based decision-making practices. Results include a predictive model that displays the contribution and significance of these factors on school counselor program evaluation and related data-based decision-making behaviors.

Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.

The quality of writing in a second language (L2) is one of the indicators of the level of proficiency for many college students to be eligible for departmental studies. Although certain software programs, such as Intelligent Essay Assessor or IntelliMetric, have been introduced to evaluate second-language writing quality, an overall assessment of writing proficiency is still largely achieved through trained human raters. The question that needs to be addressed today is whether generative artificial intelligence (AI) algorithms of large language models (LLMs) could facilitate and possibly replace human raters when it comes to the burdensome task of assessing student-written academic work. For this purpose, first-year college students (n=43) were given a paragraph writing task which was evaluated through the same writing criteria introduced to the generative pre-trained transformer, ChatGPT-3.5, and five human raters. The scores assigned by the five human raters revealed a statistically significant low to high positive correlation. A slight to fair but significant level of agreement was observed in the scores assigned by ChatGPT-3.5 and two of the human raters. The findings suggest that reliable results could be obtained when the scores of an application and multiple human raters are considered and that ChatGPT may potentially assist human raters in assessing L2 college writing.

Local tour guides' English speaking competence determines inbound international tourists' travel experiences. This study investigated student tour guides' English learning by adopting role-play tasks. The communicative tasks were designed to explore how real interactions prepared English as Foreign Language (EFL) learners to become cultural ambassadors who introduce Taiwanese features to the world. Twenty-nine students were recruited from an "English for Tourism Purposes" course to interact with international students who acted as tourists. The student tour guides took turns introducing a topic unique to Taiwan. Data included recorded conversational interactions between the international and local students, comments on tour-guiding performances, and reflective notes. Adopting conversation and thematic analysis, the researcher analyzed the objective learning outcomes by observing how students performed in the tour-guiding tasks and examined the subjective viewpoints from the participants' reflective notes. The results revealed distinctive spoken features for communicative competence and the benefits of gaining a deeper intercultural understanding of how to communicate Taiwanese culture in English and developing various communication skills for interacting with foreigners. Pedagogical implications are discussed in terms of the design of English for Tourism teaching.

One of the main generic competencies in Higher Education is "Learning to Learn". The key component of this competence is the capacity for self-regulated learning (SRL). For this competence to be developed, peer feedback seems useful because it fosters evaluative judgement. Following the principles of peer feedback processes, an online peer feedback proposal was created in the context of complex tasks in several degree courses (n = 448 students) at the University of Barcelona. A specific didactic sequence supported by digital technologies through Moodle was designed and validated after a deep analysis of the theoretical underpinnings of SRL models. A peer review proposal was ratified by the students, who pointed out the benefits of both receiving and, especially, giving feedback. The integration of Moodle tools with external tools such as Personalized Learning Design was highly valued by reviewers (n = 3 expert judges) and the possibilities for students to monitor their own progress through task completion was revealed as a positive aspect. Furthermore, the creation of personalized alerts to support learning processes also received a very high rating. The final model includes the changes derived from the validation process and suggests a reliable technology-supported intervention in an LMS, which is transferable to a larger scale.

In this article, we focus on a set of remote activities regarding simulated juries, judges' evaluations, communication of verdicts, and a post-discussion conversation. These activities aimed to promote learning of the preservice teachers about the differences between remote and in-person teaching in a chemistry teaching methods course. The set of activities was developed at the beginning of the course, during the Pandemic COVID-19, and the interchanging of roles was applied by the teacher educator. These structured activities were accomplished through the preservice teachers' interactions and their interactions with the teacher educator by means of arguments, explanations, dialogues, and injunctions. We used a multi-level method for discourse analysis to map, sample, and analyze virtual interactions, which afforded the following results: 1) the preservice teachers strongly engaged in the remote activities and constructed arguments and counterarguments in the simulated jury activities, showing active roles as knowledge producers in all set of activities; 2) several themes and subthemes were developed; 3) the teacher educator assumed the roles of instructor, manager of the discussions, and commentator; 4) the preservice teachers evaluated positively their experience with the set of activities; 5) the application of the multi-level method allowed us to make explicit the discursive moves of the participants and a model for the simulated juries and related activities.

The award of a PhD degree is based on a rigorous examination process which is unique in that examiners can recommend that the degree is awarded subject to minor or major amendments to the thesis. In their examiner reports, examiners also include formative feedback that is intended to help the doctoral researchers improve their work. Using data from examiner reports from two faculties at the University of Malta (Arts and Science) for the years 2017-2018, the study looks at the formative feedback comments provided by doctoral examiners. Results suggest that the feedback comments provided can be described as editorial, instructional and reflective, with the most common type of feedback comments being editorial and instructional. This indicates that the focus of doctoral examiner reports is on giving advice to doctoral researchers so that it reaches the expected doctoral standards. This study points to: the need for a rethinking of the way in which feedback is provided in doctoral examiner reports so that the focus is shifted to reflective comments that lead to in-depth learning; and for more professional development for doctoral examiners to that they can make this shift in their provision of formative feedback.

This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on "G studies"--a method of breaking down measurement variance--and "D studies"--a predictive study of the impact on reliability when modifying the number of raters, items, or other facets that assist the field in adopting measurement designs that include comprehensibility, accentedness, and intelligibility. When data from a single audio sample per learner were subjected to D-studies, we find that both semantic differential and rubric scales for comprehensibility were reliable at the 0.90 level with about 15 trained raters or 50 untrained crowdsourced raters. In order to offer generalizable and dependable evaluations, empirically informed recommendations are given, including considerations for the number of speech samples rated, or the granularity of the scales for various assessment and research purposes.

Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment. The mean of the overall population correlation between automated and human scoring in essay writing was 0.78. The overall common d effect size was 0.001. Results from this meta-analysis indicated a strong relationship with no discrepancies between automated and human scoring. Both the I[superscript 2] and Q values suggested that the population correlation values studied seemed to be heterogeneous, in contrast to homogenous d effect sizes. Therefore, it is necessary to investigate the sources of the between-studies variations for r correlations. Practical implications for ways of reporting results of automatic-scoring systems research and limitations of the study are also discussed.

This paper provides an experimental study of interlanguage phonological characteristics of Chinese students learning Thai as a foreign language and the accentedness perceived by native Thai speakers. Both production and perception experiments were designed to see how Chinese students acoustically produced Thai final nasal consonants and how Thai native speakers perceived these Chinese-accented nasals. The production experiment compared the acoustic features of Thai final nasal consonants (i.e. /m/, /n/, and /[eng]/) produced by Chinese students and native Thai speakers (n = 5 in each group), who provided speech samples from a wordlist reading task, consisting of 28 words (840 tokens). Nasal acoustic properties of 840 tokens (duration, nasal murmurs, and formant transitions) were examined. The findings showed that the Chinese students produced significantly longer nasal duration and more drastic formant transitions compared to the native speakers. The perception experiment analyzed how native Thai raters (n = 10) rated speech samples concerning degrees of accentedness by using a 5-point Likert scale with 5 as the most native level. Based on this, the native Thai raters rated the Chinese students' speech as 3.22 on average, while native Thai speech was judged with an average score of 4.65, which demonstrated that native Thai raters could distinguish foreign speech from those pronounced by native speakers. To find out to what extent nasal acoustic characteristics in Chinese students' interlanguage phonology contributed to the degree of accentedness, stepwise regression analyses were utilized to discover that nasal duration was particularly important in accurately predicting accentedness in Thai with Chinese accents.

This study aims to examine the relationship between the productive knowledge of some lexical and phraseological indices and the quality of English as a Foreign Language (EFL) learners' writing. A sample of 120 expository essays, written by semesters 1 and 5 university students in a less proficient EFL context, are rated by human evaluators and automatically examined for the target indices. The results show that, unlike the index of lexical diversity, both indices of content word frequency and range could significantly discriminate between different proficiency levels. For the phraseological indices, both the proportions of rare and frequent bigrams yielded between-group differences, with higher proficiency students performing significantly better in both categories. Using a regression analysis, the results show that the use of rare and contextually restricted content words and the production of larger proportions of rare and frequent bigrams could be considered indicators of better writing proficiency. The study suggests implications for the teaching of EFL.

TESOL practitioners, especially in Asia, tend to believe that reliable assessment of students' L2 English speech can be done solely by L1 English native speakers with sufficient teaching and assessment experiences. Such a belief, however, may need to be reconsidered from a new perspective of "diversity and inclusivity." This study used data from the ICNALE Global Rating Archives, a newly compiled assessment dataset, to examine the degree of the effects of raters' L1, assessment experience, and teaching experience on their assessment of Chinese, Japanese, and Thai learners' L2 English speech. The quantitative analyses showed that (1) raters' L1 significantly influenced the assessment scores for all three learner groups, but the difference between native speaker raters and non-native speaker raters was not clear, (2) raters' assessment experience influenced the scores only for Japanese learners, and (3) raters' teaching experience did not significantly influence the assessment scores for any of the three learner groups. These findings, which cast doubt on the dependence on native speaker raters alone, suggest the need to involve a greater variety of raters in L2 speech assessment.

According to expectation violation theory, job applicants can be upgraded or downgraded during an interview when their accent does not match employers' speech expectations. Focusing on the employment of second language French job candidates in Québec, this study explored this issue dynamically in terms of how expectations may impact the trajectory of interview evaluations. Participants included 60 Québec French raters and 6 female job candidates applying to a waitress or pizza cook position, presented through their resumes as either first (L1) or second (L2) language French speakers. Each speaker's interview audios were presented to raters in expectancy-congruent and expectancy-incongruent scenarios. Raters first provided resume-based employability assessments, then two more evaluations throughout a typical sequence of interview questions. The congruent and incongruent scenarios revealed similar evaluation patterns, where the L2 French cook applicant's employability improved after initially being downgraded. Implications are discussed regarding listeners' readjustment of their perceptions following first-impression biases.

Assessment materials must be checked for errors before they are presented to candidates. Any errors have the potential to reduce validity. For example, in the most extreme cases, an error may turn an otherwise well-designed exam question into one that is impossible to answer. In Cambridge University Press & Assessment, assessment materials are checked by multiple assessment specialists across different stages during assessment development. While human checkers are critical to this process, we must acknowledge that there is ample research showing the shortcomings of being human (e.g., we have cognitive biases, and memory and attentional limitations). It is important to provide assessment checkers with tools that help overcome or mitigate these limitations. This article is about one type of checking tool -- checklists. We describe a research-informed, collaborative project to support assessors in performing their checks of exam papers. This project focused on redesigning the instructional, training and task materials provided to assessors. A key part of this was to design checklists for assessors to use when performing their checks. In this article, we focus primarily on the approach that we took for these checklists in order to draw readers' attention to the complexity that is involved in designing them and to provide a practical example of how research can be used strategically to inform key design decisions.

Differential rater severity (DRS), one prevalent case of differential rater functioning (aka rater bias or rater interaction) effects, manifests itself when a rater assigns unusually severe or lenient ratings, threatening the validity and fairness of rater-mediated assessment. Building on a many-facets Rasch measurement (MFRM) approach, this study aimed to detect whether teachers exercised DRS towards rating criteria and student subgroups (classroom, proficiency, and gender) in a high-stakes EFL classroom writing examination. Data were collected from three teachers who applied a four-point five-criteria analytic rubric to rate opinion essays written by 42 English-major undergraduates during the examination. Main findings revealed that the teachers were not uniform in their severity levels, with the most experienced teacher exhibiting the highest severity and the least experienced teacher exercising the lowest severity. Whilst the most experienced and most severe teacher exposed slight DRS towards student genders, the less experienced and less severe teachers exerted substantial DRS in reverse pattern towards rating criteria and student classrooms. Surprisingly, the less experienced teachers scored their own classroom students less severely but marked each other's classroom students more severely than expected. The current findings raise the attention and awareness of teachers, educators, and policymakers concerning the impact of rater effects on the validity and fairness of rater-mediated assessment in the classroom context.

The Canadian Agility and Movement Skill Assessment (CAMSA) is a valid and reliable circuit-based test of motor competence which can be used to assess children's skills in a live or recorded performance and then coded. We aimed to analyze the intrarater reliability of the CAMSA scores (total, time, and skill score) and time measured, by comparing the live audio with the video assessment method. We also aimed to assess the interrater reliability using both audio- and video coding on a sample of 177 Spanish children. We found moderate-to-excellent inter- and intrarater video-audio intraclass correlation coefficients for the CAMSA score, time measured, time score, and skill score. Nonsignificant differences were found between video and audio recordings in the CAMSA score, time measured, and time score. Our findings support the rationale that different raters and scoring methods can accurately assess the participants' motor competence level using the CAMSA Spanish version.

Higher education accreditation has spread internationally as a vehicle for quality assurance and improvement but is strongly influenced by accreditation practices in the United States. The organisational structure and processes of seven United States health professions accreditors were analysed to identify common characteristics that reflect general fitness for purpose and infrequent features that show potential to enhance the effectiveness of accreditation processes, increase engagement of and support by accredited institutions and programmes and provide greater transparency.

In an era of test-based accountability, school inspections can offer a more nuanced understanding of why schools fail. Yet, we have limited knowledge of how inspectors arrive at their decisions on school quality. Analyzing inspectors' decision-making can reveal the underlying views regarding school accountability and open opportunities for school improvement. We use a comparative case study of contrasting inspection systems in the United States, the Netherlands, and Argentina. Based on in-depth interviews with inspectors, our findings reveal that inspectors' sensemaking and decisions are strongly influenced by local culture, professional traditions, and views on school accountability. These contrasting processes illustrate trade-offs between rigid and flexible approaches to school inspection that have consequences for school improvement.

Classroom observational protocols, in which raters observe and score the quality of teachers' instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In this study, we present an item response theory model that improves precision of observation scores by accounting for multiple sources of measurement error. Unlike the sum-scoring approach typically used in practice, the model presented here identifies raters and protocol items that are more reliable and should therefore be weighted more in scores. We then show that teachers identified as substandard using our model versus the sum-scoring approach are oftentimes very different. Our results suggest that accountability systems could improve observation scores by increasing the scoring weights of items and raters that provide greater information.

Previous research that explores the effect of ethnicity in the perception of speaker accentedness and personality traits often finds that Asian appearance contributes to a more accented and less competent impression. Importantly, most of the work done to date employed only Caucasian first language-speaking listeners; moreover, ethnicity and gender were not considered together. This study describes a series of evaluations of nine Korean (Asian) and nine German (Caucasian) second language English speakers by 41 Chinese listeners. The participants were asked to rate each speaker on an accentedness and 12 personality scales under two conditions: audio-only and audiovisual. The accentedness result indicates that with the availability of visual information, Korean speakers were rated significantly more accented than German speakers and male speakers were rated less accented than females. As for personality evaluation, the result suggests that power and competence traits share a similar trend, with Korean and female speakers being rated lower when listeners can see them, but solidarity traits show a slightly different trend. In general, Chinese listeners' bias in relation to speaker ethnicity and gender in both accentedness and personality evaluation was highlighted by the use of visual information, and a relationship between linguistic and social information was demonstrated.

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment of major depression and binge eating disorder or bulimia nervosa (N = 212) to illustrate the limits of the intraclass coefficient. Randomly selected therapy sessions were rated for therapeutic alliance quality by independent observers using the well-validated Vanderbilt Psychotherapy Process Scale and Vanderbilt Therapeutic Alliance Scale. Scores on subscales related to therapist behavior were restricted, indicating consistent alliance-supportive actions. Inter-rater reliability estimates were low despite high agreement between raters, however. The use of Bland-Altman plots to visualize agreement and data spread is suggested as a useful tool for researchers, consistent with the ideal of exploring reliability from a number of perspectives.

This paper proposes to use depth perception to represent raters' decision in holistic evaluation of ESL essays, as an alternative medium to conventional form of numerical scores. The researchers verified the new method's accuracy and inter/intra-rater reliability by inviting 24 ESL teachers to perform different representations when rating 60 essays written by Chinese ESL learners. During numerical representation, the raters expressed their evaluation in the form of numbers, while with the DP-based method, the raters expressed their evaluation by marking distances with nail tags on a wood ruler with hidden scale marks. Then the researchers translated the distance results into numbers and compared the accuracy and inter/intra-rater consistency of the two approaches by referring to these essays' criteria scores from eight expert raters. The results showed that DP-based method could improve raters' performance, producing more similar results to experts' scores, with higher consistency both among different raters and within the same individual rater.

The current study investigated whether the difficulty in finding group differences in prosody between speakers with autism spectrum disorder (ASD) and neurotypical (NT) speakers might be explained by identifying different acoustic profiles of speakers which, while still perceived as atypical, might be characterized by different acoustic qualities. We modelled the speech from a selection of speakers (N = 26), with and without ASD, as a network of nodes defined by acoustic features. We used a community-detection algorithm to identify clusters of speakers who were acoustically similar and compared these clusters with atypicality ratings by naïve and expert human raters. Results identified three clusters: one primarily composed of speakers with ASD, one of mostly NT speakers, and one comprised of an even mixture of ASD and NT speakers. The human raters were highly reliable at distinguishing speakers with and without ASD, regardless of which cluster the speaker was in. These results suggest that community-detection methods using a network approach may complement commonly-employed human ratings to improve our understanding of the intonation profiles in ASD.

Purpose: This study builds a new system for automatically assessing learners' speech elicited from an oral discourse completion task (DCT), and evaluates the prediction capability of the system with a view to better understanding factors deemed influential in predicting speaking proficiency scores and the pedagogical implications of the system. Methodology: We developed a system with a tripartite structure using an automatic speech recogniser, a set of modules that compute a number of speech features, and a scoring model. In total, 210 participants with intermediate English language proficiency level were administered a multi-turn oral DCT, a task which closely resembles discourse in real-life situations. The collected speech and transcribed files were eyeballed and rated by human raters first. Eighty percent of the original dataset was then used to train and examine our prediction model against the remainder of the dataset. Findings: The exact agreement between human and machine scores was 72%, moderately high, and comparable to the literature on automated speech scoring. It could provide a basis for deploying the system in a low-stakes practice environment. Originality/value: This study makes a unique contribution to the wider scholarship, where the use of single-turn DCTs remains prevalent, by presenting a new reliable scoring system for learner speech using an automated DCT with multiple turns. It offers useful insight into ways in which the system could be used in a low-stakes environment, including foreign language classroom settings.

Psychological theory suggests that evaluators' individual values and traits play a fundamental role in evaluation practice, though few empirical studies have explored those constructs in evaluators. This paper describes an empirical study on evaluators' individual, work, and political values, as well as their personality traits to predict evaluation practice and methodological orientation. The results suggest evaluators value benevolence, achievement, and universalism; they lean socially liberal but are slightly more conservative on fiscal issues; and they tend to be conscientious, agreeable, and open to new experiences. In the workplace, evaluators value competence and opportunities for growth, as well as status and independence. These constructs did not statistically predict evaluation practice, though some workplace values and individual values predicted quantitative methodological orientation. We conclude by discussing strengths, limitations, and next steps for this line of research.

The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information, or resources. Gatekeepers limit both the participation of key community members and, therefore, the exchange of relevant information related to the evaluation - a process called gatekeeping. Little research attention has been placed on studying gatekeeping, resulting in a dearth of knowledge about the influence of gatekeeping on stakeholder-engaged evaluations and social-structural dynamics that potentially perpetuate gatekeeping practices. In this article, we propose a gatekeeping influence theory grounded in the findings from 14 interviews. With a constructed theory of gatekeeping, we document the emergent social-structural and relational dynamics involved in stakeholder-engaged evaluation, with a focus on evaluations that include community partners and members.

The emergence of ChatGPT as one of the most advanced chatbots and its ability to generate diverse data has given room for numerous discussions worldwide regarding its utility, particularly in advancing medical education and research. This study seeks to assess the performance of ChatGPT in medical biochemistry to evaluate its potential as an effective self-learning tool for medical students. This evaluation was carried out using the university examination question papers of both parts 1 and 2 of medical biochemistry which comprised theory and multiple choice questions (MCQs) accounting for a total of 100 in each part. The questions were used to interact with ChatGPT, and three raters independently reviewed and scored the answers to prevent bias in scoring. We conducted the inter-item correlation matrix and the interclass correlation between raters 1, 2, and 3. For MCQs, symmetric measures in the form of kappa value (a measure of agreement) were performed between raters 1, 2, and 3. ChatGPT generated relevant and appropriate answers to all questions along with explanations for MCQs. ChatGPT has "passed" the medical biochemistry university examination with an average score of 117 out of 200 (58%) in both papers. In Paper 1, ChatGPT has secured 60 ± 2.29 and 57 ± 4.36 in Paper 2. The kappa value for all the cross-analysis of Rater 1, Rater 2, and Rater 3 scores in MCQ was 1.000. The evaluation of ChatGPT as a self-learning tool in medical biochemistry has yielded important insights. While it is encouraging that ChatGPT has demonstrated proficiency in this area, the overall score of 58% indicates that there is work to be done. To unlock its full potential as a self-learning tool, ChatGPT must focus on generating not only accurate but also comprehensive and contextually relevant content.

Rater comments tend to be qualitatively analyzed to indicate raters' application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The data consisted of ratings on 987 essays by 36 raters (a total of 3948 analytic scores and 1974 rater comments) on a post-admission English Placement Test (EPT) at a large US university. We computed a set of comment-based features based on the analytic components and evaluative language the raters used to infer whether raters were aligned to the scale. For data triangulation, we performed correlation analyses between the MFRM measures of rater performance and the comment-based measures. Although the EPT raters showed overall satisfactory performance, we found meaningful associations between rater comments and performance features. In particular, raters with higher precision and fit to what the Rasch model predicts used more analytic components and used evaluative language more similar to the scale descriptors. These findings suggest that NLP techniques have the potential to help language testers analyze rater comments and understand rater behavior.

In this Brief Report, we describe an evaluation of and revisions to a rubric adapted from the Jacobs et al.'s (1981) ESL COMPOSITION PROFILE, with four rubric categories and 20-point rating scales, in the context of an intensive English program writing placement test. Analysis of 4 years of rating data (2016-2021, including 434 essays) using many-facet Rasch measurement demonstrated that the 20-point rating scales of the Jacobs et al. rubric functioned poorly due to (a) questionably small distinctions in writing quality between successive score categories and (b) the presence of several disordered categories. We reanalyzed the score data after collapsing the 20-point scales into 4-point scales to simulate a revision to the rubric. This reanalysis appeared promising, with well-ordered and distinct score categories, and only a trivial decrease in person separation reliability. After implementing this revision to the rubric, we examined data from recent administrations (2022-2023, including 93 essays) to evaluate scale functioning. As in the simulation, scale categories were well-ordered and distinct in operational rating. Moreover, no raters demonstrated exceedingly poor fit using the revised rubric. Findings hold implications for other programs adopting/adapting the PROFILE or a similar rubric.

This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings ([chi-squared] = 88.005, df = 3, p < 0.0001), with presentation mode having a stronger effect in the interaction than accent familiarity ([chi-squared] = 59.328, df = 1, p < 0.0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio-only or audio-visual modes of presentation should be evaluated through research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-COVID-19.

In the context of policy-driven language testing for citizenship, a growing body of research examines the political justifications and ethical implications of language requirements and test use. However, virtually no studies have looked at the role that language testers play in the evolution of language requirements. Critical gaps remain in our understanding of language testers' first-hand experiences interacting with policymakers and how they perceive the use of tests in public policy. We examined these questions using an exploratory design and semi-structured interviews with 28 test executives representing 25 exam boards in 20 European countries. The interviews were transcribed and double coded in NVivo (weighted kappa = 0.83) using a priori and inductive coding. We used a horizontal analysis to evaluate responses by participant and a vertical analysis to identify between-case themes. Findings indicate that language testers may benefit from policy literacy to form part of policy webs wherein they can influence instrumental decisions concerning language in migration policy.

Assessing the content of learners' compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical thinking skills and subject knowledge. This study aims to conceptualize the construct of content in CLIL by exploring subject specialists' perspectives on essays' content quality in a CLIL context. Eleven researchers of English as a lingua franca (ELF) rated the content quality of research-based argumentative essays on ELF submitted in a CLIL course and produced think-aloud protocols. This study explored some essay features that have not been considered relevant in language assessment but are essential in the CLIL context, including the accuracy of the content, presence and quality of research, and presence of elements required in academic essays. Furthermore, the findings of this study confirmed that the components of content often addressed in language assessment (e.g., elaboration and logicality) are pertinent to writing assessment in CLIL. The manner in which subject specialists construe the content quality of essays on their specialized discipline can deepen the current understanding of content in CLIL.

It is widely acknowledged in research that common criteria and aligned standards do not result in consistent assessment of such a complex performance as the final undergraduate thesis. Assessment is determined by examiners' understanding of rubrics and their views on thesis quality. There is still a gap in the research literature about how analytic and holistic judgments are made and how they are integrated in decisions about the final grade. This interview study aims to identify the sources of inconsistency in analytic and holistic assessment. Ten examiners assessed three final undergraduate theses from a primary school teacher education programme at a Swedish university. The analytic assessment focused on two parts: "theoretical framework" and "academic language" style, which revealed inconsistencies in assessment. The analysis of the holistic assessment showed how examiners weighted different dimensions of the thesis and how they made judgements for the final grade awarded. The findings reveal several sources of inconsistency, such as examiners' own constructs, their interpretations and expectations about students' ability to manage academic work. The study calls for further discussion about whether it is possible to make criterion-based assessment reliable.

Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay scoring (AES) and automated essay revising (AER) have emerged to alleviate this burden. However, current AES and AER models require large amounts of training data and lack generalizability, which makes them hard to implement in daily teaching activities. Moreover, online sites offering AES and AER services charge high fees and have security issues uploading student content. In light of these challenges and recognizing the advancements in large language models (LLMs), we aim to fill these research gaps by analyzing the performance of open-source LLMs when accomplishing AES and AER tasks. Using a human-scored essay dataset (n = 600) collected in an online assessment, we implemented zero-shot, few-shot, and p-tuning AES methods based on the LLMs and conducted a human-machine consistency check. We conducted a similarity test and a score difference test for the results of AER with LLMs support. The human-machine consistency check result shows that the performance of open-source LLMs with a 10 B parameter size in the AES task is close to that of some deep-learning baseline models, and it can be improved by integrating the comment with the score into the shot or training continuous prompts. The similarity test and score difference test results show that open-source LLMs can effectively accomplish the AER task, improving the quality of the essays while ensuring that the revision results are similar to the original essays. This study reveals a practical path to cost-effectively, time-efficiently, and content-safely assisting teachers with student essay scoring and revising using open-source LLMs.

All students at Norwegian universities and colleges have the right to complain about ordinary grading decisions. When an appeal is made, two new examiners are appointed, at least one of whom should be external. The handling of appeals in the current system is to be blind, meaning that the examiners handling the complaint should not be aware of the original grade. It also means that they are not to receive information regarding the examiners' reasoning for the original grade or the student's reasoning for the complaint. In this article, we discuss whether the current appeal grading system with blind grading is appropriate when ordinary appeal grading is conducted by two examiners. We argue that the students' legal rights are not in conflict with the principle that good decision-making processes should be well-informed before decisions are made. We conclude that, in cases where two assessors are used for the ordinary examination, we ought to abolish the blind aspect in the current appeal grading system.

In the context of the COVID-19 restrictions and the pivot to online teaching and learning, teacher educators were forced to consider new spaces for School Placement and the assessment of these new sites of practice. This paper explores the process of the redesigning of the assessment of school placement components from the perspective of ten teacher educators across five teacher education programmes in one university. Hybridity theory, 'third space', and figurational sociology allowed us to understand who and what influenced the redesigning of assessment practices. The three themes relating to assessment included: (i) A network of interdependent relationships influencing teacher educators' reimagining of assessment processes; (ii) The influence of external interdependent relationships and context(s); and (iii) The role(s) of assessor for the teacher educator. We explicitly argue for the need to continue to advocate and advance these practices to 'build Initial Teacher Education back better'.