Search

The World Health Organization (WHO) officially declared the novel coronavirus (COVID-19) as a pandemic on 11 March 2020, and educational institutions have had to modify most of their activities (face-to-face activities were suspended). This situation forced academic institutions to modify the evaluation format of students. The use of proctoring systems quickly became widespread, although some controversies arose. The two main discussions regarding these systems are the integrity of the assessment and the capacity of the students to adapt to this new assessment method, without changes in theirs scores. To elucidate two controversies, we have analyzed the preferences and the scores obtained from a trial of 660 scores from 332 students of the third grade of Veterinary Medicine. The experiment involved three modalities of exam: an online format from home using the Respondus Lockdown Browser system (Modality 1), online in person using the Respondus Lockdown Browser system with the supervision of a teacher (Modality 2), or paper format in person with the supervision of a teacher (Modality 3). The results obtained showed that the students preferred Modality 1 (online at home with Respondus Lockdown Browser system). No statistical differences between the scores obtained by students were found between the three modalities analyzed. The proctoring system is a good method to adjudicate exams in higher education institutions, and the scores of students are similar to those obtained through traditional evaluation and control systems.

Testing as an assessment technique has been widely used at all levels of education--from primary to higher school. The main purpose of the paper is to evaluate the effect of context-based testing in teaching and learning of analytical chemistry in a Russian university. The paper formulates the objectives of context-based testing, discusses its features and compares with conventional testing; proposes a model of constructing and administering context-based testing; provides examples of context-based tests. The experiment was conducted at the Ural Sate University of Economics (Russia) with bachelor's degree students with nonscience majors. Students were assigned to one of the experimental conditions: in the control group, traditional tests were carried out, while the experimental group students experienced context-based testing. The statistical data--students' test scores--obtained at all stages of the experiment were analyzed on the basis of statistical criteria (Shapiro-Wilk, Student's T, Fisher). The findings of our experiment enable us to answer the guided research questions. Context-based testing may be considered as an essential component of context-based teaching and learning. In comparison with conventional testing, context-based testing could impact developing knowledge of fundamental analytical chemistry concepts and contribute to more solid knowledge.

Assessing learners' performance is crucial as it informs about learners' achievement levels in addition to their strengths and weaknesses, not only in face-to-face but also in online teaching. The study aimed at investigating the challenges faced by language teachers in assessing learners during online teaching and to offer some suggestions regarding language assessment courses in teacher training programs. The study adopted a qualitative approach to data collection and analysis. The participants included 22 language teachers aged between 24 to 48. Of these teachers, 8 were male, while the rest were female. Their teaching experience varied from 2 to 25 years, with an average of 11.8 years. The data was based on the responses of the participants during semi-structured interviews. The participants were asked to attend ZOOM meetings to be held, considering their availability. The interviews lasted 10 to 25 minutes. The results of the study indicated that the challenges faced by the participants included five major themes: Assessment type, assessment item formats, support, previous training (assessment literacy), and academic integrity. Several suggestions were offered considering online testing and assessment literacy and language assessment courses in teacher training programs.

The aim of this study is to determine students' opinions about open-ended question exam practice during 2018-2019 academic year for the following programs of Anadolu University Open Education System: Economy, Hospitality Management, Philosophy, History, Sociology, and Turkish Language and Literature. The study was designed as a quantitative study that describes an existing situation. The measurement tool developed to achieve that purpose includes 5 Likert type questions, 8 closed-ended question -- 3 of which are yes-no questions-, and 2 open-ended questions. The data were collected between February 14th 2020 and February 16th 2020 and 3236 students responded to the tool in total. Percentage-frequency tables and graphs were used for the analysis of the data collected. Since each question in the tool measured a different and independent dimension of the research, the tool was not taken as a scale, so each question was analyzed separately. The data were recoded for certain questions and chi-square analysis was done by using "age", "gender" and "the program they attend" variables. The results of the study showed that majority of the participants find open-ended questions more difficult than multiple-choice questions and have negative opinions about open-ended questions in terms of measuring subject matter competency and fair assessment.

This study examined information integration cognitive mechanisms underlying the test anxiety judgments of 474 engineering students. The experimental design considered the orthogonal combination of three factors (teaching style, exam type, and test mode), resulting in 12 experimental scenarios. During the experiments, participants were provided one scenario at a time and were asked to rate their anticipated anxiety level in the described situation. Subsequent analyses failed to reveal statistically significant differences in the anxiety levels reported by females and males. However, the factor selection and valuation female students adopted to make their anxiety judgments differed from those employed by their male peers. Cluster analysis identified three groups based on the anxiety level (low, medium, and high). The most relevant factor for all clusters was test mode, and only the medium anxiety group considered a second factor (exam type) to make their anxiety judgments, which was integrated through an additive cognitive rule. These findings suggest that participants place a higher weight on the examination context than its type when making their test anxiety judgments. Identifying these cognitive mechanisms underlying test anxiety could help regulate conditions that undermine the students' ability to cope with test anxiety.

Vocabulary learning is an incremental process. Vocabulary knowledge, especially for second-language learners, may develop across a lifetime. Teachers with experience in providing feedback on their students' vocabulary use in writing or speech might have noticed that it is sometimes difficult to pinpoint one aspect of word knowledge. The reason is that assessing learners' productive vocabulary involves unpredictable cases and/or inconsistent situations due to the multifaceted nature of word knowledge. Thus, identifying test formats that assess learners' productive vocabulary use can be difficult. This article presents five test formats that English language teachers can use or adapt to gauge their students' productive vocabulary knowledge.

The forced-choice (FC) item formats used for noncognitive tests typically develop a set of response options that measure different traits and instruct respondents to make judgments among these options in terms of their preference to control the response biases that are commonly observed in normative tests. Diagnostic classification models (DCMs) can provide information regarding the mastery status of test takers on latent discrete variables and are more commonly used for cognitive tests employed in educational settings than for noncognitive tests. The purpose of this study is to develop a new class of DCM for FC items under the higher-order DCM framework to meet the practical demands of simultaneously controlling for response biases and providing diagnostic classification information. By conducting a series of simulations and calibrating the model parameters with a Bayesian estimation, the study shows that, in general, the model parameters can be recovered satisfactorily with the use of long tests and large samples. More attributes improve the precision of the second-order latent trait estimation in a long test, but decrease the classification accuracy and the estimation quality of the structural parameters. When statements are allowed to load on two distinct attributes in paired comparison items, the specific-attribute condition produces better a parameter estimation than the overlap-attribute condition. Finally, an empirical analysis related to work-motivation measures is presented to demonstrate the applications and implications of the new model.

Data science and informatics tools are developing at a blistering rate, but their users often lack the educational background or resources to efficiently apply the methods to their research. Training resources and vignettes that accompany these tools often deprecate because their maintenance is not prioritized by funding, giving teams little time to devote to such endeavors. Our group has developed Open-source Tools for Training Resources (OTTR) to offer greater efficiency and flexibility for creating and maintaining these training resources. OTTR empowers creators to customize their work and allows for a simple workflow to publish using multiple platforms. OTTR allows content creators to publish training material to multiple massive online learner communities using familiar rendering mechanics. OTTR allows the incorporation of pedagogical practices like formative and summative assessments in the form of multiple choice questions and fill in the blank problems that are automatically graded. No local installation of any software is required to begin creating content with OTTR. Thus far, 15 training courses have been created with OTTR repository template. By using the OTTR system, the maintenance workload for updating these courses across platforms has been drastically reduced. For more information about OTTR and how to get started, go to ottrproject.org. Supplementary materials for this article are available online.

Assuring the quality of education is a concern of learning institutions. To do so, it is necessary to have assertive learning management, with consistent data on students' outcomes. This research provides associate deans and researchers, a roadmap with which to gather evidence to improve the quality of open-ended assessments. Based on statistical methods, we used our results to discuss the validity and consistency of six assessments applied in Insper--an AACSB accredited school in Brazil. The roadmap was capable of targeting aspects needing improvement, which is better than having to discard the assessments and developing new ones, and stimulating faculty awareness to consistent metrics to accurately manage the curriculum.

The effects of question or task complexity on second language speaking have traditionally been investigated using complexity, accuracy, and fluency measures. Response processes in speaking tests, however, may manifest in other ways, such as through nonverbal behavior. Eye behavior, in the form of averted gaze or blinking frequency, has been found to play an important role in regulating information in studies on human cognition, and it may therefore be an important subconscious signal of test question difficulty in language testing. In this study, 15 CEFR B2/C1-level-English learners took a Zoom-based English test with ten questions spanning six CEFR complexity levels. The participants' eye behaviors were recorded and analyzed between the moment the test question ended and the beginning of their response. The participants additionally provided self-report data on their perceptions of test-question difficulty. Results indicated that as test questions increased in difficulty, participants were more likely to avert their gaze from the interlocutor. They did not, however, blink more frequently as difficulty changed. These results have methodological implications for research on test validation and the study of nonverbal behavior in speaking tests.

Saving information onto external resources can improve memory for subsequent information--a phenomenon known as the saving-enhanced memory effect. This article reports two preregistered online experiments investigating (A) whether this effect holds when to-be-remembered information is presented before the saved information and (B) whether people choose the most advantageous strategy when given free choice of which information to save. Participants studied two lists of words; test order and whether and which list was saved (and re-presented again later) were manipulated. The saving-enhanced memory effect was only found when the first list (List A) was saved and tested after the second list (List B). When free to choose which list to save, participants preferred to save List A, but only when it was recalled after List B--that is, when it benefited memory. These findings suggest boundary conditions for the saving-enhanced memory effect and that people offload the most profitable information.

The Reading the Mind in the Eyes Test (RMET) is a purported theory of mind measure and one that reliably differentiates autistic and non-autistic individuals. However, concerns have been raised about the validity of the measure, with some researchers suggesting that the multiple-choice format of the RMET makes it susceptible to the undue influence of compensatory strategies and verbal ability. We compared the performance of autistic (N = 70) and non-autistic (N = 71) adults on the 10-item multiple-choice RMET to that of a free-report version of the RMET. Both the autistic and non-autistic groups performed much better on the multiple-choice than the free-report RMET, suggesting that the multiple-choice format enables the use of additional strategies. Although verbal IQ was correlated with both multiple-choice and free-report RMET performance, controlling for verbal IQ did not undermine the ability of either version to discriminate autistic and non-autistic participants. Both RMET formats also demonstrated convergent validity with a well-validated adult measure of theory of mind. The multiple-choice RMET is, however, much simpler to administer and score.

In online environments, listening involves being able to pause or replay the recording as needed. Previous research indicates that control over the listening input could improve the measurement accuracy of listening assessment. Self-pacing also supports the second language (L2) comprehension processes of test-takers with specific learning difficulties (SpLDs) or, more specifically, of learners with reading-related learning difficulties who might have slower processing speed and limited working memory capacity. Our study examined how L1 literacy skills influence L2 listening performance in the standard single-listening and self-paced administration mode of the listening section of the Test of English as a Foreign Language (TOEFL) Junior Standard test. In a counterbalanced design, 139 Austrian learners of English completed 15 items in a standard single-listening condition and another 15 in a self-paced condition. L1 literacy skills were assessed via a standard reading, non-word reading, word-naming, and non-word repetition test. Generalized Linear Mixed-Effects Modelling revealed that self-pacing had no statistically significant effect on listening scores nor did it boost the performance of test-takers with lower L1 literacy scores indicative of reading-related SpLDs. The results indicate that young test-takers might require training in self-pacing or that self-paced conditions may need to be carefully implemented when they are offered to candidates with SpLDs.

This article describes the implementation of a special accommodations policy for a suite of localised English language and numeracy tests, the Workplace Literacy and Numeracy (WPLN) Assessments. The WPLN are computer-delivered assessments, part of the WPLN training and assessment programme, which exists to provide access to workforce skills training across the Singaporean population. Operational and interview data were analysed to investigate three main areas: the extent to which WPLN accommodations are considered appropriate and effective for the test-taker population; the impact of the accommodations on test-takers' future opportunities; and the main factors perceived by stakeholders as playing an important role in test accommodations. The findings indicate that while the accommodations provided are generally considered to be appropriate and effective for the diverse WPLN test-taker population and facilitate improved future educational and workplace opportunities, some areas of the process are problematic or worthy of further consideration. Specifically, recommendations are made for future improvement in special accommodations policy development, dissemination, and implementation in large-scale test systems to safeguard access and inclusion.

Against the backdrop of internationalisation of education and the emergence of English as a global language, medium of instruction (MOI) has long been a controversial issue globally. The present study examined the relationships between secondary-level MOI (English, EMI; Chinese, CMI) and learners' tertiary-level academic achievement and perceptions. 434 tertiary-level business, science, and social science students in Hong Kong completed a survey and 18 of them also attended an interview. An overall significant difference was found in tertiary-level academic achievement between the EMI and CMI students. Significant differences were also found in their self-rated performance in and preferences for some but not all kinds of assessments. When analysed by tertiary-level disciplines, business students showed no differences while science students showed some, and social science students showed the greatest differences between the CMI and EMI groups. The interviewees highlighted academic and psychological challenges facing CMI students. Implications for educators and researchers are discussed.

The traditional format for neuroanatomy lab practical exams involves stations with a time limit for each station and inability to revisit stations. Timed exams have been associated with anxiety, which can lead to poor performance. In alignment with the universal design for learning (UDL), "Timed Image Question" and "Untimed Image Question" exam formats were designed to determine which format supports student success, especially for those who performed poorly in the traditional format. Only the "Untimed Image Question" format allowed students to revisit questions. All three formats were administered in a randomized order within a course for three cohorts of medical students. When all students' scores were analyzed together, the type of format had no effect. However, when analyses were conducted only on students who performed poorly in the traditional format, the type of format had an effect. These students increased their score, on average, by at least one grade level in the "Untimed Image Question" format compared to the traditional format. Students who performed well in the traditional format maintained their A, on average, in the two new formats. More students indicated "Untimed Image Question" as their most preferred format after experiencing all three formats. Most students associated the inability to revisit questions with high levels of anxiety. A neuroanatomy lab exam format was therefore identified as consistent with the UDL framework such that all students, regardless of test anxiety levels, can equally demonstrate what they learned. This format allowed for unlimited time per question and ability to revisit questions.

The aim of this study is to examine whether a preference for specific assessment methods in higher education is associated with personality and character strengths. Two-hundred and seventy Portuguese students completed a survey of character strengths, a Big Five personality test and their preference for each of six higher education assessment methods. Participants most favoured continuous assessment and multiple choice over viva voce and dissertations. Regression analysis showed that demographic factors, character strengths and personality accounted for 3-7% of the variance in the preferred examination method. Findings partially replicate previous investigations. Limitations and further research options are suggested.

The use of accommodations is often recommended to remove barriers to academic testing among English Learners (ELs). However, it is unclear whether accommodations are particularly effective at improving ELs' test scores. A growing foundation of empirical work has explored this topic. We conducted a meta-analysis that examined several possible moderators of accommodation effectiveness for improving EL test performance. Results showed substantial variability among estimates, an overall small positive significant effect on test scores and a statistically significant positive effect for indirect linguistic supports and combined accommodations compared to native language accommodations.

The Canadian English Language Proficiency Index Program (CELPIP) Test was designed for immigration and citizenship in Canada. CELPIP is a computer-based English-language proficiency test which covers all four skills. This test review provides a description of the test and its construct, tasks, and delivery. Then, it appraises CELPIP for reliability, fairness, and validity before identifying future directions for research.

We developed the Physics Inventory of Quantitative Literacy (PIQL) to assess students' quantitative reasoning in introductory physics contexts. The PIQL includes several "multiple-choice-multipleresponse" (MCMR) items (i.e., multiple-choice questions for which more than one response may be selected) as well as traditional single-response multiple-choice items. In this paper, we discuss differences in performance on MCMR items that seems to result from differences in administration method (paper versus online). In particular, we find a tendency for "clickiness" in online administration: students choose more responses to MCMR items when taking the electronic version of the assessment. Student performance on single-response multiple-choice items was not affected by administration method. These results suggest that MCMR items may provide a unique opportunity to probe differences in online and on-paper administration of low-stakes assessments.

Published work on vocabulary assessment has grown substantially in the last 10 years, but it is still somewhat outside the mainstream of the field. There has been a recent call for those developing vocabulary tests to apply professional standards to their work, especially in validating their instruments for specified purposes before releasing them for widespread use. A great deal of work on vocabulary assessment can be seen in terms of the somewhat problematic distinction between breadth and depth of vocabulary knowledge. Breadth refers to assessing vocabulary size, based on a large sample of words from a frequency list. New research is raising questions about the suitability of word frequency norms derived from large corpora, the choice of the word family as the unit of analysis, the selection of appropriate test formats, and the role of guessing in test-taker performance. Depth of knowledge goes beyond the basic form-meaning link to consider other aspects of word knowledge. The concept of word association has played a dominant role in the design of such tests, but there is a need to create test formats to assess knowledge of word parts as well as a range of multi-word items apart from collocation.

Among the variety of selected response formats used in L2 reading assessment, multiple-choice (MC) is the most commonly adopted, primarily due to its efficiency and objectiveness. Given the impact of assessment results on teaching and learning, it is necessary to investigate the degree to which the MC format reliably measures learners' L2 reading comprehension in the classroom context. While researchers have claimed that the longer the reading test (i.e., more test items and passages), the higher its overall reliability, few studies have investigated the optimal number of items and passages required for reliable classroom-based L2 reading assessment. To address this research gap, I adopted generalizability (G) theory to investigate the score reliability of the MC format in classroom-based L2 reading tests. A total of 108 ESL students at an American college completed an English reading test that included four passages, each of which was accompanied by five MC comprehension questions. The results showed that the score reliability of the L2 reading test was critically influenced by the number of items and passages, inasmuch as a different combination of the number of passages and items altered the degree of reliability. Implications for practitioners and educational researchers are discussed.

As access to smartphones and emerging technologies has become ubiquitous in our daily lives and in language learning, technology-mediated social interaction has become common in teaching and assessing L2 speaking. The changing ecology of L2 spoken interaction provides language educators and testers with opportunities for renewed test design and the gathering of context-sensitive validity evidence of interactive speaking assessment. First, I review the current research on interactive speaking assessment focusing on commonly used test formats and types of validity evidence. Second, I discuss recent research that reports the use of artificial intelligence and technologies in teaching and assessing speaking in order to understand how and what evidence of interactive speaking is elicited. Based on the discussion, I argue that it is critical to identify what features of interactive speaking are elicited depending on the types of technology-mediated interaction for valid assessment decisions in relation to intended uses. I further discuss opportunities and challenges for future research on test design and eliciting validity evidence of interactive speaking using technology-mediated interaction.

Purpose: By focusing on a Greek traditional learning university, during and post COVID-19 restrictions, the study aims at examining the concurrent effects of teaching and assessment format on students' academic performance. The inclusion of case studies in course assessment post COVID-19 restrictions is also expected to give a rough insight into students' employability skills and workplace readiness. Design/methodology/approach: The academic performance of 489 undergraduate students, as determined by the grades they earned, was measured in the final exams of business-to-business marketing and integrated marketing communication courses, held in January 2021 and 2022 at a Greek public university. The primary predictor variable or interest was "teaching and assessment format", and took two values: (1) online teaching with multiple-choice assessment format (during COVID-19 restrictions); and (2) traditional classroom teaching with in-person case study and open-ended assessment format (post COVID-19 restrictions). Findings: Grades were found to be affected by the participants' year of study, the type of marketing course, in which they were examined, and the teaching and assessment format employed. Either in whole or by gender, students appear to perform significantly worse in the traditional teaching with in-person case study and open-ended questions assessment format. Practical implications: Good pedagogical practice in the use of digital technology is advised to incorporate diverse teaching tools and assessment methods. Originality/value: Examination of the concurrent effects of teaching and assessment on academic performance unveils significant variation in students' academic performance under different formats, which may be attributable to multiple reasons.

With the exponential increase in the volume of data available in the 21st century, data literacy skills have become vitally important in work places and everyday life. This paper provides a systematic review of available data literacy assessments targeted at different audiences and educational levels. The results can help researchers and practitioners better understand the current state of data literacy assessments in terms of issues related to 1) educational levels and audiences; 2) data literacy definitions and competencies; 3) assessment types and item formats; and 4) reliability and validity evidence. The results from the present review led us to conclude that teaching and assessing data literacy is still an emerging field in education. Therefore, high-quality assessment tools are greatly needed to provide valuable insights for students and instructors to monitor progress as well as facilitate and support teaching and learning.

Careful considerations are necessary when there is a need to choose an anchor test form from a list of old test forms for equating under the random groups design. The choice of the anchor form potentially affects the accuracy of equated scores on new test forms. Few guidelines, however, can be found in the literature on choosing the anchor form. Five indices were proposed in this study to aid the selection and their performances were evaluated through a real-data-based simulation. The results shed some light on how to choose an anchor form in practice.

This study is a follow-up to previous research conducted in 2012 on computer-assisted language testing (CALT) that applied a survey approach to investigate the use of technology in sign language testing worldwide. The goal of the current study was to replicate the 2012 study and to obtain updated information on the use of technology in sign language testing. Another goal was to broaden the scope of the previous study by also considering the use of automatic sign language recognition and production (by avatars) in applied testing scenarios through technologies that make use of artificial intelligence (AI). Thirty-two sign language testing professionals participated in the current study. The results of the survey confirm the findings from 2012, but also raise new issues for future research (trends) related to sign language testing, such as the use of automatic sign language recognition for automatic scoring and the problem of acquiring funding for developing and maintaining web-based sign language testing platforms.

Research has suggested the important role of vocabulary knowledge in second language (L2) speaking proficiency. However, earlier studies tended to disregard the congruence in test format between assessing vocabulary knowledge and speaking skills with the former predominantly measured in written format. The current study measured vocabulary knowledge in spoken format to university students speaking English as an L2, and investigated whether spoken vocabulary knowledge predicts speaking proficiency. Forty-six university learners completed written and spoken forms of productive vocabulary test (i.e., Lex30) as well as a story narrative task. Elicited speech samples were rated in terms of four aspects of L2 speaking proficiency (fluency, vocabulary, pronunciation, grammar), and the rating scores were compared with productive vocabulary scores. Results showed a significant correlation between the spoken and written vocabulary scores with a closer examination of the data indicating a gap between the two forms. Results of the vocabulary-speaking link indicated that spoken vocabulary knowledge was associated with all but one of the L2 speech ratings, while written vocabulary knowledge was not related to any of the rating scores. The current study provided methodological and practical implications with respect to the central role of modality in vocabulary testing.

Purpose: The current study applies an innovative approach to the assessment of second language listening comprehension skills. This is an important focus in need of innovation because scores generated through language assessment tasks should reflect variation in the target skill and the literature broadly suggests that conventional methods of assessing listening may fall short of achieving this principle (Field [2019]. "Rethinking the Second Language Listening Test from Theory to Practice." Equinox). Design: The study investigated interactions between different methods of presenting listening comprehension questions, the focus of the comprehension questions and the relative authenticity of the sound file in an English-medium university entrance listening test. In a balanced design, 61 participants completed a listening test featuring both scripted and unscripted sound files by answering explicit and implicit information comprehension questions under five counterbalanced question preview and presentation conditions. Test scores were analysed using ANOVA and examined for interactions. Findings: The results revealed interesting relationships between text authenticity and item focus whereby item responses were most frequently correct overall on the explicit items on scripted tasks. However, the reverse was observed on the implicit items, which were more frequently correct on the unscripted tasks. Value: The research findings have important implications for construct definition and highlight possible directions for the development of pedagogical listening tasks and assessments for English-medium academic study.

This study investigates the relationship between take-home (open-book) examinations (THE) and in-class (closed-book) examinations (ICE) on academic performance and student wellbeing. Two social science courses (one bachelor and one master) were included in the study. In the first cohort (2019), students from both courses performed an ICE, whereas students in the second cohort (2020) performed a THE. Four to six months following course completion, students were approached to fill out a survey pertaining to their academic performance and wellbeing during the course, and to complete a test measuring knowledge retention on the course materials. No significant differences were found between the ICE and THE cohorts in academic performance and knowledge retention for either the bachelor or the master students. Bachelor students who completed a THE in 2020 reported significantly lower wellbeing in comparison to their peers completing the ICE a year earlier. The implications of the results in the context of the COVID-19 pandemic are discussed.

Background: Mode effects, the variations in item and scale properties attributed to the mode of test administration (paper vs. computer), have stimulated research around test equivalence and trend estimation in PISA. The PISA assessment framework provides the backbone to the interpretation of the results of the PISA test scores. However, an identified gap in the current literature is whether mode effects have affected test score interpretation as defined by the assessment framework, and whether the interpretations of the PBA and CBA test scores are comparable. Methods: This study uses the 2015 PISA field trial data from thirteen countries to compare test modes through a construct representation approach. It is investigated whether item facets defined by the assessment framework (e.g., different cognitive demands) affect item difficulty comparably across modes using a unidimensional two-group generalized partial credit model (GPCM). Results: Linking the assessment framework to item difficulty using linear regression showed that for both maths and science domains, item categorisation relates to item difficulty, however for the reading domain no such conclusion was possible. In comparing PBA to CBA in representations across the three domains, maths had one facet with a significant difference in representation, reading had all three facets significantly different, and for science, four out of six facets had significant differences. Modelling items labelled "mode invariant" in PISA 2015, the results indicated that in every domain, two facets showed significant differences between the test modes. The graphical inspection of difficulty patterns confirmed that reading shows stronger differences while the patterns of the other domains were quite consistent between modes. Conclusions: The present study shows that the mode effects on difficulty vary within the task facets proposed by the PISA assessment framework, in particular for reading. These findings shed light on whether the comparability of score interpretation between modes is compromised. Given the limitations of the link between the reading domain and item difficulty, any conclusions in this domain are limited. Importantly, the present study adds a new approach and empirical findings to the investigation of the cross-mode equivalence in PISA domains.

To assemble a high-quality test, psychometricians rely on subject matter experts (SMEs) to write high-quality items. However, SMEs are not typically given the opportunity to provide input on which content standards are most suitable for multiple-choice questions (MCQs). In the present study, we explored the relationship between perceived MCQ suitability for a given content standard and the associated item characteristics. Prior to item writing, we surveyed SMEs on MCQ suitability for each content standard. Following field testing, we then used SMEs' average ratings for each content standard to predict item characteristics for the tests. We analyzed multilevel models predicting item difficulty (p value), discrimination, and nonfunctioning distractor presence. Items were nested within courses and content standards. There was a curvilinear relationship between SMEs' ratings and item difficulty such that very low MCQ suitability ratings were predictive of easier items. After controlling for item difficulty, items with higher MCQ suitability ratings had higher discrimination and were less likely to have one or more nonfunctioning distractors. This research has practical implications for optimizing test blueprints. Additionally, psychometricians may use these ratings to better prepare for coaching SMEs during item writing.

Some statewide testing programs allow students to receive corrective feedback and revise their answers during testing. Despite its pedagogical benefits, the effects of providing revision opportunities remain unknown in the context of alternate assessments. Therefore, this study examined student data from a large-scale alternate assessment that allows students to make multiple attempts until they find the correct answer to multiple-choice items. The students receive partial credit based on the number of attempts being made. The effects of the multiple-attempt approach on both test characteristics and student performance were investigated. The results indicated that, despite making most items on the assessment relatively easier, the availability of partial credit improved the strength of the items in distinguishing low-achieving and high-achieving students while maintaining high internal consistency among the test items. Although the students were able to increase their scores due to the inclusion of partial credit based on the number of attempts, the relative positions of the students remained nearly the same.

In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced by other factors such as target population or input modality. Extending previous work, we distinguish "conceptual," "realization," and "nonconformity variance," which are differentially impacted by the various factors. While conceptual variance relates to different concepts embedded in the text responses, realization variance refers to their diverse manifestation through natural language. Nonconformity variance is added by aberrant response behavior. Furthermore, besides its performance, the feasibility of using an automatic scoring system depends on external factors, such as ethical or computational constraints, which influence whether a system with a given performance is accepted by stakeholders. Our work provides (i) a framework for assessment practitioners to decide a priori whether automatic content scoring can be successfully applied in a given setup as well as (ii) new empirical findings and the integration of empirical findings from the literature on factors that influence automatic systems' performance.

In this study, I examined how 34 Spanish-speaking English language learners (ELLs) used their linguistic resources (English and Spanish) and language modes (oral and written language) to demonstrate their knowledge of proportional reasoning in a dual language mathematics assessment task. The assessment allows students to see the item in both English and Spanish, to listen to someone read aloud the item in English or Spanish, to provide oral or written responses in English, Spanish, or both for selected constructed-response items. The assessment was piloted in three algebra classrooms at a middle school and a high school in the United States. Data included students' responses, the amount of time it took students to complete each item, the language that the students used in each item, the language mode they used to complete the constructed-response items, the number of times the students listened to the read alouds, and responses to a short survey. This study reports on how Spanish-speaking ELLs performed on the assessment task, the linguistic resources and the language modes they used to access and to respond to the task, and how language and mode use impacted performance and the time spend on the test items.

Nearly all undergraduate biology courses rely on quizzes and exams. Despite their prevalence, very little work has been done to explore how the framing of assessment questions may influence student performance and affect. Here, we conduct a quasi-random experimental study where students in different sections of the same course were given isomorphic questions that varied in their framing of experimental scenarios. One section was provided a description using the self-referential term "you", placing the student in the experiment; another section received the same scenario that used classmate names; while a third section's scenario integrated counterstereotypical scientist names. Our results demonstrate that there was no difference in performance throughout the semester between the sections, nor were there differences in students' self-reported stress and identity. However, students in all three sections indicated that they most preferred the self-referential framing, providing a variety of reasons that suggest that these variants may influence how well a student reads and processes the question. In addition, our results also indicate that the framing of these scenarios can also have a large impact on some students' affect and attitude toward the question. We conclude by discussing implications for the biology education research community and biology instructors.

The outbreak of the COVID-19 pandemic has transformed the educational landscape in a way unseen before. Educational institutions are navigating between offline and online learning worldwide. Computer-based testing is rapidly taking over paper-and-pencil testing as the dominant mode of assessment. In some settings, computer-based and paper-and-pencil assessments can also be offered side-by-side, in which case test developers should ensure the evidence of equivalence between both versions. This study aims to establish the equivalency evidence of different delivery modes of the English Competency Test, an English language assessment for civil service officers developed and used by the Human Resources Development Education and Training Center, a civil service training institution under the Ministry of Finance of the Republic of Indonesia. Psychometric analyses were carried out with the Rasch model to measure the unidimensionality, reliability, separation, and standard error of measurement estimates. The findings demonstrate that the paper-and-pencil and computer-based versions of the language assessment exhibit comparatively equivalent psychometric properties. The computer-based version of the English Competency Test is proven to offer a reliable and comparable alternative to the paper-and-pencil version.

In language assessment, AI technology has been incorporated in task design, assessment delivery, automated scoring of performance-based tasks, score reporting, and provision of feedback. AI technology is also used for collecting and analyzing performance data in language assessment validation. Research has been conducted to investigate the efficiency and functionality of assessment technologies, but empirical explorations on test-taker engagement in AI technology-mediated language assessment are remarkably scarce. In this commentary, we first examine the impact of AI technology on test takers, in terms of both the benefits and the challenges that it poses in the critical stages of language assessment development and validation. Next, we propose a conceptual model to facilitate the implementation and evaluation of test-taker engagement in AI technology-mediated language assessment. The model delineates two forms of test-taker engagement: test takers' participation in technology-mediated assessment activities and their perceptions of technological innovations in language assessment. We then review the articles in this special issue based on this model and discuss directions for future research. We conclude by offering some guidance for maximizing test-taker engagement in technological innovations, thereby promoting learning-oriented and equity-minded language assessment.

Numerous primary studies and a recent meta-analytic confirmatory factor analysis (Meta-CFA; Said-Metwaly, Fernández-Castilla, Kyndt, & Van den Noortgate, 2018) have shown that Torrance Tests of Creative Thinking -- Figural (TTCT-F) consists of two factors. However, recent research has raised questions regarding factor analysis of the TTCT-F due to poor discriminant validity and artificial clustering of indices. Additionally, the original meta-analysis did not include databases of dissertations and theses. In this study, we replicated and extended the meta-CFA study of the TTCT-F using 56 correlation matrices from 45 studies and a total of 11,142 participants. We investigated whether the problem of discriminant validity, participants' age, the use of Form A or B, and the use of standardized or raw scores influence the factor structure. Our results show that a two-factor structure remains stronger than a one-factor structure in both replication and extension studies, and this conclusion is robust to score type, participant age, and issues related to discriminant validity. However, further research is needed to determine if this structure applies to less commonly used Form B, which had different factor loadings from Form A. We discuss the implications of our results for creativity research, the psychometrics of TTCT-F, and gifted identification.

Computer-based test has been administered in e-learning environments as part of ICT integration in education. Recently, online test is gaining attention in both regular and distance education institutions, and students' preference or perception of an online test versus paper-based test is crucial in successful adoption or implementation of either of the test modes by any educational institution. For this reason, this study examined undergraduate students' computer usage experience as a prerequisite for online test completion, and their perception and preference toward online test as opposed to paper-based test. It also examined how students' perception of online test differed across sex, program major, and computer usage experiences. The findings would contribute to knowledge on factors for consideration, especially by faculty and instructional designers, in successful adoption of online test in undergraduate programmes. An online questionnaire was used to collect data from 213 undergraduate students at a university in Ghana, and the data were analyzed using median, standard deviation, Mann-Whitney U Test, Kruskal--Wallis Test, and Spearman correlation. The results indicated that students had high level of experience in computer usage, they had positive perception and preference for online test, there was no significant differences in perception of online test between female and male students, and among students of different majors. However, there was a positive correlation between computer use experience of the students and their perception of online test. It was recommended that universities seeking to implement online assessment ensure their students are equipped with adequate skills and experiences in computer usage.

Two-tiered assessment structures with paired content and confidence items are frequently used within chemistry assessments to stimulate and measure students' metacognition. The confidence judgment is designed to promote students' reflection on their application of content knowledge and can be characterized as calibrated or miscalibrated based on their accuracy. Previous studies often attributed students' miscalibrated confidence rankings to metaignorance, however, in this qualitative study, interviews with general chemistry students were thematically analysed to provide a more robust understanding of the processes and factors students use when engaging with these metacognitive prompts in a chemistry assessment. Both calibrated and miscalibrated confidence judgments were observed independent of accuracy. Students who provided miscalibrated confidence judgments often used unreliable metrics such as processing fluency which can mimic content mastery whereas students who provided more accurate evaluations of their confidence relied more heavily on their stable understanding of chemistry concepts. Many students cited previous experiences, underlying self-efficacy beliefs, and/or the use of test-taking strategies which negatively or positively impacted their confidence. These findings suggest that the confidence tier is indeed capturing students' self-assessment, however, students' confidence judgments are based on a range of factors independent of content knowledge which may impede on the utility of this metacognitive tool for students, researchers, and instructors.