Search

This paper describes methods and findings from student cognitive labs, teacher cognitive labs, and test administration observations as evidence evaluated in a validity argument for a computer-based alternate assessment for students with significant cognitive disabilities. Validity of score interpretations and uses for alternate assessments based on alternate academic achievement standards (AA-AAAS) for students with the most significant cognitive disabilities require nontraditional sources of evidence about student-item interactions and the influences teachers have on those interactions. Our findings provide evidence that the assessment has been designed so students can show what they know and can do on multiple choice, technology-enhanced, and teacher administered item types and that teachers administer the assessments in a way that allows students to respond as intended. We conclude with a discussion on how the findings inform future test development, limitations, and implications for the use of these research methods for gathering validity evidence for an AA-AAAS.

Many 4-year public institutions face significant pedagogical challenges due to the high ratio of students to teaching team members. To address the issue, we developed a workflow using the programming language R as a method to rapidly grade multiple-choice questions, adjust for errors, and grade answer-dependent style multiple-choice questions, thus shifting the teaching teams' time commitment back to student interaction. We provide an example of answer-dependent style multiple-choice questions and demonstrate how the output allows for discrete analysis of questions based on various categories such as Fundamental Statements or Bloom's Taxonomy Levels. Additionally, we show how student demographics can be easily integrated to yield a holistic perspective on student performance in a course. The workflow offers dynamic grading opportunities for multiple-choice questions and versatility through its adaptability to assessment analyses. This approach to multiple-choice questions allows instructors to pinpoint factors affecting student performance and respond to changes to foster a healthy learning environment.

This research goal to develop a multiple-choice closed-ended test to assessing and evaluate students' digital literacy skills. The sample in this study were students at MTsN 1 Blitar City who were selected using a purposive sampling technique. The test was also validated by experts, namely 2 Doctors of Physics and Science from Yogyakarta State University. The test instrument was developed based on five aspects of digital literacy skills: information, communication, content creation, security and problem-solving. Data have been analyzed descriptively and inferentially using the Rasch version and the assist of Quest software. The results showed that eight multiple-choice closed-ended test instruments were declared valid based on expert validation with an Aiken V value of 1.00. The reliability result is 0.97 with a very high category, and the INFIT MNSQ standard deviation value is 0.86-1.16, so seven items are by the Rasch model. Thus, the seven items in the multiple-choice closed-ended test instrument can be used to assessing and evaluate students' digital literacy skills in learning science.

This research uses Rasch model analysis to identify the reliability and separation index of an integrated mathematics test instrument with a cultural architecture structure in measuring students' mathematical thinking abilities. The study involved 357 students from six eighth-grade public junior high schools in Bima. The selection of schools was based on average school exam scores and considered the effectiveness of the learning process that used cultural settings to explore mathematical content. Data analysis was conducted using Microsoft Excel to calculate the content validity of Aiken's index with four experts and the jMetrik software to measure reliability and the separation index. The research results indicate that the mathematics test instrument passed validation by mathematics experts and measurements with a valid content validity level. Rasch model calibration shows a very high level of instrument reliability. Separation analysis on the logit scale indicates the instrument's ability to differentiate students with different ability levels with good homogeneity in the distribution of test items and individual abilities. Scale quality statistics show good item response variability, low error rates and a high separation index. This study has limitations because it focuses solely on multiple-choice questions. Similar research must be conducted using other types of questions (such as those used in PISA, namely openconstructed and closed-constructed questions) and integrating other mathematical materials within relevant cultural architectural structures.

The extent to which L2 television is viewed by foreign language learners will depend on the degree to which it is understood. The addition of captions has been shown to support comprehension (e.g., Birulés-Muntané & Soto-Faraco, 2016; Montero-Perez, Peters, & Desmet, 2014), especially when proficiency is low (e.g., Lavaur & Bairstow, 2011). Yet, little is known about the extent to which captions benefit comprehension as L2 proficiency increases. This study seeks to investigate the effect of captions at different proficiency levels, and to identify the level at which captions cease to enhance comprehension. A total of 250 Catalan/Spanish university students, who had L2 English proficiency ranging from A1 to C2, viewed nine episodes of an English TV series with and without captions. Results showed that captioned viewing had a significant advantage over uncaptioned viewing in comprehension tests with multiple-choice and true-false items, and that learners with higher L2 proficiency and larger vocabulary performed better. While having access to captions increased the odds of a correct response independently of learners' L2 proficiency and vocabulary knowledge, the additive benefits of captions were no longer significant at the C2 level, suggesting a threshold beyond which uncaptioned viewing does not negatively impact comprehension. Pedagogical implications are discussed.

This study aimed to develop a valid and reliable multiple-choice achievement test for the subject area of ecology. The study was conducted within the framework of exploratory sequential design based on mixed research methods, and the study group consisted of a total of 250 middle school students studying at the sixth and seventh grade level. In the process of preparing the question pool, the level and scope of the questions in the pool were shaped by taking into account the achievements of science, environmental education and climate change courses. A question pool consisting of questions obtained as a result of the literature review and prepared by the researcher was created. The pool of 41 questions was reduced to 29 questions in line with expert opinions. The 29-question multiple-choice test, whose language comprehensibility was examined and necessary arrangements were made, was made ready for implementation. The draft form of the achievement test, which was piloted, was applied to 250 students. Item difficulty indices and item discrimination indices of the test answers obtained from the students were calculated. After the item analysis, a total of 4 questions were removed from the test. As a result of the analyses, the average difficulty value of the test was calculated as 0.58 and the average discrimination value was calculated as 0.60. The KR-20 reliability coefficient of the developed test was 0.85 and Cronbach alpha reliability coefficient was 0.88. As a result of the research, a valid and reliable ecology achievement test was developed.

ChatGPT has surged interest to cause people to look for its use in different tasks. However, before allowing it to replace humans, its capabilities should be investigated. As ChatGPT has potential for use in testing and assessment, this study aims to investigate the questions generated by ChatGPT by comparing them to those written by a course instructor. To investigate this issue, this study involved 36 junior students who took a practice test including 20 multiple-choice items generated by ChatGPT and 20 others by the course instructor, resulting in a 40-item test. Results indicate that there was an acceptable degree of consistency between the ChatGPT and the course instructor. Post-hoc analyses point to consistency between the instructor and the chatbot in item difficulty, yet the chatbot's results were weaker in item discrimination power and distractor analysis. This indicates that ChatGPT can potentially generate multiple-choice exams similar to those of the course instructor.

The Vocabulary Size Test (VST) measures English learners' decontextualised receptive vocabulary knowledge of written English and has nine bilingual versions with multiple-choice options written in other languages. This study used the English-Japanese version of the VST to investigate the extent to which loanword items were answered correctly by Japanese first language (L1) university students compared to non-loanword items, and whether it was easier to answer these loanword items when the correct answer option was written in loanwords rather than Japanese-words. Paired t-tests showed a significant difference in correct response rates between the loanword and non-loanword items, and the loanword options and Japanese-word options, with a large effect size. The results suggest the relative ease of learning English loanwords compared to non-loanwords for L1 Japanese users, and the need to consider the use of loanwords in vocabulary tests to measure test-takers' vocabulary size more accurately.

The present study has aimed to develop and validate a protein concept inventory (PCI) consisting of 25 multiple-choice (MC) questions to assess students' understanding of protein, which is a fundamental concept across different biology disciplines. The development process of the PCI involved a literature review to identify protein-related content, validation interviews to iteratively validate and refine the created items (n = 26), and data collection from a large sample (n = 291) for statistical analysis. An expert interview was held with two different field experts regarding the content validity of the draft PCI tool, the suitability of the options, and the clarity of the items. Free choice format (multiple marking) was used to answer the developed MC items. In scoring these items, positive points were given to correct options, and negative points were given to incorrect options. Evidence regarding the psychometric properties of the PCI trial form was collected through factor analysis, group differentiation, internal consistency, and item analysis using quantitative data. The evidence collected demonstrates that the validity and reliability of the PCI as a measurement tool have been confirmed. PCI's scoring approach and the use of response patterns created by multiple markings in teaching are discussed.

The 21st century learning demanded students to master the 4C competencies: critical thinking and problem-solving, communication, collaboration, and creativity and innovation, with problem-solving skills significantly influencing students' cognitive learning outcomes. This study utilized the Flipped Classroom instructional model, assisted by the Khan Academy application, as a key factor in educational change. This research aimed to determine the effects related to the implementation of the Flipped Classroom instructional model with the assistance of the Khan Academy application on students' problem-solving abilities and cognitive learning outcomes. The research design employed was quasi-experimental with data collection through observation, interviews, tests, and documentation. The results obtained through Independent Sample t-Test in measuring problem-solving abilities indicated that the Sig. value (2-tailed) was 0.028 for the first indicator (understanding the problem); 0.028 for the second indicator (planning the solution); 0.000 for the third indicator (implementing the solution plan); and 0.001 for the last indicator (checking back). This meant that the results were < 0.05 for each indicator. Another result, namely the Analysis of Covariance (ANCOVA) for cognitive learning outcomes was 0.000. This result showed that the Sig. value was < 0.05. Both results indicated that there was an influence of implementing the Flipped Classroom instructional model with the assistance of the Khan Academy application on student's problem-solving abilities and cognitive learning outcomes of class X students at SMAN 1 Kutorejo regarding the topic of viruses and their roles.

High-order cognitive skills (HOCS) are required to solve problems and make decisions. ARVi or augmented reality virus is a learning medium used to enhance HOCS. Aims of this research is to find out the effect of implementing ARVi media on enhancing high-level cognitive skills of students. The research is a quasi-experiment with a pretest-posttest control group design. The research was conducted in MAN 1 Kota Bogor class 10 using the Merdeka curriculum. The research instrument is a multiple-choice test that has been validated to measure high-level cognitive skills. The result of the t-test shows t-value (3.14) > t-table (1.98), indicating that there is a significant difference between the posttest scores of the control group and the experimental group. This means that the use of ARVi media has an effect on high-level cognitive skills. The improvement in high-level cognitive skills was continuous in C5 (evaluation) at 78%, C6 (creation) at 72%, and C4 (analysis) at 70%. These results are not independent of the support provided by the various content offered by ARVi media, such as augmented reality, literacy, and high-level questionnaires. Current education can benefit from technologies like ARVi media that help increase the high-level cognitive skills of students.

In this study, it was aimed to develop an achievement test taking into account the subject acquisitions of the sound and properties unit in the sixth-grade science course. In the test development phase, firstly, literature review for the study was conducted. Then, 30 multiple choice questions in align with the subject acquisition in the 2018 science curriculum. This 30-question test was presented to opinion of three academicians and a science teacher for the validity of the test. Necessary adjustments were made in line with the opinions of the experts. Subsequently, the draft test, modified according to the experts' feedback, was applied to a total of 300 seventh grade students. After the analysis, the number of questions in the test was reduced to 27. As a result of item analysis, mean difficulty index value of the test was 0.41, and item discrimination index was 0.49. The reliability analysis for the developed test was found 0.92 by calculating the KR-20 reliability coefficient value. By considering, the validity and reliability analysis results, it was concluded that the final version of the developed test, grounded in Bloom's Taxonomy, is a valid and reliable test with different difficulty levels.

The increasing popularity of the Common European Framework of Reference (CEFR) in non-native English-speaking countries has generated a demand for concrete examples in the creation of CEFR-based tests that assess the four main English skills. In response, this research endeavors to provide insight into the development and validation of a CEFR-based test aimed at evaluating undergraduate students' English proficiency for placement tests and exit exams. The CEFR served as the framework for item development while Classical Test Theory informed the test evaluation process. A sample of 2,248 first-year students participated in Testing 1 and 3,655 first- and second-year students took part in Testing 2. The results of the analysis of the multiple-choice listening and reading tests indicated favorable levels of item difficulty and discrimination indices, as well as high reliability coefficients obtained from Cronbach's alpha, Kuder-Richardson, and split-half reliability. The correlation and regression analyses revealed close relationships between the subtests and between each subtest and the total score, supporting the test's criterion validity. The study also demonstrated significant predictive validity on TOEIC scores. The findings of this study offer implications for the development of university-level English proficiency tests that integrate CEFR levels and CTT analysis.

This study aimed at testing the effectiveness of an e-module based on ProblemBased Learning combined with Socratic Dialogue on the topic of environmental change in improving school students' critical thinking skills. This research was a quasi-experimental study and used a non-randomised control group, pre-test/posttest design. The sampling technique was cluster random sampling. The sample in this study was two experimental classes (n = 72) and two control classes (n = 72). The data collection instruments included multiple-choice tests and a questionnaire. The test results were analysed using the t-test and the N-gain score test. The results showed significant difference in students' critical thinking skills between the experimental and control classes. Students who used the e-module showed better critical thinking skills. Students who used the e-module showed a greater increase in critical thinking skills after the learning process, compared to students who did not use the e-module. Thus, it can be concluded that the e-module based on PBL combined with Socratic Dialogue effectively improved students' critical thinking skills.

This design case details the design process of a multiple-choice assessment of socio-scientific systems thinking. This assessment is situated within a larger project that aims to understand the ways students use multiple scientific models to understand complex socio-scientific issues. In addition to the research component, this project entails developing curriculum and assessment resources that support science teaching and learning. We begin this paper by framing the needs that motivated the design of this assessment and introducing the design team. We then present a narrative outlining the design process, focusing on key challenges that arose and the ways these challenges influenced our final design. We conclude this paper with a discussion of the compromises that had to be made in the process of designing this instrument.

Understanding the concepts related to real function is essential in learning mathematics. To determine how students understand these concepts, it is necessary to have an appropriate measurement tool. In this paper, we have created a web application using 32 items from conceptual understanding of real functions (CURF) item bank. We conducted a psychometric analysis using Rasch model on 207 first-year students. The analysis showed that CURF is a dependable and valid instrument for measuring students' CURF. The test is uni-dimensional; all items are consistent with the construct and have excellent item fit statistics. The results indicate that the items are independent of each other and unbiased towards the gender and high school background of the students.

This paper aims to assess the impact of the reciprocal teaching strategy on the academic performance of student-teachers enrolled in the Higher Professional Diploma (HDE) program at Ajman University, United Arab Emirates. The study adopts a quasi-experimental design, employing pre- and post-tests and a control-experimental group configuration. The sample comprises 80 student-teachers selected from Ajman University during the second semester of the academic year 2022-2023. An academic achievement test in a multiple-choice format serves as the research instrument. The study's outcomes reveal noteworthy differences, indicating significant mean variations in the post-test scores between the experimental and control groups, favoring the control group in the context of the Reciprocal Teaching strategy. These results hold implications for international students, prompting considerations about the transferability and generalizability of teaching strategies across diverse cultural and educational settings. The findings suggest the broad effectiveness of the reciprocal teaching method, emphasizing the necessity of aligning teaching strategies with international students' specific needs and preferences. The research underscores the importance of tailoring teaching approaches to meet the unique requirements of a diverse cohort of global learners. It suggests that reciprocal teaching demonstrates efficacy across varied groups of international students, emphasizing the importance for instructors to select instructional methods that cater to the distinct needs of their diverse international student body.

Large introductory classes, with their expansive curriculum, demand assessment strategies that blend efficiency with reliability, prompting the consideration of multiple-choice (MC) tests as a viable option. Crafting a high-quality MC test, however, necessitates a meticulous process involving reflection on assessment format appropriateness, test blueprint design, and adherence to item-writing guidelines aligned with learning objectives and teaching strategies. This inherently time-consuming undertaking ideally requires a collaborative effort from a team of writers who possess expertise in both the subject domain and the specific course context -- an aspiration complicated by the multifaceted demands of higher education instruction. Given these challenges, educators often seek pragmatic solutions, including the adoption or adaptation of existing MC tests. However, the utility of these tests is ambiguous if the original test blueprint and the classification of questions are unknown. This paper introduces a structured four-step "reverse engineering" test blueprint process and proposes a systematic approach to identify test questions that align with the targeted learning objectives. One crucial step incorporates the Taxonomy Table (Anderson & Krathwohl, 2001) facilitating the classification of questions in the cognitive process dimension. As we delve into the intricacies of this analytical journey, we aim to provide a valuable resource for educators seeking to optimize the effectiveness and relevance of MC tests as a high-stakes assessment option.

Scientific thinking constitutes a vital component of scientific competencies, crucial for citizens to adapt to the evolving societal landscape. To cultivate students' scientific thinking, teachers should possess an adequate professional knowledge foundation, which encompasses pedagogical content knowledge (PCK). Assessing teachers' PCK of scientific thinking facilitates the development of effective curricula tailored to their continuous professional development. Despite its significance, empirical studies on biology teachers' PCK of scientific thinking are notably lacking. Hence, this research aimed to create a reliable and valid tool to evaluate upper-secondary school biology teachers' PCK of scientific thinking. The results showed that the instrument exhibits high reliability and good validity, affirming its efficiency for investigative purposes. A collective of 292 in-service biology teachers from upper-secondary schools participated in this investigation through the completion of an online survey. The results indicated that, overall, as well as for each component, upper-secondary school biology teachers' performance on PCK of scientific thinking fell within the lower to middle range. Specifically, the performance levels of four components: knowledge of students (KSU), knowledge of instructional strategies (KIS), knowledge of curriculum (KC), and knowledge of assessment (KA) declined sequentially.

Complex subjects such as physiology can be challenging for students to learn. These challenges are not uncommon in implementing the learning process in physiology and affect learning outcomes. Dramatization is an interactive and effective method to improve learning outcomes. In a project designed by senior medical students, junior medical students were guided in creating dramatizations related to three topics. Senior students were trained and assisted to prepare scenarios and make videos. The dramatizations were then carried out with junior medical students to help them better understand physiology and pathophysiology topics. A group of junior students receiving the same topics in a lecture format served as a control group. Pretest and posttest questionnaires were used to measure the improvement of learning outcomes. Assessment results showed an increase in performance in both groups. This study shows that dramatizations provide an effective alternative to lectures for instructing junior medical students.

Practical classes are an important and essential part of undergraduate programs in Chemical Engineering, as each experiment provides an opportunity to reinforce the theory of discrete unit operations that are taught elsewhere in the course. While an expensive pedagogical method, when practical sessions are delivered well, they can be one of the best learning experiences for students. As with all pedagogical methods, for students to gain maximum benefit of practical classes, a high level of engagement is required. Consequently, lab assignments need to be designed in a way that guides and instructs students on the theory, procedure, and risks associated with any practical and its associated assessments. This paper describes the outcomes of a qualitative investigation that evaluated student perceptions of updated prelab content combined with a new variation in postlab assessments and a renewed focus on practical skills during practical classes. The overall aim was to improve the student experience in practical classes. Paradoxically, periods of remote teaching enforced by the COVID-19 pandemic created further opportunities to make innovative changes to practical class resources. Subsequent student evaluations also indicated perceptions about each newly introduced component (instructional videos, online multiple-choice prelab quiz, variation in postlab assessment, introduction of grading rubrics, and a practical skills assessment), and more than 75% wanted these resources retained.

Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.

Diabetes mellitus and obesity are major public health issues that significantly impact the health care system. The next generation of health care providers will need a deep understanding of the pathophysiology of these diseases if we are to prevent, treat, and eventually cure these diseases and ease the burden on patients and the health care system. Physiology core concepts are a set of core principles, or "big ideas," identified by physiology educators that are thought to promote long-term retention, create a deeper understanding, and help with formation of critical thinking skills. Here we describe our scaffolded teaching approach in an upper year undergraduate pathophysiology course to educate students about these two diseases and discuss how learning about the basis of these highly integrative diseases from the biochemical to whole body level is a meaningful tool in the physiology educator toolbox to reinforce physiology core concepts. This teaching strategy is designed to engage students in the scientific process and hone their problem-solving skills such that they are hopefully equipped to treat and eventually cure these diseases as they move forward in their careers.

Causal reasoning is important to children's cognition and academic development. However, there have been few empirical studies on the impact of visual cues and non-verbal scaffolding on children's reasoning in continuous causal processes. Hence, the present study aims to explore how causal reasoning in continuous processes is facilitated by visual mind maps and multiple-choice questions through science experiments. By randomly selecting 136 children aged 9-13, the following results were obtained: Children provided with a mind map containing visual causal cues performed significantly better than the non-cue group on explanation tasks regardless of age differences, and children assessed using non-verbal multiple-choice questions scored significantly higher in explaining causal relationships than those using only verbal reports. This suggests that identification and explanation need to be differentiated for a more accurate evaluation of causal reasoning ability. These results have valuable implications for science curriculum and pedagogy at primary schools.

Emotional intelligence (EI) has a positive correlation with the academic performance of medical students. However, why there is a positive correlation needs further exploration. We hypothesized that the capability of answering higher-order knowledge questions (HOQs) is higher in students with higher EI. Hence, we assessed the correlation between EI and the capability of medical students to answer HOQs in physiology. First-year undergraduate medical students (n = 124) from an Indian medical college were recruited as a convenient sample. EI was assessed by the Schutte Self-Report Emotional Intelligence Test (SSEIT), a 33-item self-administered validated questionnaire. A specially designed objective examination with 15 lower-order and 15 higher-order multiple-choice questions was conducted. The correlation between the examination score and the EI score was tested by Pearson's correlation coefficient. Data from 92 students (33 females and 59 males) with a mean age of 20.14±1.87 yr were analyzed. Overall, students got a percentage of 53.37±14.07 in the examination, with 24.46±9.1 in HOQs and 28.91±6.58 in lower-order knowledge questions (LOQs). They had a mean score of 109.58±46.2 in SSEIT. The correlation coefficient of SSEIT score with total marks was r ¼ 0.29 (P ¼ 0.0037), with HOQs was r ¼ 0.41 (P < 0.0001), and with LOQs was r = 0.14 (P = 0.19). Hence, there is a positive correlation between EI and the capability of medical students to answer HOQs in physiology. This study may be the foundation for further exploration of the capability of answering HOQs in other subjects.

Digital automated assessment is a valuable and time-efficient tool for educators to provide immediate and objective feedback to learners. Automated assessment, however, puts high demands on the quality of the questions, alignment with the intended learning outcomes, and the quality of the feedback provided to the learners. We here describe the development and use of a large number (N > 180) of question banks with multiple items (N [greater than or equal to] 20) that are aligned to the intended learning outcomes of an undergraduate general chemistry course. Even though the development of a large number of high-quality question banks is a formidable task, they allow for flexible and easy-to-implement solutions for formative and summative assessment once developed. We here discuss three applications of the question banks: formative assessment in large online classes, practice tests that provide formative assessment outside classes, and summative assessment through a digital exam. We highlight the importance of aligning the question banks with intended learning outcomes, providing additional feedback to the learners and of quality assurance of the question banks, and show how the combined use of the question banks supports student self-regulated learning. We hope that the present work can inspire others to discover the various applications of question banks for formative and summative assessment.

This research utilizes a unique, validated, multiple-choice exam design that allows researchers to observe and measure the degree to which students copy answers from their peers. Using data collected from the exam, this study investigates whether asking students to sign an honor code at the start of the exam reduced instances of cheating relative to a control group. Empirical results demonstrate that the classroom of students who signed the honor code had less overall copying of answers and also fewer individual students who engaged in cheating. This study contributes to the literature by conducting analysis on a sample that measures actual cheating frequency, as opposed to relying on reported cheating in surveys, and by demonstrating that a positive intervention can effectively discourage unethical behavior.

The Multiple-choice (MC) item format is commonly used in educational assessments due to its economy and effectiveness across a variety of content domains. However, numerous studies have examined the quality of MC items in high-stakes and higher-education assessments and found many flawed items, especially in terms of distractors. These faulty items lead to misleading insights about the performance of students and the final decisions. The analysis of distractors is typically conducted in educational assessments with multiple-choice items to ensure high-quality items are used as the basis of inference. Item response theory (IRT) and Rasch models have received little attention for analyzing distractors. For that reason, the purpose of the present study was to apply the Rasch model, to a grammar test to analyze items' distractors of the test. To achieve this, the current study investigated the quality of 10 instructor-written MC grammar items used in an undergraduate final exam, using the items responses of 310 English as a foreign language (EFL) students who had taken part in an advanced grammar course. The results showed an acceptable fit to the Rasch model and high reliability. Malfunctioning distractors were identified.

Testing organizations are faced with increasing demand to provide subscores in addition to the total test score. However, psychometricians argue that most subscores do not have added value to be worth reporting. To have added value, subscores need to meet a number of criteria: they should be reliable, distinctive, and distinct from each other and from the total score. In this study, the quality of subscores from two subscore augmentation models (Wainer and Yen) was compared in terms of distinctness and variability. The reliabilities of the Wainer-augmented subscores were also examined. The models were applied to a high-stakes English language proficiency test in Iran. The results of the study showed that Yen better-satisfied subscore distinctness while Wainer best-preserved variability and had high-reliability subscores. In other words, Yen-augmented subscores had lower correlations while Wainer-augmented subscores better discriminated examinees with different ability levels. Thus, none of the examined models of subscoring satisfied all criteria. The results of the study are discussed and suggestions for future research are provided.

This study expands arguments calling for a more rigorous approach to high-frequency vocabulary list-based learning in EFL learning environments. Test and flashcard item designs were validated through quantitative (midterm and final test) and qualitative (survey) results to explore the impact of digital flashcards designed to build recall-level comprehension on both timed gap-fill and traditional multiple-choice posttests. Quizlet was chosen as the platform due to the affordance it provides teachers to create flashcard content and monitor practice. The results showed that multiple-choice, recognition-level test items result in a 20% overestimation of knowledge relative to gap-fill posttests. Additionally, a post-semester survey of 138 Japanese, pre-intermediate students of English showed a highly positive response to the recall-focused practice and testing system. The results demonstrate that for high-frequency L2 vocabulary, a paradigm shift from form-meaning recognition to form-meaning recall is an important direction for high-frequency vocabulary instruction and testing.

This study addressed a gap in existing research on Multiple-Choice (MC) cloze tests by focusing on the learners' perspective, specifically examining the difficulties faced by vocational high school students (VHSs). A nationwide sample of 293 VHSs participated, providing both quantitative and qualitative data through a self-developed questionnaire. The results revealed that vocabulary and grammar posed the greatest challenges in the MC cloze test, while sentence patterns were perceived as the least difficult by VHSs. Factors contributing to these difficulties included the need for increased focus on vocabulary and grammar learning. Some participants attributed challenges to personal perceptions of intellectual capability, while others highlighted the influential role of teachers' attitudes on their learning motivations and outcomes. The study suggested implications for test designs and teaching approaches. Despite these contributions, the study acknowledged limitations and offered suggestions for future research directions.

Previous studies on lexical focus on form (FonF) have mostly centred on FonF in reading with a few pre-selected lexical items. This study investigated the contribution of oral incidental FonF to developing learners' lexical knowledge in a free discussion EFL class. Incidental FonF was provided to 15 upper-intermediate learners who participated in 10 sessions of a meaning-oriented class. To gauge the retention rate of the lexical focus on form episodes (FFEs), two individualized multiple-choice achievement tests based on the lexical FFEs the learners reported to have no previous knowledge of in their uptake sheets were administered every five sessions, and two delayed posttests were given five weeks following each immediate posttest. Moreover, the participants were asked to compose two prompt-based writings on topics selected out of the covered themes in the posttests. Thematically relevant lexical FFEs extracted from each learner's uptake sheets made up the tailored prompts per learner. Results indicated rather high effectiveness of oral incidental FonF in developing learners' receptive and productive lexical knowledge in both short and long terms. The findings also revealed that learners fail to develop knowledge of the grammatical aspects of some lexical FFEs if their attention is merely drawn to their meaning aspects.

The purpose of this study is to introduce a method for converting scored 4-option multiple-choice (MC) items into scored 3-option MC items without re-pretesting the 3-option MC items. This study describes a six-step process for achieving this goal. Data from a professional credentialing exam was used in this study and the method was applied to 24 forms of the exam. The results found 100% accuracy in predicting the rounded passing score for all forms.

The impacts of a technology-integrated formative assessment technique on students' conceptual and procedural knowledge in studying chemical equilibrium are studied in this study. To attain the purpose, a quasi-experimental pretest-posttest strategy nested with a qualitative study method was adopted. The study has three groups: two experimental and one comparative. A random sample strategy was utilized to select two intact classes for treatment and one intact class for comparison groups. Data was collected using the chemical equilibrium conceptual test, the procedural test, and classroom observation. The data was examined using descriptive (mean and standard deviation) and inferential statistics (one-way ANOVA, one-way MANOVA, and Pearson product moment correlation). According to the findings, technology-integrated formative assessment processes outperformed traditional techniques and formative assessment strategies alone in enhancing students' conceptual and practical understanding of chemical equilibrium. Similarly, when technology-integrated formative assessment processes are used, classroom observations show that students have a strong motivation to learn and that the instructor is more skilled than the other two teachers. Technology-integrated formative assessment processes were shown to be more effective than the other two groups in promoting students' conceptual and procedural understanding when learning chemical equilibrium.

Currently, we are facing the rapid development of information and communication technology (ICT) that characterizes the 21st century. Challenges, problems, life, and careers in the 21st century can be successfully faced if we master the 21st century skills. However, nowadays it is still very rare to develop a 21st century skill measurement model, so it is necessary to develop test instruments to measure 21st century skills. The development of test instruments to measure 21st century skills in Indonesia is still very open by paying attention to the clarity of the instrument in describing real phenomena and complete data as material for problem solving. Based on the description above, the researchers developed a multiple-choice test based on experimental activities to measure students' 21st century skills which are creativity, critical thinking, collaboration, communication (4Cs) in heat and heat transfer using research and development (R&D) methods with the analyze, design, develop, implement, and evaluate (ADDIE) stage. Development begins with conducting literature and field studies, designing and making videos of experiment activities, compiling open-ended questions, distributing them to 12th graders of science, analyzing answers based on references, developing answer options, developing multiple choice tests, conducting validation by experts, and testing limited to students. The product of this research is expected to be used by educators and students for independent learning activities or as an example to develop multiple choice tests based on experimental activities to measure 21st century skills in other physics materials and other subjects.

This paper explores the impact of AI applications in adaptive learning on learners with varying levels of Chinese proficiency and diverse learning needs. The study investigates the robustness of VR technology in enhancing Chinese oral proficiency and pragmatic skills among 1st year and 2nd year learners. It compares the established VR app Mondly with Uptale, an Immersive VR authoring platform, focusing on AI-based voice recognition and analysis features. Additionally, it compares the adaptability of the Uptale VR platform with Mondly by analyzing factors such as difficulty adjustments, interactive features, learner engagement, and support for learner autonomy. It was found that VR technology, in general, enhances the Chinese learning experience. Uptale's additional features extend its potential utility in student assessment. These features include custom learning modules that incorporate students' self-shot 360-degree videos and interactive elements, such as multiple-choice questions, audio prompts, speaking/recording tags, feedback mechanisms, and analysis reports.

This study describes the characteristics of the test and its items used in the national-standardized school examination by applying classical test theory and focusing on the item difficulty, item discrimination, test reliability, and distractor analysis. We analyzed response data of 191 12th graders from one of public senior high schools in Yogyakarta City, Indonesia,to the examination on the elective mathematics subject. The results showed that both multiple-choice and essay items contained in the test were at a moderate level of difficulty. The lowest item difficulty index went to the multiple-choice item where students failed in interpreting straight and dashed lines and went to the essay item that required complex intra-mathematical connections. In addition, we only found one item which was poor in distinguishing student competence. Furthermore, students' test scores on multiple-choice and essay items were reliable. Although most multiple-choice items had at least two functioning distractors, it was still found two items whose all distractors were not functioning. In this article, we provide some suggestions concerning improvement towards mathematics learning practices.

Following Richards's (2015) ideas of using movie clips, videos, and games as valuable technological tools for language learning, the present study was conducted to investigate the effectiveness of using novel extracts and computer game cutscenes based on the same story to develop the vocabulary knowledge of English as a foreign language (EFL) learners. The study employed a quasi-experimental design with a control group and an experimental group. Initially, both groups studied a target vocabulary list of 20 words and their definitions. The participants were asked to complete a multiple-choice definition matching test after studying the list to find out whether their levels of vocabulary knowledge were comparable prior to the applications. For the experimental group, a second vocabulary test equivalent to the first one was conducted after they read a novel extract including the target vocabulary items, and a third test was carried out after they watched two computer game cutscenes that were based on the same events as the novel extract to understand whether seeing the target vocabulary items in these contexts repetitively would result in a significant increase in their target vocabulary knowledge. For the control group, the second and the third vocabulary tests were undertaken immediately after the first one to investigate whether mere exposure to equivalent vocabulary tests without the interventions would lead to a significant increase in their target vocabulary knowledge. The findings of the study indicated that the scores of the participants in the experimental group increased descriptively on the second test and significantly on the third test, whereas the scores of the participants in the control group decreased significantly on the second test and descriptively on the third test. The study provides practical and academic implications based on the findings.

The pandemic brought a whole newfound collection of words into our everyday language, some of which had been terms that were infrequently used outside academic and medical settings. However, as familiar as this new pandemic vocabulary has become, there are still several terms that often get mixed in everyday conversations, media communications, and even in the medical field. Some of these terms are often mistaken, which may lead to issues in the understanding of important concepts. Science communication, as a facet of scientific literacy, intends to educate people in an easy-to-understand manner, and this communication is even more important in health care, when a literate patient's medical decisions depend frequently on this process. In this article, we explore the use and misuse of some of the most common terms utilized during the 2019 coronavirus disease (COVID-19) pandemic by nursing and medical students, i.e., future health care professionals, in the hospital environment. This single-center hospital-based cross-sectional study, performed throughout September 2022, included 30 medical and nursing students. All participants completed a self-administered 15-item anonymous questionnaire at a single time point. Nine multiple-choice questions evaluated knowledge, diagnosis, disease manifestations, and vaccines related to the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus. Six questions provided demographics and quality assessment information. The analysis of the answers indicated that for at least 75% of the population surveyed, COVID-19 literacy was better for terms related to technology and actions (3/5 questions answered correctly) than for terms related to the disease (1/4 questions answered correctly). The overall median score for questions 1 to 9 was 4.5 of 9 points in total. Based on the results, the language used in the questionnaire was considered easy to understand, with an easy to medium level of complexity, and the perceived time required to complete the questionnaire was less than 5 min. In conclusion, our results showed that efforts need to be made in continuous professional education to increase the knowledge in COVID-19 literacy in the health care environment for medical and nursing students. Larger studies are recommended to identify and to fulfill the challenges that COVID-19 brought to medical and nursing education.

College students in a lower-division psychology course made metacognitive judgments by predicting and postdicting performance for true-false, multiple-choice, and fill-in-the-blank question sets on each of three exams. This study investigated which question format would result in the most accurate metacognitive judgments. Extending Koriat's (1997) cue-utilization framework to these judgments, each format gave students different cues on which to base judgments. Further, each format has different probabilities of correctly guessing, which can skew accuracy. Students reported the lowest estimates for fill-in-the-blank questions. Accuracy measured using bias scores showed students' predictions and postdictions were most accurate for multiple-choice items. Accuracy measured using gamma correlations showed students' predictions were most accurate for multiple-choice items and postdictions were most accurate for fill-in-the-blank items. Based on the findings, educators are encouraged to consider what implications question format have on metacognitive processes when testing students over studied material. And, for researchers, the findings support the use of different accuracy measures to get a more detailed understanding of factors influencing metacognitive judgment accuracy.

The goal of this study was to assess students' level of conceptual understanding of chemical kinetics and equilibrium using topic-specific pedagogical content knowledge (TSPCK)-based instruction in chemistry grade 11 students. The research was a mixed approach that involved 159 grade 11 chemistry class students from Addis Ababa administrative city, Ethiopia, and employed a mixture of qualitative and quantitative methodologies. Purposive sampling was used for the sample procedure and four intact classes were selected as three intervention groups and one control group: curricular saliency-based, representation-based, conceptual teaching strategy-based instruction, and comparison groups. The level of conceptual understanding was assessed using reasoned questions, specifically a two-tiered multiple-choice pretest, and posttest. The findings revealed that students' conceptual understanding of chemical kinetics was higher than that of chemical equilibrium concepts. According to the findings of the students' pretest and posttest, the level of conceptual understanding increased from 27.08% to 32.21% in chemical kinetics and from 15.91% to 23.68% in chemical equilibrium. Overall, representation-based instruction and conceptual teaching tactics boosted students' conceptual knowledge and accomplishment in chemical kinetics and equilibrium concepts. This study proposes that the level of conceptual understanding of grade 11 students in chemical kinetics should take into account the horizontal relationship between mathematics and the chemical kinetics concepts, which should be taught using technology-assisted teaching strategies such as conceptual teaching strategies and three-fold mental representations.

This study aimed to develop a useful test to measure university students' spatial abilities validly and reliably. Following a sequential explanatory mixed methods research design, first, qualitative methods were used to develop the trial items for the test; next, the psychometric properties of the test were analyzed through quantitative methods using data obtained from 456 university students. As a result, a multiple-choice spatial ability test with 27 items and five options was created, divided into three subtests: spatial relations, spatial visualization, and spatial orientation. The results suggested that scores obtained from the spatial ability test and its subtests are valid and reliable.

The goal of this study was to determine how the 3 Tier Multiple Choice Diagnostic Test developed in primary students' scientific understanding. This research and development includes fundamental studies, model development, and model testing. 161 fourth-graders served as research subjects. Tests, surveys, and observations are all used to collect data. The Miles and Huberman interactive model was used to analyze qualitative data, and the paired sample t-test and N-Gain Score test were used to analyze quantitative data. The findings indicate that 3 Tier Multiple Choice Diagnostic Test instrument has a V index of 0,80 and a Cronbach's Alpha value of 0,890, indicating that the instrument is valid. Thus, all test items are legitimate and dependable. The instrument was able to identify misconceptions regarding science learning in light material that fit the following profile based on the results of the trials: 28,7% of students do not grasp the idea of light sources; 43,3% have misunderstandings; 17,8% have incomplete comprehension with misconceptions; 20,2% have incomplete understanding without misconceptions; and 14,9% have complete understanding.

This study aims to describe the effect of the Nyerayo-based Science, Environment, Technology, and Society learning model on the scientific literacy skills of elementary school students in East Kluet District, South Aceh Regency. Learning Science, Environment, Technology, Society (SETS) has an important role in improving the scientific literacy and cooperation skills of Elementary School students. This research includes research that uses a quasi-experimental method with "nonequivalent control group design." The population in this study amounted to 182 fifth grade elementary school students and the sample of this study consisted of 17 male students and 37 female students with a total of 54 students. The research data was obtained through multiple choice test instruments on scientific literacy skills. The data obtained were analyzed using an independent sample t test. The results of this study indicate that "Nyerayo-based Science, Environment, Technology, and Society" learning has a significant effect on students' scientific literacy skills based on a significance value of 0.00 < 0.05.

This paper describes an experience based on the use of an active method in which students of a basic physics course prepare multiple choice questions (MCQs) to prepare for exams in the subject. The objective of the research was to provide the students with a method that would enhance their desire to learn physics, and consequently lead to an improvement in their meaningful learning. The participants were 57 first-year students from various engineering degrees. The quasi-experimental design with pre- and post-tests and a control group is described. The results showed that students who use the method of generating MCQs to prepare exams sometimes improve their learning significantly in comparison with the control group. The method also provides a new way to detect students' misconceptions about the concepts covered in class, which thus allows the evolution of their knowledge to be evaluated. Finally, it should be noted that the students valued the method used positively.