Search

This paper describes methods and findings from student cognitive labs, teacher cognitive labs, and test administration observations as evidence evaluated in a validity argument for a computer-based alternate assessment for students with significant cognitive disabilities. Validity of score interpretations and uses for alternate assessments based on alternate academic achievement standards (AA-AAAS) for students with the most significant cognitive disabilities require nontraditional sources of evidence about student-item interactions and the influences teachers have on those interactions. Our findings provide evidence that the assessment has been designed so students can show what they know and can do on multiple choice, technology-enhanced, and teacher administered item types and that teachers administer the assessments in a way that allows students to respond as intended. We conclude with a discussion on how the findings inform future test development, limitations, and implications for the use of these research methods for gathering validity evidence for an AA-AAAS.

Many 4-year public institutions face significant pedagogical challenges due to the high ratio of students to teaching team members. To address the issue, we developed a workflow using the programming language R as a method to rapidly grade multiple-choice questions, adjust for errors, and grade answer-dependent style multiple-choice questions, thus shifting the teaching teams' time commitment back to student interaction. We provide an example of answer-dependent style multiple-choice questions and demonstrate how the output allows for discrete analysis of questions based on various categories such as Fundamental Statements or Bloom's Taxonomy Levels. Additionally, we show how student demographics can be easily integrated to yield a holistic perspective on student performance in a course. The workflow offers dynamic grading opportunities for multiple-choice questions and versatility through its adaptability to assessment analyses. This approach to multiple-choice questions allows instructors to pinpoint factors affecting student performance and respond to changes to foster a healthy learning environment.

This research goal to develop a multiple-choice closed-ended test to assessing and evaluate students' digital literacy skills. The sample in this study were students at MTsN 1 Blitar City who were selected using a purposive sampling technique. The test was also validated by experts, namely 2 Doctors of Physics and Science from Yogyakarta State University. The test instrument was developed based on five aspects of digital literacy skills: information, communication, content creation, security and problem-solving. Data have been analyzed descriptively and inferentially using the Rasch version and the assist of Quest software. The results showed that eight multiple-choice closed-ended test instruments were declared valid based on expert validation with an Aiken V value of 1.00. The reliability result is 0.97 with a very high category, and the INFIT MNSQ standard deviation value is 0.86-1.16, so seven items are by the Rasch model. Thus, the seven items in the multiple-choice closed-ended test instrument can be used to assessing and evaluate students' digital literacy skills in learning science.

This research uses Rasch model analysis to identify the reliability and separation index of an integrated mathematics test instrument with a cultural architecture structure in measuring students' mathematical thinking abilities. The study involved 357 students from six eighth-grade public junior high schools in Bima. The selection of schools was based on average school exam scores and considered the effectiveness of the learning process that used cultural settings to explore mathematical content. Data analysis was conducted using Microsoft Excel to calculate the content validity of Aiken's index with four experts and the jMetrik software to measure reliability and the separation index. The research results indicate that the mathematics test instrument passed validation by mathematics experts and measurements with a valid content validity level. Rasch model calibration shows a very high level of instrument reliability. Separation analysis on the logit scale indicates the instrument's ability to differentiate students with different ability levels with good homogeneity in the distribution of test items and individual abilities. Scale quality statistics show good item response variability, low error rates and a high separation index. This study has limitations because it focuses solely on multiple-choice questions. Similar research must be conducted using other types of questions (such as those used in PISA, namely openconstructed and closed-constructed questions) and integrating other mathematical materials within relevant cultural architectural structures.

The extent to which L2 television is viewed by foreign language learners will depend on the degree to which it is understood. The addition of captions has been shown to support comprehension (e.g., Birulés-Muntané & Soto-Faraco, 2016; Montero-Perez, Peters, & Desmet, 2014), especially when proficiency is low (e.g., Lavaur & Bairstow, 2011). Yet, little is known about the extent to which captions benefit comprehension as L2 proficiency increases. This study seeks to investigate the effect of captions at different proficiency levels, and to identify the level at which captions cease to enhance comprehension. A total of 250 Catalan/Spanish university students, who had L2 English proficiency ranging from A1 to C2, viewed nine episodes of an English TV series with and without captions. Results showed that captioned viewing had a significant advantage over uncaptioned viewing in comprehension tests with multiple-choice and true-false items, and that learners with higher L2 proficiency and larger vocabulary performed better. While having access to captions increased the odds of a correct response independently of learners' L2 proficiency and vocabulary knowledge, the additive benefits of captions were no longer significant at the C2 level, suggesting a threshold beyond which uncaptioned viewing does not negatively impact comprehension. Pedagogical implications are discussed.

This study aimed to develop a valid and reliable multiple-choice achievement test for the subject area of ecology. The study was conducted within the framework of exploratory sequential design based on mixed research methods, and the study group consisted of a total of 250 middle school students studying at the sixth and seventh grade level. In the process of preparing the question pool, the level and scope of the questions in the pool were shaped by taking into account the achievements of science, environmental education and climate change courses. A question pool consisting of questions obtained as a result of the literature review and prepared by the researcher was created. The pool of 41 questions was reduced to 29 questions in line with expert opinions. The 29-question multiple-choice test, whose language comprehensibility was examined and necessary arrangements were made, was made ready for implementation. The draft form of the achievement test, which was piloted, was applied to 250 students. Item difficulty indices and item discrimination indices of the test answers obtained from the students were calculated. After the item analysis, a total of 4 questions were removed from the test. As a result of the analyses, the average difficulty value of the test was calculated as 0.58 and the average discrimination value was calculated as 0.60. The KR-20 reliability coefficient of the developed test was 0.85 and Cronbach alpha reliability coefficient was 0.88. As a result of the research, a valid and reliable ecology achievement test was developed.

ChatGPT has surged interest to cause people to look for its use in different tasks. However, before allowing it to replace humans, its capabilities should be investigated. As ChatGPT has potential for use in testing and assessment, this study aims to investigate the questions generated by ChatGPT by comparing them to those written by a course instructor. To investigate this issue, this study involved 36 junior students who took a practice test including 20 multiple-choice items generated by ChatGPT and 20 others by the course instructor, resulting in a 40-item test. Results indicate that there was an acceptable degree of consistency between the ChatGPT and the course instructor. Post-hoc analyses point to consistency between the instructor and the chatbot in item difficulty, yet the chatbot's results were weaker in item discrimination power and distractor analysis. This indicates that ChatGPT can potentially generate multiple-choice exams similar to those of the course instructor.

The Vocabulary Size Test (VST) measures English learners' decontextualised receptive vocabulary knowledge of written English and has nine bilingual versions with multiple-choice options written in other languages. This study used the English-Japanese version of the VST to investigate the extent to which loanword items were answered correctly by Japanese first language (L1) university students compared to non-loanword items, and whether it was easier to answer these loanword items when the correct answer option was written in loanwords rather than Japanese-words. Paired t-tests showed a significant difference in correct response rates between the loanword and non-loanword items, and the loanword options and Japanese-word options, with a large effect size. The results suggest the relative ease of learning English loanwords compared to non-loanwords for L1 Japanese users, and the need to consider the use of loanwords in vocabulary tests to measure test-takers' vocabulary size more accurately.

The present study has aimed to develop and validate a protein concept inventory (PCI) consisting of 25 multiple-choice (MC) questions to assess students' understanding of protein, which is a fundamental concept across different biology disciplines. The development process of the PCI involved a literature review to identify protein-related content, validation interviews to iteratively validate and refine the created items (n = 26), and data collection from a large sample (n = 291) for statistical analysis. An expert interview was held with two different field experts regarding the content validity of the draft PCI tool, the suitability of the options, and the clarity of the items. Free choice format (multiple marking) was used to answer the developed MC items. In scoring these items, positive points were given to correct options, and negative points were given to incorrect options. Evidence regarding the psychometric properties of the PCI trial form was collected through factor analysis, group differentiation, internal consistency, and item analysis using quantitative data. The evidence collected demonstrates that the validity and reliability of the PCI as a measurement tool have been confirmed. PCI's scoring approach and the use of response patterns created by multiple markings in teaching are discussed.

The 21st century learning demanded students to master the 4C competencies: critical thinking and problem-solving, communication, collaboration, and creativity and innovation, with problem-solving skills significantly influencing students' cognitive learning outcomes. This study utilized the Flipped Classroom instructional model, assisted by the Khan Academy application, as a key factor in educational change. This research aimed to determine the effects related to the implementation of the Flipped Classroom instructional model with the assistance of the Khan Academy application on students' problem-solving abilities and cognitive learning outcomes. The research design employed was quasi-experimental with data collection through observation, interviews, tests, and documentation. The results obtained through Independent Sample t-Test in measuring problem-solving abilities indicated that the Sig. value (2-tailed) was 0.028 for the first indicator (understanding the problem); 0.028 for the second indicator (planning the solution); 0.000 for the third indicator (implementing the solution plan); and 0.001 for the last indicator (checking back). This meant that the results were < 0.05 for each indicator. Another result, namely the Analysis of Covariance (ANCOVA) for cognitive learning outcomes was 0.000. This result showed that the Sig. value was < 0.05. Both results indicated that there was an influence of implementing the Flipped Classroom instructional model with the assistance of the Khan Academy application on student's problem-solving abilities and cognitive learning outcomes of class X students at SMAN 1 Kutorejo regarding the topic of viruses and their roles.

High-order cognitive skills (HOCS) are required to solve problems and make decisions. ARVi or augmented reality virus is a learning medium used to enhance HOCS. Aims of this research is to find out the effect of implementing ARVi media on enhancing high-level cognitive skills of students. The research is a quasi-experiment with a pretest-posttest control group design. The research was conducted in MAN 1 Kota Bogor class 10 using the Merdeka curriculum. The research instrument is a multiple-choice test that has been validated to measure high-level cognitive skills. The result of the t-test shows t-value (3.14) > t-table (1.98), indicating that there is a significant difference between the posttest scores of the control group and the experimental group. This means that the use of ARVi media has an effect on high-level cognitive skills. The improvement in high-level cognitive skills was continuous in C5 (evaluation) at 78%, C6 (creation) at 72%, and C4 (analysis) at 70%. These results are not independent of the support provided by the various content offered by ARVi media, such as augmented reality, literacy, and high-level questionnaires. Current education can benefit from technologies like ARVi media that help increase the high-level cognitive skills of students.

In this study, it was aimed to develop an achievement test taking into account the subject acquisitions of the sound and properties unit in the sixth-grade science course. In the test development phase, firstly, literature review for the study was conducted. Then, 30 multiple choice questions in align with the subject acquisition in the 2018 science curriculum. This 30-question test was presented to opinion of three academicians and a science teacher for the validity of the test. Necessary adjustments were made in line with the opinions of the experts. Subsequently, the draft test, modified according to the experts' feedback, was applied to a total of 300 seventh grade students. After the analysis, the number of questions in the test was reduced to 27. As a result of item analysis, mean difficulty index value of the test was 0.41, and item discrimination index was 0.49. The reliability analysis for the developed test was found 0.92 by calculating the KR-20 reliability coefficient value. By considering, the validity and reliability analysis results, it was concluded that the final version of the developed test, grounded in Bloom's Taxonomy, is a valid and reliable test with different difficulty levels.

The increasing popularity of the Common European Framework of Reference (CEFR) in non-native English-speaking countries has generated a demand for concrete examples in the creation of CEFR-based tests that assess the four main English skills. In response, this research endeavors to provide insight into the development and validation of a CEFR-based test aimed at evaluating undergraduate students' English proficiency for placement tests and exit exams. The CEFR served as the framework for item development while Classical Test Theory informed the test evaluation process. A sample of 2,248 first-year students participated in Testing 1 and 3,655 first- and second-year students took part in Testing 2. The results of the analysis of the multiple-choice listening and reading tests indicated favorable levels of item difficulty and discrimination indices, as well as high reliability coefficients obtained from Cronbach's alpha, Kuder-Richardson, and split-half reliability. The correlation and regression analyses revealed close relationships between the subtests and between each subtest and the total score, supporting the test's criterion validity. The study also demonstrated significant predictive validity on TOEIC scores. The findings of this study offer implications for the development of university-level English proficiency tests that integrate CEFR levels and CTT analysis.

This study aimed at testing the effectiveness of an e-module based on ProblemBased Learning combined with Socratic Dialogue on the topic of environmental change in improving school students' critical thinking skills. This research was a quasi-experimental study and used a non-randomised control group, pre-test/posttest design. The sampling technique was cluster random sampling. The sample in this study was two experimental classes (n = 72) and two control classes (n = 72). The data collection instruments included multiple-choice tests and a questionnaire. The test results were analysed using the t-test and the N-gain score test. The results showed significant difference in students' critical thinking skills between the experimental and control classes. Students who used the e-module showed better critical thinking skills. Students who used the e-module showed a greater increase in critical thinking skills after the learning process, compared to students who did not use the e-module. Thus, it can be concluded that the e-module based on PBL combined with Socratic Dialogue effectively improved students' critical thinking skills.

This design case details the design process of a multiple-choice assessment of socio-scientific systems thinking. This assessment is situated within a larger project that aims to understand the ways students use multiple scientific models to understand complex socio-scientific issues. In addition to the research component, this project entails developing curriculum and assessment resources that support science teaching and learning. We begin this paper by framing the needs that motivated the design of this assessment and introducing the design team. We then present a narrative outlining the design process, focusing on key challenges that arose and the ways these challenges influenced our final design. We conclude this paper with a discussion of the compromises that had to be made in the process of designing this instrument.

Understanding the concepts related to real function is essential in learning mathematics. To determine how students understand these concepts, it is necessary to have an appropriate measurement tool. In this paper, we have created a web application using 32 items from conceptual understanding of real functions (CURF) item bank. We conducted a psychometric analysis using Rasch model on 207 first-year students. The analysis showed that CURF is a dependable and valid instrument for measuring students' CURF. The test is uni-dimensional; all items are consistent with the construct and have excellent item fit statistics. The results indicate that the items are independent of each other and unbiased towards the gender and high school background of the students.

This paper aims to assess the impact of the reciprocal teaching strategy on the academic performance of student-teachers enrolled in the Higher Professional Diploma (HDE) program at Ajman University, United Arab Emirates. The study adopts a quasi-experimental design, employing pre- and post-tests and a control-experimental group configuration. The sample comprises 80 student-teachers selected from Ajman University during the second semester of the academic year 2022-2023. An academic achievement test in a multiple-choice format serves as the research instrument. The study's outcomes reveal noteworthy differences, indicating significant mean variations in the post-test scores between the experimental and control groups, favoring the control group in the context of the Reciprocal Teaching strategy. These results hold implications for international students, prompting considerations about the transferability and generalizability of teaching strategies across diverse cultural and educational settings. The findings suggest the broad effectiveness of the reciprocal teaching method, emphasizing the necessity of aligning teaching strategies with international students' specific needs and preferences. The research underscores the importance of tailoring teaching approaches to meet the unique requirements of a diverse cohort of global learners. It suggests that reciprocal teaching demonstrates efficacy across varied groups of international students, emphasizing the importance for instructors to select instructional methods that cater to the distinct needs of their diverse international student body.

Large introductory classes, with their expansive curriculum, demand assessment strategies that blend efficiency with reliability, prompting the consideration of multiple-choice (MC) tests as a viable option. Crafting a high-quality MC test, however, necessitates a meticulous process involving reflection on assessment format appropriateness, test blueprint design, and adherence to item-writing guidelines aligned with learning objectives and teaching strategies. This inherently time-consuming undertaking ideally requires a collaborative effort from a team of writers who possess expertise in both the subject domain and the specific course context -- an aspiration complicated by the multifaceted demands of higher education instruction. Given these challenges, educators often seek pragmatic solutions, including the adoption or adaptation of existing MC tests. However, the utility of these tests is ambiguous if the original test blueprint and the classification of questions are unknown. This paper introduces a structured four-step "reverse engineering" test blueprint process and proposes a systematic approach to identify test questions that align with the targeted learning objectives. One crucial step incorporates the Taxonomy Table (Anderson & Krathwohl, 2001) facilitating the classification of questions in the cognitive process dimension. As we delve into the intricacies of this analytical journey, we aim to provide a valuable resource for educators seeking to optimize the effectiveness and relevance of MC tests as a high-stakes assessment option.

Scientific thinking constitutes a vital component of scientific competencies, crucial for citizens to adapt to the evolving societal landscape. To cultivate students' scientific thinking, teachers should possess an adequate professional knowledge foundation, which encompasses pedagogical content knowledge (PCK). Assessing teachers' PCK of scientific thinking facilitates the development of effective curricula tailored to their continuous professional development. Despite its significance, empirical studies on biology teachers' PCK of scientific thinking are notably lacking. Hence, this research aimed to create a reliable and valid tool to evaluate upper-secondary school biology teachers' PCK of scientific thinking. The results showed that the instrument exhibits high reliability and good validity, affirming its efficiency for investigative purposes. A collective of 292 in-service biology teachers from upper-secondary schools participated in this investigation through the completion of an online survey. The results indicated that, overall, as well as for each component, upper-secondary school biology teachers' performance on PCK of scientific thinking fell within the lower to middle range. Specifically, the performance levels of four components: knowledge of students (KSU), knowledge of instructional strategies (KIS), knowledge of curriculum (KC), and knowledge of assessment (KA) declined sequentially.

Complex subjects such as physiology can be challenging for students to learn. These challenges are not uncommon in implementing the learning process in physiology and affect learning outcomes. Dramatization is an interactive and effective method to improve learning outcomes. In a project designed by senior medical students, junior medical students were guided in creating dramatizations related to three topics. Senior students were trained and assisted to prepare scenarios and make videos. The dramatizations were then carried out with junior medical students to help them better understand physiology and pathophysiology topics. A group of junior students receiving the same topics in a lecture format served as a control group. Pretest and posttest questionnaires were used to measure the improvement of learning outcomes. Assessment results showed an increase in performance in both groups. This study shows that dramatizations provide an effective alternative to lectures for instructing junior medical students.

Given the high prevalence of multiple-choice examinations with formula scoring in medical training, several studies have tried to identify other factors in addition to the degree of knowledge of students which influence their response patterns. This study aims to measure the effect of students' attitude towards risk and ambiguity on their number of correct, wrong, and blank answers. In October 2018, 233 3rd year medical students from the Faculty of Medicine of the University of Porto, in Porto, Portugal, completed a questionnaire which assessed the student's attitudes towards risk and ambiguity, and aversion to ambiguity in medicine. Simple and multiple regression models and the respective regression coefficients were used to measure the association between the students' attitudes, and their answers in two examinations that they had taken in June 2018. Having an intermediate level of ambiguity aversion in medicine (as opposed to a very high or low level) was associated with a significant increase in the number of correct answers and decrease in the number of blank answers in the first examination. In the second examination, high levels of ambiguity aversion in medicine were associated with a decrease in the number of wrong answers. Attitude towards risk, tolerance for ambiguity, and gender did not show significant association with the number of correct, wrong, and blank answers for either examination. Students' ambiguity aversion in medicine is correlated with their performance in multiple-choice examinations with negative marking. Therefore, it is suggested the planning and implementation of counselling sessions with medical students regarding the possible impact of ambiguity aversion on their performance in multiple-choice questions with negative marking.

Practical classes are an important and essential part of undergraduate programs in Chemical Engineering, as each experiment provides an opportunity to reinforce the theory of discrete unit operations that are taught elsewhere in the course. While an expensive pedagogical method, when practical sessions are delivered well, they can be one of the best learning experiences for students. As with all pedagogical methods, for students to gain maximum benefit of practical classes, a high level of engagement is required. Consequently, lab assignments need to be designed in a way that guides and instructs students on the theory, procedure, and risks associated with any practical and its associated assessments. This paper describes the outcomes of a qualitative investigation that evaluated student perceptions of updated prelab content combined with a new variation in postlab assessments and a renewed focus on practical skills during practical classes. The overall aim was to improve the student experience in practical classes. Paradoxically, periods of remote teaching enforced by the COVID-19 pandemic created further opportunities to make innovative changes to practical class resources. Subsequent student evaluations also indicated perceptions about each newly introduced component (instructional videos, online multiple-choice prelab quiz, variation in postlab assessment, introduction of grading rubrics, and a practical skills assessment), and more than 75% wanted these resources retained.

Knowledge exploration refers to actively seeking information, ideas, and experiences, often beyond immediate task requirements. Previous research on exploratory behaviour has predominantly focused on visual and perceptual forms of exploration, overlooking the academic aspect, where the aim is to bridge knowledge gaps. To test the effect of task and individual factors on knowledge exploration, an experimental study was conducted on 100 high school students in two sessions. In the first session, participants answered 15 multiple-choice questions without receiving immediate feedback but were given the opportunity to explore the answers. In the second session, immediate feedback was provided after each response. The results indicated that knowledge exploration was more likely to occur when students answered the questions wrongly, and the feedback on the accuracy of answers significantly enhanced knowledge exploration for incorrect answers. Session 2 also investigated the impact of factors such as Feeling of Confidence (FOC), confidence error, and intrinsic and extrinsic academic motivation on knowledge exploration. A multilevel ordinal logit model was employed to analyse the within-person and between-person relationships among these variables. The findings revealed that FOC, confidence error, and intrinsic motivation positively predicted knowledge exploration, whereas extrinsic motivation had a negative effect. These findings have significant implications for educational practice and policy. Educators can structure their course materials to cultivate students' metacognitive awareness, thereby promoting deep engagement and enhancing their learning experience.

Assessing students' (mis)conceptions is a challenging task for teachers as well as for researchers. While individual assessment, for example through interviews, can provide deep insights into students' thinking, this is very time-consuming and therefore not feasible for whole classes or even larger settings. For those settings, automatically evaluated multiple-choice (MC) items could be a solution. However, it is a challenge to design those items and to adapt them for other countries in a way that they adequately reveal students' (mis)conceptions. In this article, we investigate the question whether it is valid to use a German adaption of a multiple-choice test developed in Australia for formative assessment of the "letter-as-object" misconception in Germany. For this, first semi-structured interviews with five German Year 8 students were conducted, and second, 616 students were asked for short written explanations. These data were analysed with regards to the students' (mis)conceptions and compared with their automatic online diagnosis. In general, a high concordance between online SMART test results and students' explanations was observed, confirming that useful diagnoses of student misconceptions can be obtained from such a short well-designed MC test.

Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.

This article presents a simple, cognitive theory of science and learning. The first section of the paper develops the theory's two main propositions: (i) A wide range of scientific activities rely heavily on one type of reasoning, hypothetical thinking, and (ii) This type of reasoning is also useful to students for learning science content. The second section of the paper presents a taxonomy of multiple-choice questions that use hypothetical thinking and the third section of the paper tests the theory using data from a college biology course. As expected by the theory, student responses to 24 scientific reasoning questions were consistent with a one-dimensional psychometric construct. Student responses to the scientific reasoning questions explained 36% of the variance in exam grades. Several directions for additional research are identified, including studying the psychometric structure of scientific thinking in more detail, performing randomized, controlled experiments to demonstrate a causal relationship between scientific thinking and learning, and identifying the relative contribution of other factors to success in college.

Many instructors use a syllabus quiz to ensure that students learn and understand the content of the syllabus. In this project, I move beyond this exercise's primary function and examine students' syllabus quiz scores to see if they can predict how well students perform in the course overall. Using data from 495 students enrolled in 18 sections of an Introduction to Sociology course, I find that students who do not earn a maximum score on the syllabus quiz are more likely to receive a lower final course grade and are less likely to pass the course than their top-scoring counterparts. These findings allow instructors to identify struggling students as early as the first week of the semester and design interventions that will match up with students' particular needs and strengths. Furthermore, this project demonstrates the pedagogical usefulness of the syllabus quiz beyond its initial purpose of testing students' knowledge of course policies and procedures.

Recent advances in automated scoring technology have made it practical to replace multiple-choice questions (MCQs) with short-answer questions (SAQs) in large-scale, high-stakes assessments. However, most previous research comparing these formats has used small examinee samples testing under low-stakes conditions. Additionally, previous studies have not reported on the time required to respond to the two item types. This study compares the difficulty, discrimination, and time requirements for the two formats when examinees responded as part of a large-scale, high-stakes assessment. Seventy-one MCQs were converted to SAQs. These matched items were randomly assigned to examinees completing a high-stakes assessment of internal medicine. No examinee saw the same item in both formats. Items administered in the SAQ format were generally more difficult than items in the MCQ format. The discrimination index for SAQs was modestly higher than that for MCQs and response times were substantially higher for SAQs. These results support the interchangeability of MCQs and SAQs. When it is important that the examinee generate the response rather than selecting it, SAQs may be preferred. The results relating to difficulty and discrimination reported in this paper are consistent with those of previous studies. The results on the relative time requirements for the two formats suggest that with a fixed testing time fewer SAQs can be administered, this limitation more than makes up for the higher discrimination that has been reported for SAQs. We additionally examine the extent to which increased difficulty may directly impact the discrimination of SAQs.

Multiple Choice Questions (MCQs) are a popular assessment method because they enable automated evaluation, flexible administration and use with huge groups. Despite these benefits, the manual construction of MCQs is challenging, time-consuming and error-prone. This is because each MCQ is comprised of a question called the "stem", a correct option called the "key" along with alternative options called "distractors" whose construction demands expertise from the MCQ developers. In addition, there are different kinds of MCQs such as Wh-type, Fill-in-the-blank, Odd one out, and many more needed to assess understanding at different cognitive levels. Automatic Question Generation (AQG) for developing heterogeneous MCQ stems has generally followed two approaches: semantics-based and machine-learning-based. Questions generated via AQG techniques can be utilized only if they are grammatically correct. Semantics-based techniques have been able to generate a range of different types of grammatically correct MCQs but require the semantics to be specified. In contrast, most machine-learning approaches have been primarily able to generate only grammatically correct Fill-in-the-blank/Cloze by reusing the original text. This paper describes a technique for combining semantic-based and machine-learning-based techniques to generate grammatically correct MCQ stems of various types for a technical domain. Expert evaluation of the resultant MCQ stems demonstrated that they were promising in terms of their usefulness and grammatical correctness.

The testing effect refers to the gains in learning and retention that result from taking practice tests before the final test. Understanding the conditions under which practice tests improve learning is crucial, so four experiments were conducted with a total of 438 undergraduate students in Turkey. In the first study, students who took graded practice tests outperformed those who took them as ungraded practice. In the second study, students who took short-answer questions before the first exam and multiple-choice questions before the second exam scored higher on the second exam. In the third study, multiple-choice, short-answer and hybrid questions produced similar learning gains. In the fourth study, students who received detailed feedback immediately after class performed similarly to those who received feedback at the beginning of the next class. The results suggested the contribution of graded practice tests in general; however, the type of questions or the timing of feedback did not predict learning outcomes.

Diabetes mellitus and obesity are major public health issues that significantly impact the health care system. The next generation of health care providers will need a deep understanding of the pathophysiology of these diseases if we are to prevent, treat, and eventually cure these diseases and ease the burden on patients and the health care system. Physiology core concepts are a set of core principles, or "big ideas," identified by physiology educators that are thought to promote long-term retention, create a deeper understanding, and help with formation of critical thinking skills. Here we describe our scaffolded teaching approach in an upper year undergraduate pathophysiology course to educate students about these two diseases and discuss how learning about the basis of these highly integrative diseases from the biochemical to whole body level is a meaningful tool in the physiology educator toolbox to reinforce physiology core concepts. This teaching strategy is designed to engage students in the scientific process and hone their problem-solving skills such that they are hopefully equipped to treat and eventually cure these diseases as they move forward in their careers.

Causal reasoning is important to children's cognition and academic development. However, there have been few empirical studies on the impact of visual cues and non-verbal scaffolding on children's reasoning in continuous causal processes. Hence, the present study aims to explore how causal reasoning in continuous processes is facilitated by visual mind maps and multiple-choice questions through science experiments. By randomly selecting 136 children aged 9-13, the following results were obtained: Children provided with a mind map containing visual causal cues performed significantly better than the non-cue group on explanation tasks regardless of age differences, and children assessed using non-verbal multiple-choice questions scored significantly higher in explaining causal relationships than those using only verbal reports. This suggests that identification and explanation need to be differentiated for a more accurate evaluation of causal reasoning ability. These results have valuable implications for science curriculum and pedagogy at primary schools.

Emotional intelligence (EI) has a positive correlation with the academic performance of medical students. However, why there is a positive correlation needs further exploration. We hypothesized that the capability of answering higher-order knowledge questions (HOQs) is higher in students with higher EI. Hence, we assessed the correlation between EI and the capability of medical students to answer HOQs in physiology. First-year undergraduate medical students (n = 124) from an Indian medical college were recruited as a convenient sample. EI was assessed by the Schutte Self-Report Emotional Intelligence Test (SSEIT), a 33-item self-administered validated questionnaire. A specially designed objective examination with 15 lower-order and 15 higher-order multiple-choice questions was conducted. The correlation between the examination score and the EI score was tested by Pearson's correlation coefficient. Data from 92 students (33 females and 59 males) with a mean age of 20.14±1.87 yr were analyzed. Overall, students got a percentage of 53.37±14.07 in the examination, with 24.46±9.1 in HOQs and 28.91±6.58 in lower-order knowledge questions (LOQs). They had a mean score of 109.58±46.2 in SSEIT. The correlation coefficient of SSEIT score with total marks was r ¼ 0.29 (P ¼ 0.0037), with HOQs was r ¼ 0.41 (P < 0.0001), and with LOQs was r = 0.14 (P = 0.19). Hence, there is a positive correlation between EI and the capability of medical students to answer HOQs in physiology. This study may be the foundation for further exploration of the capability of answering HOQs in other subjects.

Digital automated assessment is a valuable and time-efficient tool for educators to provide immediate and objective feedback to learners. Automated assessment, however, puts high demands on the quality of the questions, alignment with the intended learning outcomes, and the quality of the feedback provided to the learners. We here describe the development and use of a large number (N > 180) of question banks with multiple items (N [greater than or equal to] 20) that are aligned to the intended learning outcomes of an undergraduate general chemistry course. Even though the development of a large number of high-quality question banks is a formidable task, they allow for flexible and easy-to-implement solutions for formative and summative assessment once developed. We here discuss three applications of the question banks: formative assessment in large online classes, practice tests that provide formative assessment outside classes, and summative assessment through a digital exam. We highlight the importance of aligning the question banks with intended learning outcomes, providing additional feedback to the learners and of quality assurance of the question banks, and show how the combined use of the question banks supports student self-regulated learning. We hope that the present work can inspire others to discover the various applications of question banks for formative and summative assessment.

This research utilizes a unique, validated, multiple-choice exam design that allows researchers to observe and measure the degree to which students copy answers from their peers. Using data collected from the exam, this study investigates whether asking students to sign an honor code at the start of the exam reduced instances of cheating relative to a control group. Empirical results demonstrate that the classroom of students who signed the honor code had less overall copying of answers and also fewer individual students who engaged in cheating. This study contributes to the literature by conducting analysis on a sample that measures actual cheating frequency, as opposed to relying on reported cheating in surveys, and by demonstrating that a positive intervention can effectively discourage unethical behavior.

The Multiple-choice (MC) item format is commonly used in educational assessments due to its economy and effectiveness across a variety of content domains. However, numerous studies have examined the quality of MC items in high-stakes and higher-education assessments and found many flawed items, especially in terms of distractors. These faulty items lead to misleading insights about the performance of students and the final decisions. The analysis of distractors is typically conducted in educational assessments with multiple-choice items to ensure high-quality items are used as the basis of inference. Item response theory (IRT) and Rasch models have received little attention for analyzing distractors. For that reason, the purpose of the present study was to apply the Rasch model, to a grammar test to analyze items' distractors of the test. To achieve this, the current study investigated the quality of 10 instructor-written MC grammar items used in an undergraduate final exam, using the items responses of 310 English as a foreign language (EFL) students who had taken part in an advanced grammar course. The results showed an acceptable fit to the Rasch model and high reliability. Malfunctioning distractors were identified.

Testing organizations are faced with increasing demand to provide subscores in addition to the total test score. However, psychometricians argue that most subscores do not have added value to be worth reporting. To have added value, subscores need to meet a number of criteria: they should be reliable, distinctive, and distinct from each other and from the total score. In this study, the quality of subscores from two subscore augmentation models (Wainer and Yen) was compared in terms of distinctness and variability. The reliabilities of the Wainer-augmented subscores were also examined. The models were applied to a high-stakes English language proficiency test in Iran. The results of the study showed that Yen better-satisfied subscore distinctness while Wainer best-preserved variability and had high-reliability subscores. In other words, Yen-augmented subscores had lower correlations while Wainer-augmented subscores better discriminated examinees with different ability levels. Thus, none of the examined models of subscoring satisfied all criteria. The results of the study are discussed and suggestions for future research are provided.

This study expands arguments calling for a more rigorous approach to high-frequency vocabulary list-based learning in EFL learning environments. Test and flashcard item designs were validated through quantitative (midterm and final test) and qualitative (survey) results to explore the impact of digital flashcards designed to build recall-level comprehension on both timed gap-fill and traditional multiple-choice posttests. Quizlet was chosen as the platform due to the affordance it provides teachers to create flashcard content and monitor practice. The results showed that multiple-choice, recognition-level test items result in a 20% overestimation of knowledge relative to gap-fill posttests. Additionally, a post-semester survey of 138 Japanese, pre-intermediate students of English showed a highly positive response to the recall-focused practice and testing system. The results demonstrate that for high-frequency L2 vocabulary, a paradigm shift from form-meaning recognition to form-meaning recall is an important direction for high-frequency vocabulary instruction and testing.

This study addressed a gap in existing research on Multiple-Choice (MC) cloze tests by focusing on the learners' perspective, specifically examining the difficulties faced by vocational high school students (VHSs). A nationwide sample of 293 VHSs participated, providing both quantitative and qualitative data through a self-developed questionnaire. The results revealed that vocabulary and grammar posed the greatest challenges in the MC cloze test, while sentence patterns were perceived as the least difficult by VHSs. Factors contributing to these difficulties included the need for increased focus on vocabulary and grammar learning. Some participants attributed challenges to personal perceptions of intellectual capability, while others highlighted the influential role of teachers' attitudes on their learning motivations and outcomes. The study suggested implications for test designs and teaching approaches. Despite these contributions, the study acknowledged limitations and offered suggestions for future research directions.

Previous studies on lexical focus on form (FonF) have mostly centred on FonF in reading with a few pre-selected lexical items. This study investigated the contribution of oral incidental FonF to developing learners' lexical knowledge in a free discussion EFL class. Incidental FonF was provided to 15 upper-intermediate learners who participated in 10 sessions of a meaning-oriented class. To gauge the retention rate of the lexical focus on form episodes (FFEs), two individualized multiple-choice achievement tests based on the lexical FFEs the learners reported to have no previous knowledge of in their uptake sheets were administered every five sessions, and two delayed posttests were given five weeks following each immediate posttest. Moreover, the participants were asked to compose two prompt-based writings on topics selected out of the covered themes in the posttests. Thematically relevant lexical FFEs extracted from each learner's uptake sheets made up the tailored prompts per learner. Results indicated rather high effectiveness of oral incidental FonF in developing learners' receptive and productive lexical knowledge in both short and long terms. The findings also revealed that learners fail to develop knowledge of the grammatical aspects of some lexical FFEs if their attention is merely drawn to their meaning aspects.

The purpose of this study is to introduce a method for converting scored 4-option multiple-choice (MC) items into scored 3-option MC items without re-pretesting the 3-option MC items. This study describes a six-step process for achieving this goal. Data from a professional credentialing exam was used in this study and the method was applied to 24 forms of the exam. The results found 100% accuracy in predicting the rounded passing score for all forms.

The impacts of a technology-integrated formative assessment technique on students' conceptual and procedural knowledge in studying chemical equilibrium are studied in this study. To attain the purpose, a quasi-experimental pretest-posttest strategy nested with a qualitative study method was adopted. The study has three groups: two experimental and one comparative. A random sample strategy was utilized to select two intact classes for treatment and one intact class for comparison groups. Data was collected using the chemical equilibrium conceptual test, the procedural test, and classroom observation. The data was examined using descriptive (mean and standard deviation) and inferential statistics (one-way ANOVA, one-way MANOVA, and Pearson product moment correlation). According to the findings, technology-integrated formative assessment processes outperformed traditional techniques and formative assessment strategies alone in enhancing students' conceptual and practical understanding of chemical equilibrium. Similarly, when technology-integrated formative assessment processes are used, classroom observations show that students have a strong motivation to learn and that the instructor is more skilled than the other two teachers. Technology-integrated formative assessment processes were shown to be more effective than the other two groups in promoting students' conceptual and procedural understanding when learning chemical equilibrium.

Currently, we are facing the rapid development of information and communication technology (ICT) that characterizes the 21st century. Challenges, problems, life, and careers in the 21st century can be successfully faced if we master the 21st century skills. However, nowadays it is still very rare to develop a 21st century skill measurement model, so it is necessary to develop test instruments to measure 21st century skills. The development of test instruments to measure 21st century skills in Indonesia is still very open by paying attention to the clarity of the instrument in describing real phenomena and complete data as material for problem solving. Based on the description above, the researchers developed a multiple-choice test based on experimental activities to measure students' 21st century skills which are creativity, critical thinking, collaboration, communication (4Cs) in heat and heat transfer using research and development (R&D) methods with the analyze, design, develop, implement, and evaluate (ADDIE) stage. Development begins with conducting literature and field studies, designing and making videos of experiment activities, compiling open-ended questions, distributing them to 12th graders of science, analyzing answers based on references, developing answer options, developing multiple choice tests, conducting validation by experts, and testing limited to students. The product of this research is expected to be used by educators and students for independent learning activities or as an example to develop multiple choice tests based on experimental activities to measure 21st century skills in other physics materials and other subjects.

This paper explores the impact of AI applications in adaptive learning on learners with varying levels of Chinese proficiency and diverse learning needs. The study investigates the robustness of VR technology in enhancing Chinese oral proficiency and pragmatic skills among 1st year and 2nd year learners. It compares the established VR app Mondly with Uptale, an Immersive VR authoring platform, focusing on AI-based voice recognition and analysis features. Additionally, it compares the adaptability of the Uptale VR platform with Mondly by analyzing factors such as difficulty adjustments, interactive features, learner engagement, and support for learner autonomy. It was found that VR technology, in general, enhances the Chinese learning experience. Uptale's additional features extend its potential utility in student assessment. These features include custom learning modules that incorporate students' self-shot 360-degree videos and interactive elements, such as multiple-choice questions, audio prompts, speaking/recording tags, feedback mechanisms, and analysis reports.

This study describes the characteristics of the test and its items used in the national-standardized school examination by applying classical test theory and focusing on the item difficulty, item discrimination, test reliability, and distractor analysis. We analyzed response data of 191 12th graders from one of public senior high schools in Yogyakarta City, Indonesia,to the examination on the elective mathematics subject. The results showed that both multiple-choice and essay items contained in the test were at a moderate level of difficulty. The lowest item difficulty index went to the multiple-choice item where students failed in interpreting straight and dashed lines and went to the essay item that required complex intra-mathematical connections. In addition, we only found one item which was poor in distinguishing student competence. Furthermore, students' test scores on multiple-choice and essay items were reliable. Although most multiple-choice items had at least two functioning distractors, it was still found two items whose all distractors were not functioning. In this article, we provide some suggestions concerning improvement towards mathematics learning practices.