Search

Many 4-year public institutions face significant pedagogical challenges due to the high ratio of students to teaching team members. To address the issue, we developed a workflow using the programming language R as a method to rapidly grade multiple-choice questions, adjust for errors, and grade answer-dependent style multiple-choice questions, thus shifting the teaching teams time commitment back to student interaction. We provide an example of answer-dependent style multiple-choice questions and demonstrate how the output allows for discrete analysis of questions based on various categories such as Fundamental Statements or Blooms Taxonomy Levels. Additionally, we show how student demographics can be easily integrated to yield a holistic perspective on student performance in a course. The workflow offers dynamic grading opportunities for multiple-choice questions and versatility through its adaptability to assessment analyses. This approach to multiple-choice questions allows instructors to pinpoint factors affecting student performance and respond to changes to foster a healthy learning environment.

ChatGPT's role in creating multiple-choice questions (MCQs) is growing but the validity of these artificial-intelligence-generated questions is unclear. This literature review was conducted to address the urgent need for understanding the application of ChatGPT in generating MCQs for medical education. Following the database search and screening of 1920 studies, we found 23 relevant studies. We extracted the prompts for MCQ generation and assessed the validity evidence of MCQs. The findings showed that prompts varied, including referencing specific exam styles and adopting specific personas, which align with recommended prompt engineering tactics. The validity evidence covered various domains, showing mixed accuracy rates, with some studies indicating comparable quality to human-written questions, and others highlighting differences in difficulty and discrimination levels, alongside a significant reduction in question creation time. Despite its efficiency, we highlight the necessity of careful review and suggest a need for further research to optimize the use of ChatGPT in question generation. Main messages Ensure high-quality outputs by utilizing well-designed prompts; medical educators should prioritize the use of detailed, clear ChatGPT prompts when generating MCQs. Avoid using ChatGPT-generated MCQs directly in examinations without thorough review to prevent inaccuracies and ensure relevance. Leverage ChatGPT's potential to streamline the test development process, enhancing efficiency without compromising quality. [ABSTRACT FROM AUTHOR]

This study aimed to identify the strategies used by 8th-grade students when solving multiple-choice questions in Physics, Chemistry, and Biology, and to explore any differences or similarities in these strategies. Five 8th-grade students from a private secondary school in Kars participated in the study. They were asked to solve multiple-choice questions in each subject using the Think Aloud strategy, followed by semi-structured interviews to understand their problem-solving approaches. The data from the recordings and interviews were transcribed and analyzed using qualitative research software. The findings indicated that students employed distinct cognitive strategies for each subject. In Physics, strategies focused on numerical operations, including using formulas, setting up equations, and simplifying expressions. Chemistry problem-solving involved strategies such as self-questioning, trial and error, and the use of inequalities. For Biology questions, students relied on strategies that facilitated comprehension, such as careful reading, underlining, and identifying key parts of the question. In addition to cognitive strategies, students also utilized meta-cognitive strategies, such as reviewing and marking graphs or figures in Physics and Chemistry, and increasing reading speed and paraphrasing in Biology. The characteristics of the questions, such as the inclusion of visuals or explanations, appeared to influence the types and number of strategies employed. [ABSTRACT FROM AUTHOR]

To determine the potential of large language models (LLMs) to be used as tools by radiology educators to create radiology board-style multiple choice questions (MCQs), answers, and rationales. Two LLMs (Llama 2 and GPT-4) were used to develop 104 MCQs based on the American Board of Radiology exam blueprint. Two board-certified radiologists assessed each MCQ using a 10-point Likert scale across five criteria—clarity, relevance, suitability for a board exam based on level of difficulty, quality of distractors, and adequacy of rationale. For comparison, MCQs from prior American College of Radiology (ACR) Diagnostic Radiology In-Training (DXIT) exams were also assessed using these criteria, with radiologists blinded to the question source. Mean scores (±standard deviation) for clarity, relevance, suitability, quality of distractors, and adequacy of rationale were 8.7 (±1.4), 9.2 (±1.3), 9.0 (±1.2), 8.4 (±1.9), and 7.2 (±2.2), respectively, for Llama 2; 9.9 (±0.4), 9.9 (±0.5), 9.9 (±0.4), 9.8 (±0.5), and 9.9 (±0.3), respectively, for GPT-4; and 9.9 (±0.3), 9.9 (±0.2), 9.9 (±0.2), 9.9 (±0.4), and 9.8 (±0.6), respectively, for ACR DXIT items (p < 0.001 for Llama 2 vs. ACR DXIT across all criteria; no statistically significant difference for GPT-4 vs. ACR DXIT). The accuracy of model-generated answers was 69% for Llama 2 and 100% for GPT-4. A state-of-the art LLM such as GPT-4 may be used to develop radiology board-style MCQs and rationales to enhance exam preparation materials and expand exam banks, and may allow radiology educators to further use MCQs as teaching and learning tools. [ABSTRACT FROM AUTHOR]

Introduction: In the field of medical education, multiple-choice questions (MCQs) represent the most commonly utilized method of assessment. It is necessary to analyze the assessment results through item analysis to ensure the quality is appropriate. This study evaluated the quality of the MCQs utilized for summative evaluation of the students in the General Surgery Course conducted in the year 2023-24, at the College of Medicine (Unaizah), Qassim University, Saudi Arabia. Methods: Using a number of established parameters for item analysis, the study evaluated the multiple-choice questions for difficulty, discrimination power, and quality of distractors. Results: The quality of the questions varied. The means of the facility index, discrimination index, discriminative efficiency, and distractor efficiency were, in order, 76.31%, 0.28, -0.7743, and 32%. Conclusion: Item analysis is a crucial technique for evaluating the quality of MCQs. There were multiple defects in the MCQs used in summative assessments, revealing the scope for further improvement in future courses. It is important to plan faculty development events often to impart knowledge and skills related to creating MCQs that are valid, reliable, and of high quality.

Introduction: The effectiveness of multiple‐choice questions (MCQs) in dental education is pivotal to student performance and knowledge advancement. However, their optimal implementation requires exploration to enhance the benefits. Materials and Methods: An educational tool incorporating MCQs was administered from the 5th to the 10th semester in a dental curriculum. The students filled out a questionnaire after the MCQ, which was linked to the learning management system. Four cohorts of four semesters generated 2300 data points analysed by Spearmen correlation and mixed model regression analysis. Results: Demonstrated a significant correlation between early exam preparation and improved student performance. Independent study hours and lecture attendance emerged as significant predictors, accounting for approximately 10.27% of the variance in student performance on MCQs. While the number of MCQs taken showed an inverse relationship with study hours, the perceived clarity of these questions positively correlated with academic achievement. Conclusion: MCQs have proven effective in enhancing student learning and knowledge within the discipline. Our analysis underscores the important role of independent study and consistent lecture attendance in positively influencing MCQ scores. The study provides valuable insights into using MCQs as a practical tool for dental student learning. Moreover, the clarity of assessment tools, such as MCQs, remains pivotal in influencing student outcomes. This study underscores the multifaceted nature of learning experiences in dental education and the importance of bridging the gap between student expectations and actual performance. [ABSTRACT FROM AUTHOR]

Background Multiple-choice questions (MCQs) are one of the most used assessment formats. However, creating MCQs is a challenging task, particularly when formulating the distractor. Numerous studies have proposed automatic distractor generation. However, there has been no literature review to summarize and present the current state of research in this field. This study aims to perform a systematic literature review to identify trends and the state of the art of automatic distractor generation studies. Methodology We conducted a systematic literature following the Kitchenham framework. The relevant literature was retrieved from the ACM Digital Library, IEEE Xplore, Science Direct, and Scopus databases. Results A total of 60 relevant studies from 2009 to 2024 were identified and extracted to answer three research questions regarding the data sources, methods, types of questions, evaluation, languages, and domains used in the automatic distractor generation research. The results of the study indicated that automatic distractor generation has been growing with improvement and expansion in many aspects. Furthermore, trends and the state of the art in this topic were observed. Conclusions Nevertheless, we identified potential research gaps, including the need to explore further data sources, methods, languages, and domains. This study can serve as a reference for future studies proposing research within the field of automatic distractor generation.

Resumen: Introducción: diseñar preguntas de opción múltiple (POM) con Microsoft Bing (Copilot) para evaluar su calidad e índices psicométricos en educación médica. Material y métodos: se diseñó un examen de 180 preguntas con Microsoft Bing. Este fue evaluado por educadores médicos en términos de relevancia y calidad de distractores. Luego, tras administrarse a estudiantes, se calcularon los índices de dificultad y discriminación. Resultados: la mayoría de preguntas fueron de alta relevancia y los distractores de alta calidad. Los índices de discriminación y dificultad de las preguntas fueron aceptables en la mayoría de preguntas. Conclusión: Microsoft Bing (Copilot) podría usarse como sustituto de ChatGPT para el diseño de POM dadas las evidencias de validez recolectadas en el estudio. Abstract: Introduction: To design multiple-choice questions (MCQs) using Microsoft Bing (Copilot) and evaluate their quality and psychometric indices in medical education. Materials and methods: A 180-question exam was designed using Microsoft Bing. It was evaluated by medical educators in terms of relevance and distractor quality. After administering the exam to students, difficulty and discrimination indices were calculated. Results: Most questions were highly relevant, and the distractors were of high quality. The discrimination and difficulty indices were acceptable for the majority of the questions. Conclusion: Microsoft Bing (Copilot) could be used as a substitute for ChatGPT in designing MCQs, given the evidence of validity collected in the study.

Abstract Background Item difficulty plays a crucial role in assessing students’ understanding of the concept being tested. The difficulty of each item needs to be carefully adjusted to ensure the achievement of the evaluation’s objectives. Therefore, this study aimed to investigate whether repeated item development training for medical school faculty improves the accuracy of predicting item difficulty in multiple-choice questions. Methods A faculty development program was implemented to enhance the prediction of each item’s difficulty index, ensure the absence of item defects, and maintain the general principles of item development. The interrater reliability between the predicted, actual, and corrected item difficulty was assessed before and after the training, using either the kappa index or the correlation coefficient, depending on the characteristics of the data. A total of 62 faculty members participated in the training. Their predictions of item difficulty were compared with the analysis results of 260 items taken by 119 fourth-year medical students in 2016 and 316 items taken by 125 fourth-year medical students in 2018. Results Before the training, significant agreement between the predicted and actual item difficulty indices was observed for only one medical subject, Cardiology (K = 0.106, P = 0.021). However, after the training, significant agreement was noted for four subjects: Internal Medicine (K = 0.092, P = 0.015), Cardiology (K = 0.318, P = 0.021), Neurology (K = 0.400, P = 0.043), and Preventive Medicine (r = 0.577, P = 0.039). Furthermore, a significant agreement was observed between the predicted and actual difficulty indices across all subjects when analyzing the average difficulty of all items (r = 0.144, P = 0.043). Regarding the actual difficulty index by subject, neurology exceeded the desired difficulty range of 0.45–0.75 in 2016. By 2018, however, all subjects fell within this range. Conclusion Repeated item development training, which includes predicting each item’s difficulty index, can enhance faculty members’ ability to predict and adjust item difficulty accurately. To ensure that the difficulty of the examination aligns with its intended purpose, item development training can be beneficial. Further studies on faculty development are necessary to explore these benefits more comprehensively.

Abstract Background Online question banks are the most widely used education resource amongst medical students. Despite this there is an absence of literature outlining how and why they are used by students. Drawing on Deci and Ryan’s self-determination theory, our study aimed to explore why and how early-stage medical students use question banks in their learning and revision strategies. Methods The study was conducted at Newcastle University Medical School (United Kingdom and Malaysia). Purposive, convenience and snowball sampling of year two students were employed. Ten interviews were conducted. Thematic analysis was undertaken iteratively, enabling exploration of nascent themes. Data collection ceased when no new perspectives were identified. Results Students’ motivation to use question banks was predominantly driven by extrinsic motivators, with high-stakes exams and fear of failure being central. Their convenience and perceived efficiency promoted autonomy and thus motivation. Rapid feedback cycles and design features consistent with gamification were deterrents to intrinsic motivation. Potentially detrimental patterns of question bank use were evident: cueing, avoidance and memorising. Scepticism regarding veracity of question bank content was absent. Conclusions We call on educators to provide students with guidance about potential pitfalls associated with question banks and to reflect on potential inequity of access to these resources.

Abstract Objectives This study aimed to investigate the utility of the RAND/UCLA appropriateness method (RAM) in validating expert consensus-based multiple-choice questions (MCQs) on electrocardiogram (ECG). Methods According to the RAM user’s manual, nine panelists comprising various experts who routinely handle ECGs were asked to reach a consensus in three phases: a preparatory phase (round 0), an online test phase (round 1), and a face-to-face expert panel meeting (round 2). In round 0, the objectives and future timeline of the study were elucidated to the nine expert panelists with a summary of relevant literature. In round 1, 100 ECG questions prepared by two skilled cardiologists were answered, and the success rate was calculated by dividing the number of correct answers by 9. Furthermore, the questions were stratified into “Appropriate,” “Discussion,” or “Inappropriate” according to the median score and interquartile range (IQR) of appropriateness rating by nine panelists. In round 2, the validity of the 100 ECG questions was discussed in an expert panel meeting according to the results of round 1 and finally reassessed as “Appropriate,” “Candidate,” “Revision,” and “Defer.” Results In round 1 results, the average success rate of the nine experts was 0.89. Using the median score and IQR, 54 questions were classified as “ Discussion.” In the expert panel meeting in round 2, 23% of the original 100 questions was ultimately deemed inappropriate, although they had been prepared by two skilled cardiologists. Most of the 46 questions categorized as “Appropriate” using the median score and IQR in round 1 were considered “Appropriate” even after round 2 (44/46, 95.7%). Conclusions The use of the median score and IQR allowed for a more objective determination of question validity. The RAM may help select appropriate questions, contributing to the preparation of higher-quality tests.

Background: Online question banks are the most widely used education resource amongst medical students. Despite this there is an absence of literature outlining how and why they are used by students. Drawing on Deci and Ryan's self-determination theory, our study aimed to explore why and how early-stage medical students use question banks in their learning and revision strategies. Methods: The study was conducted at Newcastle University Medical School (United Kingdom and Malaysia). Purposive, convenience and snowball sampling of year two students were employed. Ten interviews were conducted. Thematic analysis was undertaken iteratively, enabling exploration of nascent themes. Data collection ceased when no new perspectives were identified. Results: Students' motivation to use question banks was predominantly driven by extrinsic motivators, with high-stakes exams and fear of failure being central. Their convenience and perceived efficiency promoted autonomy and thus motivation. Rapid feedback cycles and design features consistent with gamification were deterrents to intrinsic motivation. Potentially detrimental patterns of question bank use were evident: cueing, avoidance and memorising. Scepticism regarding veracity of question bank content was absent. Conclusions: We call on educators to provide students with guidance about potential pitfalls associated with question banks and to reflect on potential inequity of access to these resources. [ABSTRACT FROM AUTHOR]

Multiple‐choice questions (MCQs) are the most popular type of items used in knowledge‐based assessments in undergraduate and postgraduate healthcare education. MCQs allow assessment of candidates' knowledge on a broad range of knowledge‐based learning outcomes in a single assessment. Single‐best‐answer (SBA) MCQs are the most versatile and commonly used format. Although writing MCQs may seem straight‐forward, producing decent‐quality MCQs is challenging and warrants a range of quality checks before an item is deemed suitable for inclusion in an assessment. Like all assessments, MCQ‐based examinations must be aligned with the learning outcomes and learning opportunities provided to the students. This paper provides evidence‐based guidance on the effective use of MCQs in student assessments, not only to make decisions regarding student progression but also to build an academic environment that promotes assessment as a driver for learning. Practical tips are provided to the readers to produce authentic MCQ items, along with appropriate pre‐ and post‐assessment reviews, the use of standard setting and psychometric evaluation of assessments based on MCQs. Institutions need to develop an academic culture that fosters transparency, openness, equality and inclusivity. In line with contemporary educational principles, teamwork amongst teaching faculty, administrators and students is essential to establish effective learning and assessment practices. [ABSTRACT FROM AUTHOR]

Purpose: Artificial intelligence, specifically large language models such as ChatGPT, offers valuable potential benefits in question (item) writing. This study aimed to determine the feasibility of generating case-based multiple-choice questions using ChatGPT in terms of item difficulty and discrimination levels. Methods: This study involved 99 fourth-year medical students who participated in a rational pharmacotherapy clerkship carried out based-on the WHO 6-Step Model. In response to a prompt that we provided, ChatGPT generated ten case-based multiple-choice questions on hypertension. Following an expert panel, two of these multiple-choice questions were incorporated into a medical school exam without making any changes in the questions. Based on the administration of the test, we evaluated their psychometric properties, including item difficulty, item discrimination (point-biserial correlation), and functionality of the options. Results: Both questions exhibited acceptable levels of point-biserial correlation, which is higher than the threshold of 0.30 (0.41 and 0.39). However, one question had three non-functional options (options chosen by fewer than 5% of the exam participants) while the other question had none. Conclusions: The findings showed that the questions can effectively differentiate between students who perform at high and low levels, which also point out the potential of ChatGPT as an artificial intelligence tool in test development. Future studies may use the prompt to generate items in order for enhancing the external validity of the results by gathering data from diverse institutions and settings. [ABSTRACT FROM AUTHOR]

Purpose: Item analysis of multiple‐choice questions (MCQs) is an essential tool for identifying items that can be stored, revised, or discarded to build a quality MCQ bank. This study analyzed MCQs based on item analysis to develop a pool of valid and reliable items and investigate stakeholders' perceptions regarding MCQs in a written summative assessment (WSA) based on this item analysis. Methods: In this descriptive study, 55 questions each from 2016 to 2019 of WSA in preclinical removable prosthodontics for fourth‐year undergraduate dentistry students were analyzed for item analysis. Items were categorized according to their difficulty index (DIF I) and discrimination index (DI). Students (2021–2022) were assessed using this question bank. Students' perceptions of and feedback from faculty members concerning this assessment were collected using a questionnaire with a five‐point Likert scale. Results: Of 220 items when both indices (DIF I and DI) were combined, 144 (65.5%) were retained in the question bank, 66 (30%) required revision before incorporation into the question bank, and only 10 (4.5%) were discarded. The mean DIF I and DI values were 69% (standard deviation [Std.Dev] = 19) and 0.22 (Std.Dev = 0.16), respectively, for 220 MCQs. The mean scores from the questionnaire for students and feedback from faculty members ranged from 3.50 to 4.04 and from 4 to 5, respectively, indicating that stakeholders tended to agree and strongly agree, respectively, with the proposed statements. Conclusion: This study assisted the prosthodontics department in creating a set of prevalidated questions with known difficulty and discrimination capacity. [ABSTRACT FROM AUTHOR]

This article explores the process of item analysis for multiple-choice questions (MCQs) in pharmacology exams. The study aimed to evaluate the validity and reliability of the MCQs and create a reliable question bank for future use. The analysis focused on factors such as difficulty index, discrimination index, and distractor effectiveness. The majority of the MCQs met the acceptable criteria for these factors, indicating the importance of regular item analysis in developing valid and reliable MCQs. However, some questions required revision due to being too difficult or too easy. The study emphasizes the need for ongoing item analysis to enhance the quality of MCQ assessments in medical education. [Extracted from the article]

Copyright of Sanamed is the property of Sanamed and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Objectives: This study aimed to investigate the utility of the RAND/UCLA appropriateness method (RAM) in validating expert consensus-based multiple-choice questions (MCQs) on electrocardiogram (ECG). Methods: According to the RAM user's manual, nine panelists comprising various experts who routinely handle ECGs were asked to reach a consensus in three phases: a preparatory phase (round 0), an online test phase (round 1), and a face-to-face expert panel meeting (round 2). In round 0, the objectives and future timeline of the study were elucidated to the nine expert panelists with a summary of relevant literature. In round 1, 100 ECG questions prepared by two skilled cardiologists were answered, and the success rate was calculated by dividing the number of correct answers by 9. Furthermore, the questions were stratified into "Appropriate," "Discussion," or "Inappropriate" according to the median score and interquartile range (IQR) of appropriateness rating by nine panelists. In round 2, the validity of the 100 ECG questions was discussed in an expert panel meeting according to the results of round 1 and finally reassessed as "Appropriate," "Candidate," "Revision," and "Defer." Results: In round 1 results, the average success rate of the nine experts was 0.89. Using the median score and IQR, 54 questions were classified as " Discussion." In the expert panel meeting in round 2, 23% of the original 100 questions was ultimately deemed inappropriate, although they had been prepared by two skilled cardiologists. Most of the 46 questions categorized as "Appropriate" using the median score and IQR in round 1 were considered "Appropriate" even after round 2 (44/46, 95.7%). Conclusions: The use of the median score and IQR allowed for a more objective determination of question validity. The RAM may help select appropriate questions, contributing to the preparation of higher-quality tests. [ABSTRACT FROM AUTHOR]

ABSTRACT Many 4-year public institutions face significant pedagogical challenges due to the high ratio of students to teaching team members. To address the issue, we developed a workflow using the programming language R as a method to rapidly grade multiple-choice questions, adjust for errors, and grade answer-dependent style multiple-choice questions, thus shifting the teaching teams’ time commitment back to student interaction. We provide an example of answer-dependent style multiple-choice questions and demonstrate how the output allows for discrete analysis of questions based on various categories such as Fundamental Statements or Bloom’s Taxonomy Levels. Additionally, we show how student demographics can be easily integrated to yield a holistic perspective on student performance in a course. The workflow offers dynamic grading opportunities for multiple-choice questions and versatility through its adaptability to assessment analyses. This approach to multiple-choice questions allows instructors to pinpoint factors affecting student performance and respond to changes to foster a healthy learning environment.

In the dynamic landscape of contemporary education, the evolution of teaching strategies such as blended learning and flipped classrooms has highlighted the need for efficient and effective generation of multiple-choice questions (MCQs). To address this, we introduce MCQGen, a novel generative artificial intelligence framework designed for the automated creation of MCQs. MCQGen uniquely integrates a large language model (LLM) with retrieval-augmented generation and advanced prompt engineering techniques, drawing from an extensive external knowledge base. This integration significantly enhances the ability of the LLM to produce educationally relevant questions that align with both the goals of educators and the diverse learning needs of students. The framework employs innovative prompt engineering, combining chain-of-thought and self-refine prompting techniques, to enhance the performance of the LLM. This process leads to the generation of questions that are not only contextually relevant and challenging but also reflective of common student misconceptions, contributing effectively to personalized learning experiences and enhancing student engagement and understanding. Our extensive evaluations showcase the effectiveness of MCQGen in producing high-quality MCQs for various educational needs and learning styles. The framework demonstrates its potential to significantly reduce the time and expertise required for MCQ creation, marking its practical utility in modern education. In essence, MCQGen offers an innovative and robust solution for the automated generation of MCQs, enhancing personalized learning in the digital era.

A fundamental challenge in medical education is creating high-quality, clinically relevant multiple-choice questions (MCQs). ChatGPT-based automatic item generation (AIG) methods need well-designed prompts. However, the use of these prompts is hindered by the time-consuming process of copying and pasting, a lack of know-how among medical teachers, and the generalist nature of standard ChatGPT, which often lacks the medical context. The Case-based MCQ Generator, a custom GPT, addresses these challenges. It has been trained by using GPT Builder, which is a platform designed by OpenAI for customizing ChatGPT to meet specific needs, in order to allow users to generate case-based MCQs. By using this free tool for those who have ChatGPT Plus subscription, health professions educators can easily select a prompt, input a learning objective or item-specific test point, and generate clinically relevant questions. It enhances the efficiency of MCQ generation and ensures the generation of contextually relevant questions, surpassing the capabilities of standard ChatGPT. It streamlines the MCQ creation process by integrating prompts published in medical education literature, eliminating the need for manual prompt input. Future development aims at sustainability and addressing ethical and accessibility issues. It requires regular updates, integration of new prompts from emerging health professions education literature, and a supportive digital ecosystem around the tool. Accessibility, especially for educators in low-resource countries, is vital, demanding alternative access models to overcome financial barriers. [ABSTRACT FROM AUTHOR]

BACKGROUND AND AIM: Access to quality health care is essential, particularly in remote areas where the availability of healthcare professionals may be limited. The advancement of artificial intelligence (AI) and natural language processing (NLP) has led to the development of large language models (LLMs) that exhibit capabilities in understanding and generating human-like text. This study aimed to evaluate the performance of a LLM, ChatGPT, in addressing primary healthcare issues. MATERIALS AND METHODS: This study was conducted in May 2023 with ChatGPT May 12 version. A total of 30 multiple-choice questions (MCQs) related to primary health care were selected to test the proficiency of ChatGPT. These MCQs covered various topics commonly encountered in primary healthcare practice. ChatGPT answered the questions in two segments—one is choosing the single best answer of MCQ and another is supporting text for the answer. The answers to MCQs were compared with the predefined answer keys. The justifications of the answers were checked by two primary healthcare professionals on a 5-point Likert-type scale. The data were presented as number and percentage. RESULTS: Among the 30 questions, ChatGPT provided correct responses for 28 yielding an accuracy of 93.33%. The mean score for explanation in supporting the answer was 4.58 ± 0.85. There was an inter-item correlation of 0.896, and the average measure intraclass correlation coefficient (ICC) was 0.94 (95% confidence interval 0.88–0.97) indicating a high level of interobserver agreement. CONCLUSION: LLMs, such as ChatGPT, show promising potential in addressing primary healthcare issues. The high accuracy rate achieved by ChatGPT in answering primary healthcare-related MCQs underscores the value of these models as resources for patients and healthcare providers in remote healthcare settings. This can also help in self-directed learning by medical students.

Background: Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive. Methods: We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests. Results: The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp² = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, ηp² = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp² = 0.002. Conclusions: Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items. [ABSTRACT FROM AUTHOR]

Abstract Background ChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-choice questions (MCQs) and providing credible rationale for its responses. Methods The study used 78 test items from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) for years 2019 to 2021. 78 test items translated from Korean to English with four lead-in prompts per item resulted in a total of 312 MCQs. The MCQs were submitted to ChatGPT and the responses were analyzed for correctness, consistency, and relevance. Results ChatGPT responded with an overall accuracy of 76.0%. Compared to its performance on recall and interpretation questions, the model performed poorly on problem-solving questions. ChatGPT offered correct rationales for 77.8% (182/234) of the responses, with errors primarily arising from faulty information and flawed reasoning. In terms of references, ChatGPT provided incorrect citations for 69.7% (191/274) of the responses. While the veracity of reference paragraphs could not be ascertained, 77.0% (47/61) were deemed pertinent and accurate with respect to the answer key. Conclusion The current version of ChatGPT has limitations in accurately answering MCQs and generating correct and relevant rationales, particularly when it comes to referencing. To avoid possible threats such as spreading inaccuracies and decreasing critical thinking skills, ChatGPT should be used with supervision.

ABSTRACTBackground Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.Methods We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as ‘new’; ‘reused, not disclosed’; or ‘reused, disclosed’. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.Results The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P

Background: ChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-choice questions (MCQs) and providing credible rationale for its responses. Methods: The study used 78 test items from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) for years 2019 to 2021. 78 test items translated from Korean to English with four lead-in prompts per item resulted in a total of 312 MCQs. The MCQs were submitted to ChatGPT and the responses were analyzed for correctness, consistency, and relevance. Results: ChatGPT responded with an overall accuracy of 76.0%. Compared to its performance on recall and interpretation questions, the model performed poorly on problem-solving questions. ChatGPT offered correct rationales for 77.8% (182/234) of the responses, with errors primarily arising from faulty information and flawed reasoning. In terms of references, ChatGPT provided incorrect citations for 69.7% (191/274) of the responses. While the veracity of reference paragraphs could not be ascertained, 77.0% (47/61) were deemed pertinent and accurate with respect to the answer key. Conclusion: The current version of ChatGPT has limitations in accurately answering MCQs and generating correct and relevant rationales, particularly when it comes to referencing. To avoid possible threats such as spreading inaccuracies and decreasing critical thinking skills, ChatGPT should be used with supervision. [ABSTRACT FROM AUTHOR]

The current study explores the impact of varying multiple-choice question preview and presentation formats in a test of second language listening proficiency targeting different levels of text comprehension. In a between-participant design, participants completed a 30-item test of listening comprehension featuring implicit and explicit information comprehension questions under one of four multiple-choice question preview and presentation conditions. Interactions between preview, presentation and comprehension in the participants' test scores were analysed using many facet Rasch analysis. The results suggest that the measurement of participants' listening ability was directly influenced by the presentation of multiple-choice questions. Test scores were highest when participants were able to preview the multiple-choice question stems before the sound file and listened to the options after the text had completed. However, interactions between preview and presentation conditions and comprehension level were only statistically significant in an analysis of the low scoring students' item responses, which were more frequently correct when preview of item stems was available for questions targeting comprehension of implicit information. The research underscores the importance of accounting for test design when making inferences about language learners' listening ability and will be of interest to teachers, practitioners and researchers developing listening assessment tasks. [ABSTRACT FROM AUTHOR]

The significant challenge faced by medical schools is the effortful process of writing a high quantity of high-quality case-based multiple-choice questions (MCQs) to assess the higher-order skills of medical students. The demand for a high volume of MCQs in education has led to the development of Automatic Item Generation (AIG), specifically template-based AIG, which involves creating cognitive and item models by subject matter experts to generate hundreds of MCQs at once using software. It demonstrated significant success in various languages and even being incorporated into national medical licensure exams. However, this method still heavily depends on the efforts of subject matter experts. This paper introduces a detailed ChatGPT prompt for quickly generating case-based MCQs and provides important research questions for future exploration into ChatGPT's potential in generating items, signaling the beginning of the artificial intelligence era in medical education, encouraging health professions education researchers to delve deeper into its potential.

Multiple-choice questions (MCQs) have been considerably used for assessing the individual’s performance in various contexts. The optimal number of options in MCQs is a debatable issue, followed by contradictions and discussions, and is needed for a firm conclusion from empirical and hypothetical findings. This study aims to link theoretical concepts, including challenge-based gamification, zone of proximal development, and prospect theory, and generate insight into educational assessment using motion-in-mind measures. Classical test theory was used to determine reliability and validity. Variations of MCQs experimented: the number of options, settings, and scoring methods. The experimental data was gathered from human and AI simulations and measured using motion-in-mind. It was found that increasing the number of options in the MCQ makes the test more challenging, explaining an increase of mass in mind $m$ . The findings also revealed that time pressure provides competitiveness while scaffolding provides support. In addition, the hybrid system demonstrates the balance of education and entertainment. Finally, the results addressed the general discussion and analogical interpretation in the education context based on physics-in-mind values. These findings can be promising for analyzing the balance between competitiveness and entertainment while enabling the learning process in the practical assessment.

Natural language processing technology has made significant progress in recent years, fuelled by increasingly powerful general language models. This has also inspired a sizeable body of work targeted specifically towards the educational domain, where the creation of questions (both for assessment and practice) is a laborious/expensive effort. Thus, automatic Question-Generation (QG) solutions have been proposed and studied. Yet, according to a recent survey of the educational QG community’s progress, a common baseline dataset unifying multiple domains and question forms (e.g., multiple choice vs. fill-the-gap), including readily available baseline models to compare against, is largely missing. This is the gap we aim to fill with this paper. In particular, we introduce a high-quality dataset in the educational domain, containing over 3,000 entries, comprising (i) multiple-choice questions, (ii) the corresponding answers (including distractors), and (iii) associated passages from the course material used as sources for the questions. Each question is phrased in two forms, normal and cloze (i.e., fill-the-gap), and correct answers are linked to source documents with sentence-level annotations. Thus, our versatile dataset can be used for both question and distractor generation, as well as to explore new challenges such as question format conversion. Furthermore, 903 questions are accompanied by their cognitive complexity level as per Bloom’s taxonomy. All questions have been generated by educational experts rather than crowd workers to ensure they are maintaining educational and learning standards. Our analysis and experiments suggest distinguishable differences between our dataset and commonly used ones for question generation for educational purposes. We believe this new dataset can serve as a valuable resource for research and evaluation in the educational domain. The dataset and baselines are made available to support further research in question generation for education (https://github.com/hadifar/question-generation).

The current study strived to delve into the response behavior and perceptions of examinees while taking a test in light of cognitive load theory. The empirical data were collected from 60 MA English major graduates and students, with a high level of language proficiency. The participants were required to answer 60 multiple-choice language items (i.e., grammar and vocabulary questions), taken from the high-stakes tests of the MA English majors of the Iranian university entrance examination (IUEE), as fast and as accurately as possible. After completing each test item, they rated their perceptions with regard to the difficulty of test items (Bratfisch et al., 1972) and the amount of mental effort (Paas, 1992). Their response time spent on each language item and their selected options were also stored by the Psychopy software (Peirce et al., 2019). Through running Pearson and Spearman rho correlations, the findings revealed that response time enjoyed a strong positive correlation with mental effort, meaning that both objective and subjective cognitive load measures matched in terms of sensitivity to cognitive load changes in language test items. Further, the subjective measures of perceived mental effort and perceived level of difficulty revealed to be the sound indicators of cognitive load changes. As predicted, response time also indicated that more difficult language test items imposed a greater amount of load. The implications of the study will be explained. [ABSTRACT FROM AUTHOR]

This study was part of a larger project to improve learning of undergraduate chemistry in Singapore through the use of self-authored three-tier multiple-choice questions (3TMCQs) and the giving/receiving of peer feedback. Specifically, we examined the quality of written feedback based on the classification by Hattie and Timperley (2007) that year 2 to 4 learners (N=31) gave each other on responses in their 3TMCQs (N=466 administered). It was found that the most common type of voluntary feedback given by test-makers was task (& self), followed by process (& self), self alone, and lastly regulation (& self) levels over seven chemistry courses. In addition, question type (based on revised Bloom's Taxonomy) had a marginal effect on the quality of feedback received; instead, items answered incorrectly garnered higher quality feedback and were four times more important than the cognitive level of questions. Feedback quality given by more experienced students was also no better than those given by less experienced ones. While there is growing evidence supporting the self-authoring of questions and giving/receiving peer feedback to enhance learning at undergraduate levels, further research is warranted into the types of peer feedback that learners may receive when attempting different question formats. [ABSTRACT FROM AUTHOR]

Background: Properly constructed single best-answer multiple choice questions (MCQs) or items assess higher-order cognitive processing of Bloom's taxonomy and accurately discriminate between high and low achievers. However, guidelines for writing good test items are rarely followed, leading to generation and application of faulty MCQs. Materials and Methods: During lockdown period in 2020, internal assessment was taken through online mode using Google Forms. There were 60 'single response type' MCQs, each consisting of single stem and four options including one correct answer and three distractors. Each item was analyzed for difficulty index (Dif I), discrimination index (DI), and distractor efficiency (DE). Results: The mean of achieved marks was 42.92± (standard deviation [SD], 5.07). Dif I, DI, and DE were 47.95± (SD 16.39) in percentage, 0.12± (SD 0.10), and 18.42± (SD 15.35), respectively. 46.67% of the items were easy and 21.66% were of acceptable discrimination. Very weak negative correlation was found between Dif I and DI. Out of total 180 distractors, 51.66% were nonfunctional one. Conclusion: Item analysis and storage of MCQs with their indices provides opportunity for an examiner to select MCQs of appropriate difficulty level as per the need of assessment and decide their placement in the question paper.

AimTo compare the frequency of misconceptions amongst dental students resulting from assessments in different subject areas using different types of multiple-choice questions (MCQs). We wanted to know whether misconceptions, or strongly held incorrect beliefs, differed by subject area or question type.MethodsA total of 104 students completed two assessments that included 20 MCQs on endodontics and 20 MCQs on dental implants. On each examination, 10 questions were scenario-type questions requiring interpretation or analysis and 10 questions were factual-based, knowledge questions. Incorrect responses and confidence levels by student and subject were recorded for a comparison of average misconceptions by question type and for correlations between scenario and knowledge question types for misconceptions on both assessments.ResultsStudents were overly confident on their incorrect responses and misconceptions for both assessments. On the endodontic examination, students held a statistically significant higher number of mean misconceptions on scenario questions than for knowledge questions, but the difference was not statistically significant for the dental implant examination. There was a moderately weak relationship between scenario and knowledge questions for misconceptions on the endodontic (r=.31) and dental implant (r=.20) assessments, suggesting students who have misconceptions on knowledge questions are somewhat more likely to have misconceptions on scenario questions.ConclusionStudents had a consistent rate of overconfidence (75%) in their incorrect responses regardless of question type or dental subject. Questions that prompted a higher per cent of incorrect responses were more likely to detect misconceptions, as students were highly confident in their mistakes, for both assessments.