Search

Introduction: The OSCE is a sociomaterial assemblage-a meshing together of human and material components producing multiple effects. Materials matter because they shape candidate performance, with potentially calamitous career consequences if materials influence performance unjustly. Although the OSCE literature refers to materials, few papers study the sociomateriality of OSCEs. Therefore, we explored OSCE stakeholders' talk about sociomaterial assemblages to better understand their importance for candidate performance., Methods: We conducted 15 focus groups with OSCE candidates (n = 42), examiners (n = 20) and simulated patients (n = 17) after an Australian postgraduate nursing OSCE. Sociomateriality informed our team-based framework analysis of data., Results: Participants identified a multiplicity of OSCE materials (objects, technologies and spaces) thought to matter for candidate performance. Candidates' unfamiliarity with materials and missing or malfunctioning materials were reported to yield numerous negative impacts (eg cognitive overload, negative affect, time-wasting), thereby adversely affecting candidate performance. Both examiners and candidates made micro-adjustments to sociomaterial assemblages during the OSCE in order to make it work (eg candidates saying what they would do rather than doing it). Sometimes, such tinkering extended so far that sociomaterial assemblages were ruptured (eg examiners ignoring rubrics to help pass candidates), potentially influencing OSCE standardisation., Discussion: Our novel empirical study extends previous conceptual work by illustrating wide-ranging sociomaterial assemblages influencing OSCE candidate performance. Further research is now needed employing sociomaterial approaches to further elucidate sociomaterial entanglements in diverse OSCEs. We encourage OSCE stakeholders to become more attuned to the productive nature of materials within all stages of OSCE design and implementation., (© 2021 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Background: Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs., Methods: Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed., Results: Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates., Conclusions: Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload., (© 2020 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: There is increasing agreement that graduates who finish tertiary education with the full complement of skills and knowledge required for their designated profession are not 'work-ready' unless they also acquire interpersonal, collaborative practice and team-working capabilities. Health workers are unable to contribute to organisational culture in a positive way unless they too attain these capabilities. These capabilities have been shown to improve health care in terms of patient safety, worker satisfaction and health service efficiency. Given the importance of interprofessional learning (IPL) which seeks to address these capabilities, why is IPL not consistently embedded into the education of undergraduates, postgraduates and vocationally qualified personnel through formal assessment?, Methods: This paper offers an argument for the formal assessment of IPL. It illustrates how the interests of the many stakeholders in IPL can benefit from, and contribute to, the integration of IPL into mainstream professional development and tertiary education. It offers practical examples of assessment in IPL which could drive learning and offer authentic, contextual teaching and learning experiences to undergraduates and health workers alike., Conclusions: Assessment drives learning and without formal assessment IPL will continue to be viewed as an optional topic of little relative importance for learners. In order to make the next step forward, IPL needs to be recognised and endorsed through formal assessment, both at the tertiary education level and within the workplace environment. This is supported by workforce initiatives and tertiary education policy which can be used to specify the capabilities or generic skills necessary for effective teamwork and collaborative practice.

Objectives: This study aims to identify whether non-science graduates perform as well as science graduates in Basic and Clinical Sciences (B & CS) assessments during Years 1-3 of a four-year graduate-entry programme at the University of Sydney (the 'USydMP')., Methods: Students were grouped into five categories: Health Professions (HP), Biomedical Sciences (BMS), Other Biology (BIOL), Physical Sciences (PHYS) or Non-Science (NONS). We examined the performance rank of students in each of the five groups for single best answer (SBA) and modified essay (MEQ) assessments separately, and also calculated the relative risk of failure in the summative assessments in Years 2 and 3., Results: Students with science-based prior degrees performed better in the SBA assessments. The same occurred initially in the MEQs, but the effect diminished with time. The HP students performed consistently better but converged with other groups over time, particularly in the MEQs. Relative performance by the NONS students improved with time in both assessment formats. Overall, differences between the highest and lowest groups were small and very few students failed to meet the overall standard for the summative assessments. HP and BMS students had the lowest failure rate. NONS students were more likely to fail the assessments in Year 2 and 3, but their pass rates were still high. Female students performed significantly better overall at the end of Year 2 and in Year 3. There were only minor differences between Australian resident and International students., Conclusion: While there are small differences in performance in B & CS early in the programme, these lessen with time. The study results will inform decisions regarding timing of summative assessments, selection policy and for providing additional support to students who need it to minimize their risk of failure. Readers should note that this paper refers to student performance in only one of the four curriculum themes, where health professional and science graduates would be expected to have a significant advantage.

Background: Debate abounds regarding the most appropriate candidates to admit to medical school. This paper examines whether there is any advantage to admitting 'graduate' entrants over secondary school leavers on selected medical school and practice outcomes., Aim: To compare the medical school experiences, research and academic achievements and practice outcomes of graduates who entered 1 medical school in Australia directly from high school (secondary school entry) to those of graduates who entered with tertiary level education (tertiary entry)., Design and Methods: Cross-sectional study using a mail-out survey to graduates from the first 16 graduating years (1983-98 inclusive) of the University of Newcastle Medical School., Results: Secondary school entrants were, on average, 8 years younger than tertiary entrants and were less likely to have received rural-based schooling. However, there were no differences with respect to gender or type of secondary school attended (public or private). Motivations for studying medicine did not generally differ according to entry type, except that more secondary students were motivated by parental expectations and more tertiary entrants were motivated by the need for professional independence and the desire to prevent disease. A greater proportion of tertiary entrants experienced stress at medical school. However, secondary students experienced more stress due to doubts about being a doctor, while tertiary entrants experienced more stress due to lack of leisure time, finances and balancing commitments. There were no significant differences between the groups in terms of academic performance (as measured by the award of medical school honours) or research outcomes (as measured by completion of a research degree during or after medical school training, publication of scientific papers or holding career posts in the research sciences). There were no differences in career positions held by clinicians, choice of general practice or another specialty as a career, practice location (rural or urban) or employment sector (public or private)., Conclusion: There is no clear advantage, at least on the outcomes measured in this study, to limiting medical school entry to either those candidates from secondary school or those with tertiary backgrounds. Medical schools could reasonably broaden their selection criteria to include more graduate entry candidates in addition to secondary school leavers without compromising medical school and practice outcomes.

Misconceptions exist in the community regarding factors that exacerbate acne vulgaris. In particular stress, diet, lifestyle and personal hygiene are often erroneously claimed to be important factors. In order to investigate whether these common misconceptions persist in medical graduates, we analysed the answers of 215 sixth year medical students, who all subsequently graduated from The University of Melbourne, to a short-answer question on acne management and exacerbating factors in one of their final year examination papers. With respect to exacerbating factors, 67% of students identified stress, 10% identified lifestyle factors (smoking and alcohol consumption), and 25% claimed poor facial hygiene exacerbated acne. Diet was stated to be an important factor by 41% of students, of whom 12% specifically mentioned chocolate. Persistence of these misconceptions among medical graduates is likely to perpetuate misinformation in the community.

The examination for Fellowship of the Royal Australian College of General Practitioners is a long-standing examination that is regarded as a valid and comprehensive assessment of competence for general practice in Australia. Income and employment opportunities now depend on passing this battery of eight subtests. This paper describes the structures of the examination and reports an analysis of the reliability and efficiency of the examinations, based on data from five consecutive examinations. Results indicate that the examination achieves acceptable overall reliability, although suggestions for changes to improve reliability are made. A decision on implementing any changes must take into consideration any consequent changes in validity.

This paper reports a study which compared the performance of different groups of students and doctors on identical and equivalent tests set in an objective-type format and in a free-response format. The tests were designed to ensure that the content was relevant to clinical practice at the hospital intern level. In all test situations candidates' scores were significantly higher in the objective tests than in the free-response tests. This difference was greater for the more junior and less competent students than for the more competent doctors. The cueing effect of the options was thought to be the main factor responsible for the difference in performance. The results of a questionnaire survey demonstrated that students were aware of the deficiencies in multiple-choice tests. A large majority of the students believed that the free-response tests gave a more accurate assessment of their clinical ability. It was found that in these tests, aimed at measuring aspects of clinical competence, multiple-choice questions appeared to overestimate the candidate's ability to an extent that made them less suitable than free-response questions for this purpose. It was also found that free-response tests, of the type used in this study, provide a suitable alternative to multiple-choice tests for use in the written section of clinical examinations. It was concluded that the written component of the final examination in the medical course should have a preponderance of free-response items over multiple-choice items.

This paper has reported some aspects of an analysis of the correlates of success in candidates' first attempts at the postgraduate psychiatric examinations of the R.A.N.Z.C.P. These are developed to suggest some attributes which selectors of trainees could take into account. Profitable future areas for research are delineated.

In a previous study we described a problem-based criterion-referenced test of the clinical competence of medical students which was felt to offer advantages over the traditional final-year examination. This paper reports the validity and reliability studies on which it is possible to judge the value of this new test when compared to the traditional approach. The results demonstrate a high level of content validity and provide evidence of the construct validity of the test. Efforts to obtain measures of concurrent and predictive validity were thwarted by a failure to attain reliable assessments of ward performance from resident and consultant staff. Satisfactory levels of internal consistency were established for the whole test. Marker reliability was satisfactory in all sections of the test except for those requiring examiners to rate practical clinical skills. This was so despite the use of simulated patients, behavioural check-lists and rater training. Possible solutions to this problem are discussed. It is concluded that this new approach overcomes many of the measurement problems inherent in the traditional final examination. It has been shown to be feasible to construct and administer in the medical school setting without the need for the allocation of additional resources.

This paper describes a situation where an alteration in the final-year assessment scheme led to changes in student learning activities which were the exact opposite of those intended. Students were seen to be spending a disproportionate amount of time studying the theoretical components of the course relative to the practical and clinical aspects. The paramount importance of the assessments and examinations in influencing student learning behaviour led the departments concerned to develop a new clinical examination which more clearly reflected the objectives of the course. A questionnaire survey was undertaken to determine how the different sections of the final assessment affected the students' approach to studying. The questionnaire was administered to graduates during their intern year for the 3 years following the introduction of the new clinical examination. Results were also obtained for the year preceding the change. The survey showed that the students developed a high regard for the new examination and its validity as a test of clinical competence. The students found that an increase in ward-based learning activities was essential for success in the final examinations. The new clinical examination has thus influenced students' learning and successfully restored the balance of their learning activities between the clinical and theoretical components of the course.

The method chosen for important examinations strongly influences the nature of student learning. The Newcastle Medical School in Australia developed a 5-year problem-based curriculum and adopted the Modified Essay Question (MEQ) as the main written instrument for assessing students' problem-solving skills. Even with the best of intentions the MEQ has been abused by its over-use, as shown in this review of annual assessment for years 1, 3 and 5 of the course. Analysis of staff and students' dissatisfaction with this instrument showed some difficulties attributable to the nature of MEQ items. However, a number of problems also emerged as a result of subtle changes in the teaching staff's approach to learning and assessment. This paper concludes by acknowledging the strengths and limitations of the MEQ. It recommends ways in which the MEQ can be effective, as part of a balanced programme involving other suitable instruments.