Reliability of non-pretested versus pretested questions in the applied knowledge test (AKT) of the MRCGP: evidence of quality assurance.

The applied knowledge test (AKT) is a computer-delivered multiple choice test offered three times a year. It forms part of the Membership of the Royal College of General Practitioners (MRCGP) examination, which licenses doctors for UK general practice. This paper investigates the effect of current quality assurance processes in the AKT on the reliability of questions which are not pretested. Test questions in the AKT are based on the RCGP curriculum blueprint, referenced to current evidence, peer-reviewed and critically appraised before they are added to the question bank. There is a standardised process for question selection and test construction. Standard setting for the AKT follows a modified Angoff process where a panel of experts make individual judgements on each question of the test estimating the likelihood of the 'just-passing candidate' answering correctly. In the three examinations delivered in 2011 (AKT 11 (January), 12 (May) and 13 (October)) Cronbach's alpha was 0.88, 0.92, 0.91, respectively. For pretested questions, the predicted alpha was 0.90, 0.93, 0.92 and for non-pretested questions 0.84, 0.91, 0.92, respectively adjusting for test length using the Spearman-Brown formula. Pretesting of questions is unnecessary provided there is a systematic process of question and test construction which quality assures the AKT. This ensures that pretesting does not delay the use of questions to examine current knowledge and guidelines. [ABSTRACT FROM AUTHOR]