The Modified 5-Year-Olds' Index : testing reliability and the impact of training level

Aim: To test the reliability of the Modified 5-Year-Olds’ Index using assessors with a range of experience, and to determine whether calibration improves reliability. Design: Prospective reliability study. Setting: Bristol Dental School, University of Bristol. Methods: Fifty study models of non-syndromic 5-year-olds with a repaired UCLP were selected from the CCUK archives. Fifteen participants with a range of clinical experience in orthodontics were divided into three groups of equal experience. Each group was given differing amounts of information: Group 1 information sheet; Group 2 information sheet and reference models; Group 3 calibration course, information sheet and reference models. Each participant scored the 50 models using the Modified 5-Year-Olds’ Index on two occasions at least four weeks apart. ICCs calculated from a two-way random effects model were used to calculate intra-rater reliability and inter-rater reliability comparing assessors’ scoring to the experts’ consensus scores (gold standard). Results: Group 2 (ICC 0.80 – 0.93) and Group 3 (ICC 0.80 – 0.91) demonstrated high levels of intra-rater agreement, with lower levels shown by Group 1 (0.68 – 0.93). Inter-rater agreement was high in Group 2 (ICC 0.87 – 0.93) and Group 3 (0.82 – 0.91), with Group 1 showing the lowest levels of agreement (ICC 0.69 – 0.94). The level of training of the assessors in Group 1 appears to influence reliability scores, with high intra- and inter-rater reliability scores of the consultant and post-CCST trainee at a similar level to those of the same grade in Groups 2 and 3. Conclusions: The Modified 5-Year-Olds’ Index is a reliable method of assessing outcomes when model scoring is carried out by consultants or post-CCST trainees. Calibration in use of the Index does not improve reliability.