The validation of a rating scale to assess dietitians use of behaviour change skills

Background: Evidence suggests that education alone is unlikely to elicit dietary-behavioural change (Contento, 1995). Consequently, many dietitians are moving from a traditional advice-giving role to one which utilises ‘behaviour change skills’ (BCS) in dietary counselling. BCS is an umbrella term used to cover a wide range of skills and techniques drawn from the fields of counselling, motivational interviewing (MI) and cognitive behavioural therapy (CBT). In order to assess the efficacy of this approach, a means of quantifying BCS-use is required. This two-stage study aimed to validate a newly-devised scale to assess dietitians’ BCS-use in one-to-one dietary counselling. Methods: Items for the scale were generated by drawing on the literature, syllabi for training in BCS and its parent disciplines (counselling, MI and CBT), and specialist dietitians. The resulting scale and manual were revised following assessment of content validity by expert panel and piloting. In stage one, 21 dietetic consultations were audiotaped and rated for BCS-use by three BCS-trained dietitians. Inter-rater agreement was calculated using the kappa statistic and intra-class correlation (ICC), to give a ‘chance corrected’ measure of agreement. Validity was tested using a psychologist's subjective assessment of BCS-use as a proxy ‘gold-standard’ compared with the dietitians’ ratings, again using kappa and ICC. In stage two the scale was further revised before an additional 20 audiotaped consultations were analysed using the same procedure. Ethical approval for the study was given by the appropriate NHS and university research ethics committees. Results: At stage one, although kappas were fairly poor for agreement on individual criteria, the ICC for overall scores indicated a ‘fair’ level of agreement, according to Shrout's (1998) classifications: ICC = 0.584 (CI 0.339–0.784). Results for validity were poor with the psychologist frequently rating higher than the dietitians. At stage two, following scale revision, results for inter-rater agreement improved with more criteria showing ‘moderate’ or ‘substantial’ agreement. Ten out of the 21 criteria achieved levels of agreement classified as ‘fair’ or higher for all three rater pairs. The ICC for overall scores improved to indicate ‘moderate’ agreement: ICC = 0.640 (CI 0.404–0.821). Validity results remained poor. Discussion: The moderate level of overall inter-rater agreement observed in the revised scale is considered acceptable (Jones, 2006) and indicates this tool is useful. This measure is more relevant to the purpose of the tool than agreement on individual criteria given it is intended to classify consultations overall as low/medium/high use of BCS rather than to examine individual skills. However, in terms of validity, the discrepancy between dietitian and psychologist ratings requires further investigation. It is hypothesized that the dietitians had higher expectations of what a dietitian could achieve in terms of proficiency in BCS and, as such, rated more stringently than the psychologist. Achieving a clear picture of validity usually necessitates a series of assessments (Murphy & Davidshofer, 2005); the BCS rating scale is no exception with further testing required. Conclusions: The revised scale shows acceptable inter-rater reliability and robust content validity in our study sample. However, quantitative examination of validity gave poor results and further assessment is required to provide a tool with which we can confidently assess dietitians’ use of BCS. References Contento, I., Balch, G.I., Bronner, Y.L. et al. (1995) The effectiveness of nutrition education and implications for nutrition education policy, programs, and research: a review of the research. J. Nutr. Educ.27, 355–364. Jones, J.M. (2006) Nutritional Screening and Assessment Tools. New York: Nova Science Publishers. Murphy, K.R. & Davidshofer, C.O. (2005) Psychological Testing – Principles and Applications, 6th edn. New Jersey: Pearson Education Inc. Shrout, P. (1998) Measurement reliability and agreement in psychiatry. Stat. Methods Med. Res. 7, 301–317.