Development of 'VAScoR': A Rubric to Qualify and Score Responses to the Views of Nature of Science ('VNOS') Questionnaire

We report on the development of a rubric to reliably qualify and score responses to the "Views of Nature of Science Questionnaire" (VNOS): The "VNOS Analysis and Scoring Rubric" (VAScoR). The "VAScoR" is designed to (a) provide systematic guidance for the qualitative analysis, and score assignment to nuanced categories, of "VNOS" responses, (b) explicitly scaffold qualitative inferencing and standardize score assignment to substantially lessen the burden of, and variance in, analyzing and scoring the "VNOS," and (c) improve the viability and meaningfulness of cross-study comparisons drawing on "VNOS" data. The rubric adopted the "VNOS's" consensus NOS framework and further delineated core and related elements across 10 target NOS aspects. The "VAScoR's" reliability was examined in two studies that drew on "VNOS" questionnaires completed by 185 preservice secondary science teachers (58% female; 126 undergraduate and 59 graduate students) enrolled over several years in a combined undergraduate and graduate licensure program in a large U.S. Midwestern university. In Study I, "VAScoR" analyses of 86 "VNOS" questionnaires undertaken by a single author were used to examine the rubric's intra-rater reliability, which resulted in a robust Cronbach's alpha value of 0.81. In Study II, analyses by four authors of a randomly generated, overlapping set of 18 questionnaires were used to examine inter-rater reliability, which was supported with substantial consensus among raters as indicated by a Cohen's kappa of 0.71. Further evidence for the "VAScoR's" inter-rater reliability was indicated by moderate to strong consistency among four raters with an overall Pearson's correlation coefficient of 0.82, and coefficient values ranging from 0.77 to 0.89 for six possible rater pairings.