Identifying Informative Features to Evaluate Student Knowledge as Causal Maps

Knowledge maps have been widely used in knowledge elicitation and representation to evaluate and guide students' learning. To effectively evaluate maps, instructors must select the most informative map features that capture students' knowledge constructs. However, there is currently no clear and consistent criteria to select such features, as empirical studies continue to reflect the (implicit) preferences of scholars. This is challenging for instructors, who may thus ignore critical aspects of a map and/or waste their efforts by examining highly correlated features. To address the research gap of selecting informative graph metrics to assess knowledge maps, this study adopts the machine learning technique of Unsupervised Feature Selection (UFS). Specifically, we extract 12 features used in the prior literature on map assessment (e.g., density, diameter) and use 8 UFS algorithms to rank their importance. By using 202 maps originating from four case studies, we identify features that are generally (un)informative and observe nuances due to context (e.g., learning task, participant profiles). Results suggest that features commonly reported (e.g., number of edges) may not be as informative as less commonly examined aspects (e.g., average degree). Differences exist between maps: the diameter is valuable when learners produce maps from detailed studies, but less informative when maps are directly elicited from the learners' perspectives. The 8 UFS algorithms show five distinct ways to rank features in maps, hence future works may elicit the preferences of instructors for grading and map these preferences to an algorithmic approach (i.e., UFS) that produces a desired ranking.