Fit Indices of Dynamic Bayesian Networks: A Comparison Based on Posterior Predictive Model Checking

Research in education has become increasingly reliant on statistical modeling frameworks to be reflective of the subject matter, to accurately assess what students know and can do, to assist instructors with curriculum design by supplementing informative feedback, and to support policy-makers when making evidence-based decisions. A common feature of these assessment uses is their requirement of valid inferences from the statistical models. From a methodological perspective, this requires the underlying statistical modeling framework to be thoroughly evaluated. Advances in modern technology and learning science bring about unique opportunities for learning and assessment (Mislevy et al., 2014), such as cognitive tutoring systems and simulation-based and game-based assessments. However, the statistical models that are used to analyze the data from such technology assessments must be validated. Due to their representational power to complex learning theories, their capability to model complex assessment data, and their potential to support real-time diagnosis, Bayesian networks (BNs) and the extension--dynamic Bayesian networks (DBNs)--are attracting more and more researchers in educational measurement. However, existing literature in this area has focused dominantly on model evaluations of static BNs and their findings may not necessarily be generalizable to DBNs. In this study I research DBNs as a means to model changes in assessed knowledge and with a major focus on evaluating the performance of selected fit indices to detect misspecifications of temporal dependence of DBNs. Based on a Monte Carlo simulation, seven indices--logarithmic loss, mean absolute error, mean squared error, relative absolute error, root mean square error, root mean squared logarithmic error, and root relative squared error--were compared in terms of their performance to detect two types of specification errors of DBNs: edge inclusion and edge exclusion. In addition, sample size and test length were also manipulated to examine their relationship to the performance of the studied fit indices. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]