Evidence-Centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments

We are currently at an exciting juncture in developing effective means for assessing so-called 21st-century skills in an innovative yet reliable fashion. One of these avenues leads through the world of "epistemic games" (Shaffer, 2006a), which are games designed to give learners the rich experience of professional practica within a discipline. They serve to develop domain-specific expertise based on principles of collaborative learning, distributed expertise, and complex problem-solving. In this paper, we describe a comprehensive research program for investigating the methodological challenges that await rigorous inquiry within the epistemic games context. We specifically demonstrate how the "evidence-centered design framework" (Mislevy, Almond, & Steinberg, 2003) as well as current conceptualizations of reliability and validity theory can be used to structure the development of epistemic games as well as empirical research into their functioning. Using the epistemic game "Urban Science" (Bagley & Shaffer, 2009), we illustrate the numerous decisions that need to be made during game development and their implications for amassing qualitative and quantitative evidence about learners' developing expertise within epistemic games. (Contains 4 figures.)