Using a Glicko-Based Algorithm to Measure In-Course Learning

One significant challenge in the field of measuring ability is measuring the current ability of a learner while they are learning. Many forms of inference become computationally complex in the presence of time-dependent learner ability, and are not feasible to implement in an online context. In this paper, we demonstrate an approach which can estimate learner skill over time even in the presence of large data sets. We use a rating system derived from the Elo rating system and its relatives, which are commonly used in chess and sports tournaments. A learner's submission of a course assignment is interpreted as a single match. We apply this approach to Coursera's online learning platform, which includes millions of learners who have submitted assignments tens of millions of times in over 3000 courses. We demonstrate that this provides reliable estimates of item difficulty and learner ability. Finally, we address how this scoring framework may be used as a basis for various applications that account for a learner's ability, such as adaptive diagnostic tests and personalized recommendations. [For the full proceedings, see ED599096.]