Why Deep Knowledge Tracing Has Less Depth than Anticipated

Knowledge tracing allows Intelligent Tutoring Systems to infer which topics or skills a student has mastered, thus adjusting curriculum accordingly. Deep Knowledge Tracing (DKT) uses recurrent neural networks (RNNs) for knowledge tracing and has achieved significant improvements compared with models like Bayesian Knowledge Tracing (BKT) and Performance Factor Analysis (PFA). However, DKT is not as interpretable as other models because the decision-making process learned by recurrent neural networks is not wholly understood by the research community. In this paper, we critically examine the DKT model, visualizing and analyzing the behaviors of DKT in high dimensional space. We modify and explore the DKT model and discover that Deep Knowledge Tracing has some critical pitfalls: 1). instead of tracking each skill through time, DKT is more likely to learn an 'ability' model; 2) the recurrent nature of DKT reinforces irrelevant information that it uses during the tracking task; 3) an untrained recurrent network can achieve similar results to a trained DKT model, supporting a conclusion that recurrence relations are not properly learned and, instead, improvements are simply a benefit of projection into a high dimensional, sparse vector space. Based on these observations, we propose improvements and future directions for conducting knowledge tracing research using deep models. [For the full proceedings, see ED599096.]