SQL-DP: A Novel Difficulty Prediction Framework for SQL Programming Problems

In an Intelligent Tutoring System (ITS), problem (or question) difficulty is one of the most critical parameters, directly impacting problem design, test paper organization, result analysis, and even the fairness guarantee. However, it is very difficult to evaluate the problem difficulty by organized pre-tests or by expertise, because these solutions are labor-intensive, time-consuming, leakage-prone, or subjective in some way. Thus, it is of importance to automatically evaluate problem difficulty via information technology. To this end, we propose a novel difficulty prediction framework, named SQL-DP, for Structured Query Language (SQL) programming problems, mastering which plays a vital role in learning the database technology. In SQL-DP, semantic features of problem stems and structure features of problem answers in the form of SQL codes are both computed at first, using the NLP and the neural network techniques. Then, these features are used as the input to train a difficulty prediction model with the statistic error rates in tests as the training labels, where the whole modeling does not introduce any experts, some as knowledge labeling. Finally, with the trained model, we can automatically predict the difficulty of each SQL programming problem. Moreover, SQL programming problem answering log data of hundreds of undergraduates from Guangxi University of China are collected, and the experiments conducted on the collected log data demonstrate the propped SQL-DP framework outperforms the state-of-the-art solutions apparently. In particular, SQL-DP decreases the RMSE of difficulty prediction by at most 7.23%, compared with the best-related framework. [For the full proceedings, see ED623995.]