Evaluating Machine Learning for Projecting Completion Rates for VET Programs. Technical Paper

This paper summarises exploratory analysis undertaken to evaluate the effectiveness of using machine learning approaches to calculate projected completion rates for vocational education and training (VET) programs, and compares this with the current approach used at the National Centre for Vocational Education Research (NCVER) -- Markov chains methodology. While the Markov chains methodology currently used by NCVER has demonstrated that it is reliable, with predictions aligning well with the actual rates of completion for historical estimates, it has not been reviewed for some time and it does have some limitations. The evaluation of machine learning techniques for predicting VET program completion rates was undertaken to overcome some of these limitations and with a view to improving our current predictions. This report includes: (1) an overview of the methodologies: Markov chains and two machine learning algorithms that were applied to predict completion rates for VET programs (XGBoost and CatBoost); (2) a comparison of the accuracy of the predictions generated by both methodologies; and (3) an evaluation of the relative strengths and limitations of both methodologies.