Search

Causal machine learning methods can be used to search for treatment effect heterogeneity in high-dimensional datasets even where we lack a strong enough theoretical framework to select variables or make parametric assumptions about data. This paper uses causal machine learning methods to estimate heterogeneous treatment effects in the case of an experimental study carried out in Morocco which evaluated the effect of conditionalizing a cash transfer program on several outcomes including maths test scores which is the focus of this work. We explore treatment effects across a dataset of 1936 pre-treatment variables. For the most part, heterogeneity is modelled by two different factors, participation in education (at the baseline) and more general measures of poverty. Those who are more disadvantaged at the baseline benefit less from any treatment. While conditioning generally has a negative effect this more disadvantaged group is also hurt more by conditioning. The second purpose of this paper is to demonstrate and reflect upon a causal machine learning approach to policy evaluation. We propose a novel causal tree method for interpretable modelling of causal effects and reflect on the difficulty of explaining atheoretical results., Comment: 24 pages, 8 figures

Causal machine learning tools are beginning to see use in real-world policy evaluation tasks to flexibly estimate treatment effects. One issue with these methods is that the machine learning models used are generally black boxes, i.e., there is no globally interpretable way to understand how a model makes estimates. This is a clear problem in policy evaluation applications, particularly in government, because it is difficult to understand whether such models are functioning in ways that are fair, based on the correct interpretation of evidence and transparent enough to allow for accountability if things go wrong. However, there has been little discussion of transparency problems in the causal machine learning literature and how these might be overcome. This paper explores why transparency issues are a problem for causal machine learning in public policy evaluation applications and considers ways these problems might be addressed through explainable AI tools and by simplifying models in line with interpretable AI principles. It then applies these ideas to a case-study using a causal forest model to estimate conditional average treatment effects for a hypothetical change in the school leaving age in Australia. It shows that existing tools for understanding black-box predictive models are poorly suited to causal machine learning and that simplifying the model to make it interpretable leads to an unacceptable increase in error (in this application). It concludes that new tools are needed to properly understand causal machine learning models and the algorithms that fit them., Comment: 31 pages, 8 figures

Causal machine learning methods which flexibly generate heterogeneous treatment effect estimates could be very useful tools for governments trying to make and implement policy. However, as the critical artificial intelligence literature has shown, governments must be very careful of unintended consequences when using machine learning models. One way to try and protect against unintended bad outcomes is with AI Fairness methods which seek to create machine learning models where sensitive variables like race or gender do not influence outcomes. In this paper we argue that standard AI Fairness approaches developed for predictive machine learning are not suitable for all causal machine learning applications because causal machine learning generally (at least so far) uses modelling to inform a human who is the ultimate decision-maker while AI Fairness approaches assume a model that is making decisions directly. We define these scenarios as indirect and direct decision-making respectively and suggest that policy-making is best seen as a joint decision where the causal machine learning model usually only has indirect power. We lay out a definition of fairness for this scenario - a model that provides the information a decision-maker needs to accurately make a value judgement about just policy outcomes - and argue that the complexity of causal machine learning models can make this difficult to achieve. The solution here is not traditional AI Fairness adjustments, but careful modelling and awareness of some of the decision-making biases that these methods might encourage which we describe., Comment: 13 pages, 1 figure

Methods for learning optimal policies use causal machine learning models to create human-interpretable rules for making choices around the allocation of different policy interventions. However, in realistic policy-making contexts, decision-makers often care about trade-offs between outcomes, not just single-mindedly maximising utility for one outcome. This paper proposes an approach termed Multi-Objective Policy Learning (MOPoL) which combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach to explore the trade-off between multiple outcomes. It does this by building a Pareto frontier of non-dominated models for different hyperparameter settings which govern outcome weighting. The key here is that a low-cost greedy tree can be an accurate proxy for the very computationally costly optimal tree for the purposes of making decisions which means models can be repeatedly fit to learn a Pareto frontier. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya., Comment: 24 pages, 7 figures

The COVID-19 pandemic hit many countries at a time when their education systems were facing multiple challenges. Economic, public-health and social impacts from the pandemic have exacerbated many of these challenges. The aim of this paper is to explore the way in which the behavioural sciences can help support the policy response to the COVID-induced education crisis, and to serve as a learning experience for other future crises. The paper involves an empirical exploration of the factors associated with a range of outcomes using large nationally representative datasets, and interpreting these relationships in the context of a detailed literature review. By using data that it is generally representative of the populations of interest, and is available for many dozens of countries with different histories, languages, cultures, and socioeconomic outcomes, this paper highlights how identifying behavioural biases can direct education systems towards more effective targeted policy interventions.

This report examines two sets of issues, the first being whether Indigenous Australians obtain a lower return on investment in education and training than other Australians. If they do, then this would partly explain why, in general, Indigenous participation in education and training is relatively low. The second issue is whether Indigenous participation is different once background characteristics--such as remoteness--are taken into account. To investigate these questions, the research uses previous research and a number of datasets: the National Aboriginal and Torres Strait Islander Social Survey, the Census of Population and Housing, the Australian Early Development Index and the Longitudinal Surveys of Australian Youth. The overall message is that, on the whole, Indigenous Australians have a positive return from education and training. Therefore it can be concluded that differential returns are not especially important in understanding differences in participation. The authors also find that, almost universally, background characteristics (including academic achievement at an earlier age) do not explain differential participation. Differences appear at an early age and then compound through the schooling system. (Contains 3 footnotes.)