Using an Explainable Machine Learning Approach to Characterize Earth System Model Errors: Application of SHAP Analysis to Modeling Lightning Flash Occurrence.

Computational models of the Earth System are critical tools for modern scientific inquiry. Efforts toward evaluating and improving errors in representations of physical and chemical processes in these large computational systems are commonly stymied by highly nonlinear and complex error behavior. Recent work has shown that these errors can be effectively predicted using modern Artificial Intelligence (A.I.) techniques. In this work, we go beyond these previous studies to apply an explainable A.I. technique to not only predict model errors but also move toward understanding the underlying reasons for successful error prediction. We use XGBoost classification trees and SHapley Additive exPlanations analysis to explore the errors in the prediction of lightning occurrence in the NASA Goddard Earth Observing System model, a widely used Earth System Model. This explainable error prediction system can effectively predict the model error and indicates that the errors are strongly related to convective processes and the characteristics of the land surface. Plain Language Summary: Computer models of the Earth are very important tools in the modern Earth scientist's toolkit. Understanding when and why these models are wrong is a major challenge facing the scientific community. Work published in the last few years has shown that you can actually predict when these models are wrong using artificial intelligence (A.I.). We build on that work by applying existing fancy mathematical tools to these A.I. methods to understand why these computer models are wrong. We demonstrate this approach to predictions of lightning in a model created by NASA, and find that the lightning in the model is wrong in ways that are strongly related to convection in the atmosphere and some aspects of the land surface. Key Points: Errors in simulated lightning flash occurrence are learned through a machine learning classification approachAn explainable machine learning technique is used to explore the drivers of the simulation errorsThis error prediction system indicates that errors are strongly related to convective processes and the characteristics of the land surface [ABSTRACT FROM AUTHOR]