Score decompositions in forecast verification

A forecast for an event should be based on a probability distribution of the possible outcomes of the event. In assessing the forecast, the forecast is given a penalty according to the outcome that occurs. The penalty, or score, is determined by a scoring rule and proper scoring rules are preferred; under a proper scoring rule, there is no incentive for a forecaster to issue a forecast that differs from the forecast they believe is appropriate. For proper scoring rules, the accuracy of a forecaster is defined to be their expected score over all possible forecasts and outcomes. Other measures of forecaster performance can be obtained by expressing accuracy as a sum of several terms, a process known as decomposing the accuracy. Each term of a decomposition measures a quality of the issued forecasts; qualities considered important are those that represent features of the joint distribution of the forecasts and outcomes. In the main decomposition, which we call the URR decomposition, the terms are uncertainty, resolution and reliability. The form of the URR decomposition is known for scoring rules of discrete events and their precise-probabilistic forecasts, which are forecasts issued as precisely-known probability distributions. We extend this URR decomposition to events with outcomes in any space and forecasts that may be functions of a probability distribution. We also determine in general form a second decomposition of accuracy, the RDC decomposition, in which the terms refer to the qualities of refinement, discrimination and correctness of the forecasts; the RDC decomposition has previously only be calculated in the specific instance of a binary event under the Brier scoring rule (Brier, 1950). The URR and RDC decompositions must be modified if the issued forecasts or recorded outcomes are separated into groups or bins before being assessed, and we give these amended decompositions. In a different setting, the URR and RDC decompositions we derive can also be used to examine the properties of interval-probabilistic forecasts. Interval-probabilistic forecasts are specific to binary events and issue a range of probabilities that the event will occur. There is little previous work on interval-probabilistic forecasts and to apply the URR and RDC decompositions we first define and characterise proper scoring rules for interval-probabilistic forecasts, known as interval-proper scoring rules, before establishing the URR and RDC decompositions of a particular interval-proper scoring rule, the interval-Brier scoring rule.