Risk Metrics to Measure Safety Performance of the National Airspace System: Implementation Using Machine Learning

Metrics are the key tools to monitor the safety performance of complex systems like air traffic control. Developing risk metrics is a complex process, which involves prediction of risk and identification of different potential outcomes of undesired safety events. For instance, a metric that helps monitor the risk on the surface in an airport environment needs to detect and identify events such as runway excursions, runway incursions, and taxiway incidents and assign appropriate numerical indices proportional to the outcome of each event. An accident with a fatality, injury and aircraft damage will have a corresponding weight for each outcome and incident that involves no outcome can be assigned a severity weight based on its probability of becoming an accident. Models that support such metrics needs to be able to process data from diverse sources. In this paper, we discuss key aspects of two comprehensive metrics that the office of Safety in the Air Traffic Organization has deployed recently: 1) How to employ an automation to detect all relevant events from different data sources and 2) how to assign severity weights for different types of events (accidents and incidents).1. Detection of Relevant Events: one key element of supporting a comprehensive metrics is an automated detection of relevant events by categories. Unlike in aviation, the use of AI and ML have permeated in most industries. Aviation is a safety-critical domain in which there is very little tolerance for failures. The stringent requirements of aviation have contributed to the slow adoption of AI and ML in aviation, in general. However, many aviation sectors are increasingly adopting AI systems, ranging from automating simple, yet tedious and repetitive tasks to a more sophisticated application of a complex autonomous air traffic control system to de-conflict traffic. In this paper, we outline how machine learning models were employed to identify relevant accidents and incidents to support two metrics, a surface safety metric and an airborne safety metric.2. Severity Weighting Scheme: for a metric to be a comprehensive measure and indicate the overall performance of the system, it needs to account for various types of accidents and incidents (precursors) that occur in the system. This paper shows how a weighting scheme we developed to measure the outcome of accidents, such as injuries to people and damage to property, as well as probabilistically determine the severity of incidents with no outcome. The aggregation of all weights along with the frequency of occurrence in a given period represent the overall safety performance of the system.