Evaluating Active Learning Heuristics for Sequential Diagnosis

Given a malfunctioning system, sequential diagnosis aims at identifying the root cause of the failure in terms of abnormally behaving system components. As initial system observations usually do not suffice to deterministically pin down just one explanation of the system's misbehavior, additional system measurements can help to differentiate between possible explanations. The goal is to restrict the space of explanations until there is only one (highly probable) explanation left. To achieve this with a minimal-cost set of measurements, various (active learning) heuristics for selecting the best next measurement have been proposed. We report preliminary results of extensive ongoing experiments with a set of selection heuristics on real-world diagnosis cases. In particular, we try to answer questions such as "Is some heuristic always superior to all others?", "On which factors does the (relative) performance of the particular heuristics depend?" or "Under which circumstances should I use which heuristic?"
Comment: This work was presented at the International Workshop on Principles of Diagnosis 2018 (DX-2018) and a version of this work was formally published as "Patrick Rodler, Wolfgang Schmid. On the impact and proper use of heuristics in test-driven ontology debugging. RuleML+RR, 2018."