1. Activation-Based Cause Analysis Method for Neural Networks
- Author
-
Hyung Il Koo and Yong Gyun Kim
- Subjects
Root (linguistics) ,Focus (computing) ,Optimization problem ,explaining classification ,General Computer Science ,Artificial neural network ,business.industry ,Computer science ,General Engineering ,interpretable machine learning ,Machine learning ,computer.software_genre ,Variety (cybernetics) ,Task (project management) ,Visualization ,TK1-9971 ,Range (mathematics) ,Multilayer perceptron ,General Materials Science ,Artificial intelligence ,Electrical engineering. Electronics. Nuclear engineering ,Electrical and Electronic Engineering ,business ,computer - Abstract
As neural networks are ubiquitous in the real-world environment, we encounter a variety of situations in which neural network-based systems fail. To address these failures, we need to identify their root causes. However, due to the black-box nature of neural networks, it is considered a difficult task to explain/understand internal functions, and cause analysis is usually conducted based on personal experience. To alleviate these problems, we propose a method to compute an element-wise contribution of inputs on current decisions. To this end, we focus on the physical meaning of neuron activations, and formulate an optimization problem that finds partial activations that support current decisions. Then, by accumulating these partial activations in the backward direction, we evaluate the contributions of each element in an input vector to the current decision. Experimental results have shown that the proposed method outperforms conventional methods in terms of cause localization for a range of failure scenarios.
- Published
- 2021