Search

Humans have rich experience applying linear models and logical thinking, but only experts understand the behaviour of non-linear systems. However, the deep neural network (DNN) implementation of text non-linear systems outperforms optimal linear models. Therefore, the forward DNN (the pattern recognition system in this paper) attracts attention to the necessity of interpreting the results obtained by DNN. To preserve the high performance of DNN, we focus on a post-hoc explanation; this approach means building an explainable model for the decision obtained by the black box. To avoid the interpretation of a set of millions of non-linear functions, we divide DNN into two parts: the feature extractor and the classifier. Following that, we argue for a specific interpretation of each of them. While for classifiers, we have several suitable explainable models (and we decided on the fuzzy logical function), we believe that feature interpretation is a creative scientific activity corresponding to the usual research. The paper presents a tool to help researchers and users understand extracted features not necessarily known in the specific application domain. Explaining the new features offers a way to learn from computers.