Cost-forced collective potentiality maximization by complementary potentiality minimization for interpreting multi-layered neural networks.

The present paper aims to propose a new type of information-theoretic method to augment the ratio of collective potentiality to its cost for the interpretation. Collective potentiality represents how many components, such as connection weights or neurons, are collectively used to deal with relations between inputs and outputs, while individual potentiality shows how an individual component potentially contributes to the inputs or outputs. From our viewpoint, the conventional interpretation methods have focused on the individual potentiality of components, ignoring these collective behaviors. Thus, the present paper stresses that neural networks should try to deal with the collective properties of components and to examine the final results from multiple points of view. For implementing this concept of potentiality maximization, we introduce the complementary potentiality minimization, which aims to reduce the strength of larger weights as much as possible, and at the same time, it can be used for increasing the potentiality. The method was applied to two intuitively interpretable data sets, namely, the absenteeism and online shoppers data sets. With both data sets, experimental results confirmed that the method could increase collective potentiality and considerably reduce the corresponding cost. The new method could extract many groups of weights with the same small strength, all of which tried to respond to the coming inputs, while the conventional methods, including the regularization ones, tried to produce a smaller number of stronger connection weights very selectively. In particular, the present method could produce stable compressed weights similar to the original correlation coefficients of the data sets, meaning that simple, independent, and linear relations could be detected, contrary to the presupposed non-linear ones. Thus, the collective interpretation, applied to two business data sets, revealed a possibility that, behind seemingly complicated non-linear relations between inputs and outputs, neural networks with collective potentiality augmentation with a smaller cost can extract the very simple relations for easy interpretation. All complicated relations can possibly be generated, based on those simple ones. [ABSTRACT FROM AUTHOR]