Search

The present paper aims to propose a new learning and interpretation method called "impartial competitive learning", meaning that all participants in a competition should be winners. Due to its importance, the impartiality is forced to be realised even by increasing the corresponding cost in terms of the strength of weights. For the first approximation, three types of impartial competition can be developed: componential, computational, and collective competition. In the componential competition, every weight should have an equal chance on average to win the competition. In the computational competition, all computational procedures should have an equal chance to be applied sequentially in learning. In collective computing for interpretation, all network configurations, obtained by learning, have an equal chance to participate in a process of interpretation, representing one of the most idealised forms of impartiality. The method was applied to a well-known second-language-learning data set. The intuitive conclusion, stressed in the specific science, could not be extracted by the conventional natural language processing methods, because they can deal only with word frequency. The present method tried to extract a main feature beyond the word frequency by competing connection weights and computational procedures impartially, followed by collective and impartial competition for interpretation. [ABSTRACT FROM AUTHOR]

The present paper aims to propose a new learning method based on destructive computing, contrary to the conventional progressive computing or the steady-step learning. In spite of the existence of a large amount of biased or distorted information in inputs, the conventional learning methods fundamentally aim to gradually acquire information that is as faithful as possible to inputs, which has prevented us from acquiring intrinsic information hidden in the deepest level of inputs. At this time, it is permitted to suppose a leap to that level by changing information at hand not gradually but drastically. In particular, for the really drastic change of information, we introduce the winner-lose-all (WLA) to drastically destroy the supposedly most important information for immediately reaching or leaping to intrinsic information, hidden in complicated inputs. The method was applied to a target-marketing problem. The experimental results show that, with the new method, multi-layered neural networks had an ability to disentangle complicated network configurations into the simplest ones with simple and independent correlation coefficients between inputs and targets. This was realized by drastically changing the information content in the course of learning and, correspondingly, by mixing regular and irregular properties over connection weights. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new information-theoretic method to produce explicit self-organizing maps (SOMs). Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the Gaussian function of distance between input patterns and competitive units. A property of this Gaussian function is that, as distance becomes smaller, a neuron tends to fire strongly. Cooperation processes are realized by taking into account the firing rates of neighboring neurons. We applied our method to uniform distribution learning, chemical compound classification and road classification. Experimental results confirmed that cooperation processes could significantly increase information content in input patterns. When cooperative operations are not effective in increasing information, mutual information as well as entropy maximization is used to increase information. Experimental results showed that entropy maximization could be used to increase information and to obtain clearer SOMs, because competitive units are forced to be equally used on average. Index Terms--Competition, cooperation, entropy maximization, Gaussian function, mutual information maximization, self-organizing maps (SOMs).

The present paper aims to propose a new information-theoretic method to minimize and maximize selective information repeatedly. In particular, we try to solve the incomplete information control problem, where information cannot be fully controlled due to the existence of many contradictory factors inside. For this problem, the cost in terms of the sum of absolute connection weights is introduced for neural networks to increase and decrease information against contradictory forces in learning, such as error minimization. Thus, this method is called a "cost-forced" approach to control information. The method is contrary to the conventional regularization approach, where the cost has been used passively or negatively. The present method tries to use the cost positively, meaning that the cost can be augmented if necessary. The method was applied to an artificial and symmetric data set. In the symmetric data set, we tried to show that the symmetric property of the data set could be obtained by appropriately controlling information. In the second data set, that of residents in a nursing home, obtained by the complicated procedures of natural language processing, the experimental results confirmed that the present method could control selective information to extract non-linear relations as well as linear ones in increasing interpretation and generalization performance. [ABSTRACT FROM AUTHOR]

The present paper aims to propose a new information-theoretic method to minimize and maximize selective information repeatedly. In particular, we try to solve the incomplete information control problem, where information cannot be fully controlled due to the existence of many contradictory factors inside. For this problem, the cost in terms of the sum of absolute connection weights is introduced for neural networks to increase and decrease information against contradictory forces in learning, such as error minimization. Thus, this method is called a "cost-forced" approach to control information. The method is contrary to the conventional regularization approach, where the cost has been used passively or negatively. The present method tries to use the cost positively, meaning that the cost can be augmented if necessary. The method was applied to an artificial and symmetric data set. In the symmetric data set, we tried to show that the symmetric property of the data set could be obtained by appropriately controlling information. In the second data set, that of residents in a nursing home, obtained by the complicated procedures of natural language processing, the experimental results confirmed that the present method could control selective information to extract non-linear relations as well as linear ones in increasing interpretation and generalization performance. [ABSTRACT FROM AUTHOR]

The present paper aims to propose an information-theoretic method for interpreting the inference mechanism of neural networks. The new method aims to interpret the inference mechanism minimally by disentangling complex information into simpler and easily interpretable information. This disentanglement of complex information can be realized by maximizing mutual information between input patterns and the corresponding neurons. However, because the use of mutual information has faced difficulty in computation, we use the well-known autoencoder to increase mutual information by re-interpreting the sparsity constraint, which is considered a device to increase mutual information. The computational procedures to increase mutual information are decomposed into the serial operation of equal use of neurons and specific responses to input patterns. The specific responses are realized by enhancing the results by the equal use of neurons. The method was applied to three data sets: the glass, office equipment, and pulsar data sets. With all three data sets, we could observe that, when the number of neurons was forced to increase, mutual information could be increased. Then, collective weights, or average collectively treated weights, showed that the method could extract the simple and linear relations between inputs and targets, making it possible to interpret the inference mechanism minimally. [ABSTRACT FROM AUTHOR]

The present paper aims to extend the potential learning method to overcome the problem of collective interpretation, which aims to interpret multi-layered neural networks by compressing them into the simplest ones. In the process of compression, positive, negative, and complicated weights have had unfavourable effects for interpretation. To deal with the problems of collective interpretation, the potential learning is extended only to use positive weights. In addition, to obtain more appropriate weights for interpretation, the number of candidate weights for higher potentialities is first increased as much as possible. Then, from among many candidates, more appropriate weights are selected as more important ones. This extended potentiality learning is expected to produce more stable and more simple representations for easy interpretation. The extended method was applied to three datasets, namely, an artificial dataset, a real eye-tracking dataset, and a student evaluation dataset. In all cases, it was observed that the selectivity of connection weights could be increased. Correspondingly, the majority of connection weights became positive, and the collective weights were quite similar to the regression coefficients of the logistic regression analysis. Finally, for the third dataset (student evaluations), the extended method could extract more explicit input-output relations, compared with the logistic regression analysis, while improving generalisation performance. [ABSTRACT FROM AUTHOR]

A new type of information-theoretic method is proposed to improve prediction performance in supervised learning. The method has two main technical features. First, the complicated procedures used to increase information content are replaced by the direct use of hidden neuron outputs. Information is controlled by directly changing the outputs of the hidden neurons. In addition, to simultaneously increase information content and decrease errors between targets and outputs, the information acquisition and use phases are separated. In the information acquisition phase, the autoencoder tries to acquire as much information content on input patterns as possible. In the information use phase, information obtained in the acquisition phase is used to train supervised learning. The method is a simplified version of actual information maximization and directly deals with the outputs from neurons. The method was applied to the three data sets, namely, Iris, bankruptcy, and rebel participation data sets. Experimental results showed that the proposed simplified information acquisition method was effective in increasing the real information content. In addition, by using the information content, generalization performance was greatly improved. [ABSTRACT FROM AUTHOR]

We have previously proposed a new type of information-theoretic method where a neuron is evaluated by itself (self-evaluation) and by its surrounding neurons (outer-evaluation). If contradiction between different types of evaluation exists, it is reduced as much as possible. In the present paper, we try to separate self- and outer-evaluation more explicitly and introduce the importance of neurons. First, we separate self- and outer-evaluation to enhance the characteristics shared by the two types of evaluation. Second, we introduce the importance of neurons in evaluation. By using a limited number of important neurons in evaluation, we expect the main characteristics in input patterns to emerge. We applied this contradiction resolution to two types of data, namely, the Senate data and the Euro-yen exchange rates. In both data sets, experimental results confirmed that improved prediction performance was obtained. Prediction performance was better than that obtained by the conventional self-organising map (SOM) and radial basis function networks. In addition, final representations obtained by contradiction resolution were easier to interpret than those given by the conventional SOM. Experimental results confirmed that improved interpretation and visualisation were accompanied by improved prediction performance. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new information-theoretic computational method called ‘similarity interaction’ for improving visualization. Due to the fixed arrangement of neurons in the self-organizing maps, similarity between neurons is not necessarily a faithful representation of the actual similarity between neurons. To relax the fixed arrangement, we introduce a method called ‘similarity interaction’, because we integrate the information of connection weights into that of neurons. We applied our method to three problems, namely teaching assistant evaluation, automobile data, and dermatology data. In all three problems, we succeeded in demonstrating the better performance of our method through visual inspection and quantitative evaluation. Our method is the first step towards the interaction of multiple components in a neural network for finer representations of input patterns. [ABSTRACT FROM AUTHOR]