Search

The classification and prediction accuracy of neural networks can be improved when they are trained with discretized continuous attributes as additional inputs. Such input augmentation makes it easier for the network weights to form more accurate decision boundaries when the data samples of different classes in the data set are contained in distinct hyper-rectangular subregions in the original input space. In this paper, we present first how a neural network can be trained with augmented discretized inputs. The additional inputs are obtained by dividing the original interval of each continuous attribute into subintervals of equal length. The network is then pruned to remove most of the discretized inputs as well as the original continuous attributes as long as the network still achieves a minimum preset accuracy requirement. We then discuss how comprehensible classification rules can be extracted from the pruned network by analyzing the activations of the network hidden units and the weights of the network connections that remain in the pruned network. Our experiments on artificial data sets show that the rules extracted from the neural networks can perfectly replicate the class membership rules used to create the data perfectly. On real-life benchmark data sets, neural networks trained with augmented discretized inputs are shown to achieve better accuracy than neural networks trained with the original data.

A structured 3D SOM is an extension of a Self-Organizing Map from 2D to 3D in such a way that a pre-defined structure is built into the design of the 3D map. The structured 3D SOM is a 3×3×3 structure that has a distinct core cube in the center and exterior cubes around the core. The current application of the structured SOM, as a digital music archive, only uses the 8 corner cubes among the 26 exterior cubes. Given that the SOM has a built-in structure, the SOM learning algorithm is modified to include a four-phase learning and labeling phase. The first phase is meant to position the music files in their general locations within the core cube. The second phase positions the music files in their respective corner cubes according to their music genre. The second phase is therefore a semi-supervised version of the SOM algorithm which leads to the stability of the trained SOM in terms of the general distribution of the music files in the core cube. The third phase does a fine adjustment of the weight vectors in the core cube and finalizes the training of the 3D SOM. The final fourth phase is the labeling of the core cube and the association (uploading) of music files to specific nodes in the core cube. Based on the pre-defined structure of the 3D SOM, a precise measure is developed to measure the quality of the resulting trained SOM (in this case, the music archive), as well as the quality of the different categories/genres of music albums based on a novel measure of the distortion values of music files with respect to their respective music genres.

While feedforward neural networks have been widely accepted as effective tools for solving classification problems, the issue of finding the best network architecture remains unresolved, particularly so in real-world problem settings. We address this issue in the context of credit card screening, where it is important to not only find a neural network with good predictive performance but also one that facilitates a clear explanation of how it produces its predictions. We show that minimal neural networks with as few as one hidden unit provide good predictive accuracy, while having the added advantage of making it easier to generate concise and comprehensible classification rules for the user. To further reduce model size, a novel approach is suggested in which network connections from the input units to this hidden unit are removed by a very straightaway pruning procedure. In terms of predictive accuracy, both the minimized neural networks and the rule sets generated from them are shown to compare favorably with other neural network based classifiers. The rules generated from the minimized neural networks are concise and thus easier to validate in a real-life setting.

This paper describes a business intelligence application of neural networks in analyzing consumer heterogeneity in the context of eating-out behavior in Taiwan. We apply a neural network rule extraction algorithm which automatically groups the consumers into identifiable segments according to their socio-demographic information. Within each of these segments, the consumers are distinguished between those who eat-out frequently from those who do not based on their psychological traits and eat-out considerations. The data set for this study has been collected through a survey of 800 Taiwanese consumers. Demographic information such as gender, age and income were recorded. In addition, information about their psychological traits and eating-out considerations that might influence the frequency of eating-out were obtained. The results of our data analysis show that the neural network rule extraction algorithm is able to find distinct consumer segments and predict the consumers within each segment with good accuracy.

The objectives of the study reported in this paper are: (1) to evaluate the adequacy of two data mining techniques, decision tree and neural network in analysing consumer preference for a fast-food franchise and (2) to examine the sufficiency of the criteria selected in understanding this preference. We build decision tree and neural network models to fit data samples collected from 800 respondents in Taiwan to understand the factors that determine their brand preference. Classification rules are generated from these models to differentiate between consumers who prefer the brand and those who do not. The generated rules show that while both decision tree and neural network models can achieve predictive accuracy of more than 80% on the training data samples and more that 70% on the cross-validation data samples, the neural network models compare very favourably to a decision tree model in rule complexity and the numbers of relevant input attributes.

We address an important issue in knowledge discovery using neural networks that has been left out in a recent article “Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem” by Sexton et al. [R.S. Sexton, S. McMurtrey, D.J. Cleavenger, Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem, European Journal of Operational Research 168 (2006) 1009–1018]. This important issue is the generation of comprehensible rule sets from trained neural networks. In this note, we present our neural network rule extraction algorithm that is very effective in discovering knowledge embedded in a neural network. This algorithm is particularly appropriate in applications where comprehensibility as well as accuracy are required. For the same data sets used by Sexton et al. our algorithm produces accurate rule sets that are concise and comprehensible, and hence helps validate the claim that neural networks could be viable alternatives to other data mining tools for knowledge discovery.

A three-tier knowledge management approach is proposed in the context of a cross-national study of car brand and corporate image perceptions. The approach consists of knowledge acquisition, transfer and revision using neural networks. We investigate how knowledge acquired by a neural network from one car market can be exploited and applied in another market. This transferred knowledge is subsequently revised for application in the new market. Knowledge revision is achieved by re-training the neural network. Core knowledge common to both markets is retained while some localized knowledge components are introduced during network re-training. Since the knowledge acquired by a neural network can be expressed as an accurate set of simple rules, we are able to compare the knowledge extracted from one network with the knowledge extracted from another. Comparison of the originally acquired knowledge with the revised knowledge provides us with insights into the commonalities and differences in car brand and corporate perceptions across national markets.

Learning in self-organizing maps (SOM) is considered unsupervised because training patterns do not need accompanying desired output information. Prior to its use in some real-world applications, however, a trained SOM often has to be labeled. This labeling phase is usually supervised in that labeled patterns need accompanying output information. Because such labeled patterns are not always available or may not even be possible to construct, the supervised nature of the labeling phase restricts the deployment of SOM from a wide range of potential domains of application. This work proposes a methodical and automatic SOM labeling procedure that does not require a set of prelabeled patterns. Instead, nodes in the trained map are clustered and subsets of training patterns associated to each of the clustered nodes are identified. Salient dimensions per node cluster that constitute the bases for labeling each node in the map are then identified. The effectiveness of the method is demonstrated on a SOM-based international market segmentation study.

Recent advances in algorithms that extract rules from artificial neural networks make it feasible to use neural networks as a tool for acquiring knowledge hidden in the data. Findings are reported from the use of such algorithms to separate core and noncore knowledge in a cross-national study of automobile brand image perception. Respondents from five Western European countries have been asked to associate individual and corporate brand associations for a number of well-known automobile brands. Knowledge, expressed as concise and accurate rules that distinguish between the respondents' perceptions of German and Japanese brands, is extracted from trained neural networks. This paper explains how both core knowledge, which captures the perceptions shared by the respondents in all countries, and country-specific noncore knowledge can be acquired and differentiated by a proposed two-step approach to train and extract rules from a multi-neural network system. The experimental results show that, in addition to providing a better understanding of the differences and similarities in the brand image perceptions of consumers in various countries, the proposed approach also yields better predictive accuracy than a decision tree method.

Microarray technology can be employed to quantitatively measure the expression of thousands of genes in a single experiment. It has become one of the main tools for global gene expression analysis in molecular biology research in recent years. The large amount of expression data generated by this technology makes the study of certain complex biological problems possible, and machine learning methods are expected to play a crucial role in the analysis process. In this paper, we present our results from integrating the self-organizing map (SOM) and the support vector machine (SVM) for the analysis of the various functions of zebrafish genes based on their expression. The most distinctive characteristic of our zebrafish gene expression is that the number of samples of different classes is imbalanced. We discuss how SOM can be used as a data-filtering tool to improve the classification performance of the SVM on this data set.

Data collected from a survey typically consist of attributes that are mostly if not completely binary-valued or binary-encoded. We present a method for handling such data where the underlying data analysis can be cast as a classification problem. We propose a hybrid method that combines neural network and decision tree methods. The network is trained to remove irrelevant data attributes and the decision tree is applied to extract comprehensible classification rules from the trained network. The conditions of the rules are in the form of a conjunction of M-of-N constructs. An M-of-N construct is a rule condition that is satisfied if (at least, exactly, at most) M of the N binary attributes in the construct are present. The effectiveness of the method is illustrated on data collected for a study of global car market segmentation. The results show that besides achieving high predictive accuracy, the method also allows meaningful interpretation of the relationships among the data variables.

In many applications, black-box prediction is not satisfactory, and understanding the data is of critical importance. Typically, approaches useful for understanding of data involve logical rules, evaluate similarity to prototypes, or are based on visualization or graphical methods. This paper is focused on the extraction and use of logical rules for data understanding. All aspects of rule generation, optimization, and application are described, including the problem of finding good symbolic descriptors for continuous data, tradeoffs between accuracy and simplicity at the rule-extraction stage, and tradeoffs between rejection and error level at the rule optimization stage. Stability of rule-based description, calculation of probabilities from rules, and other related issues are also discussed. Major approaches to extraction of logical rules based on neural networks, decision trees, machine learning, and statistical methods are introduced. Optimization and application issues for sets of logical rules are described. Applications of such methods to benchmark and real-life problems are reported and illustrated with simple logical rules for many datasets. Challenges and new directions for research are outlined.

Artificial neural networks have been successfully applied to a variety of business application problems involving classification and regression. They are especially useful for regression problems as they do not require prior knowledge about the data distribution. In many applications, it is desirable to extract knowledge from trained neural networks so that the users can gain a better understanding of the solution. Existing research works have focused primarily on extracting symbolic rules for classification problems with few methods devised for regression problems. In order to fill this gap, we propose an approach to extract rules from neural networks that have been trained to solve regression problems. The extracted rules divide the data samples into groups. For all samples within a group, a linear function of the relevant input attributes of the data approximates the network output. The approach is illustrated with two examples on various application problems. Experimental results show that the proposed approach generates rules that are more accurate than the existing methods based on decision trees and linear regression.

Credit-risk evaluation is a very challenging and important management science problem in the domain of financial analysis. Many classification methods have been suggested in the literature to tackle this problem. Neural networks, especially, have received a lot of attention because of their universal approximation property. However, a major drawback associated with the use of neural networks for decision making is their lack of explanation capability. While they can achieve a high predictive accuracy rate, the reasoning behind how they reach their decisions is not readily available. In this paper, we present the results from analysing three real-life credit-risk data sets using neural network rule extraction techniques. Clarifying the neural network decisions by explanatory rules that capture the learned knowledge embedded in the networks can help the credit-risk manager in explaining why a particular applicant is classified as either bad or good. Furthermore, we also discuss how these rules can be visualized as a decision table in a compact and intuitive graphical format that facilitates easy consultation. It is concluded that neural network rule extraction and decision tables are powerful management tools that allow us to build advanced and userfriendly decision-support systems for credit-risk evaluation.

Neural networks with a single hidden layer are known to be universal function approximators. However, due to the complexity of the network topology and the nonlinear transfer function used in computing the hidden unit activations, the predictions of a trained network are difficult to comprehend. On the other hand, predictions from a multiple linear regression equation are easy to understand but are not accurate when the underlying relationship between the input variables and the output variable is nonlinear. We have thus developed a method for multivariate function approximation which combines neural network learning, clustering and multiple regression. This method generates a set of multiple linear regression equations using neural networks, where the number of regression equations is determined by clustering the weighted input variables. The predictions for samples of the same cluster are computed by the same regression equation. Experimental results on a number of real-world data demonstrate that this new method generates relatively few regression equations from the training data samples. Yet, drawing from the universal function approximation capacity of neural networks, the predictive accuracy is high. The prediction errors are comparable to or lower than those achieved by existing function approximation methods.

Neural networks (NNs) have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful as function approximators because they do not require prior knowledge of the input data distribution and they have been shown to be universal approximators. In many applications, it is desirable to extract knowledge that can explain how Me problems are solved by the networks. Most existing approaches have focused on extracting symbolic rules for classification. Few methods have been devised to extract rules from trained NNs for regression. This article presents an approach for extracting rules from trained NNs for regression. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Extensive experimental results on 32 benchmark data sets demonstrate the effectiveness of the proposed approach in generating accurate regression rules.

This paper describes a novel approach to estimate the size of database query results using neural networks. Using the proposed approach, three layer neural networks are constructed and trained to learn the cumulative distribution functions of attribute values in relations. With a trained network, the estimation of the query result size could be obtained instantly by simply computing the network output from the given query predicates. The basic computational model using a cumulative distribution function to compute the query result size is described. The network construction and training is discussed. Comprehensive experiments were conducted to study the effectiveness of the proposed approach. The results indicate that the approach produces estimates with accuracies that are comparable with or higher than those reported in the literature.

The accuracy of neural networks can be improved when they are trained with discretized continuous attributes as additional inputs. Such input augmentation makes it easier for the network weights to form more accurate decision boundaries when the data samples of different classes in the data set are contained in distinct hyper-rectangular subregions in the original input space. In this paper we present first how a neural network can be trained with augmented discretized inputs. The additional inputs are obtained by simply dividing the original interval of each continuous attribute into subintervals of equal length. Thermometer encoding scheme is used to represent these discretized inputs. The network is then pruned to remove most of the discretized inputs as well as the original continuous attributes as long as the network still achieves a minimum preset accuracy requirement. We then discuss how MofN rules can be extracted from the pruned network by analyzing the activations of the network's hidden units and the weights of the network connections that remain in the pruned network. For data sets that have sample classes defined by relatively complex boundaries, surprisingly simple MofN rules with very good accuracy rates are obtained.

Keywords and key-phrases that concisely represent text documents are integral to many knowledge management and text information retrieval systems, as well as digital libraries in general. Not all text documents, however, are annotated with good keywords; and the quality of these keywords is often dependent on a tedious, sometimes manual, extraction and tagging process. To automatically extract high quality keywords without the need for a semantic analysis of the document, it is shown that artificial neural networks (ANN) can be trained to only consider in-document word features such as word frequency, word distribution in document, use of word in special parts of the document, and use of word formatting features (i.e. bold-faced, italicized, large-font size). Results show that purely local features are adequate in determining whether a word in a document is a keyword or not. Classification performance yields a G mean of a least 0.83, and weighted f-measure of 0.96 for both keywords and non-keywords. Precision for keywords alone, however, is not as high. To understand the basis for classifying keywords, C4.5 is used to extract rules from the ANN. The extracted rules from C4.5, in the form of a decision tree, show the relative importance of the different document features that were extracted.

Neural networks have been widely used as tools for prediction in medicine. We expect to see even more applications of neural networks for medical diagnosis as recently developed neural network rule extraction algorithms make it possible for the decision process of a trained network to be expressed as classification rules. These rules are more comprehensible to a human user than the classification process of the networks which involves complex nonlinear mapping of the input data. This paper reports the results from two neural network rule extraction techniques, NeuroLinear and NeuroRule applied to the diagnosis of hepatobiliary disorders. The dataset consists of nine measurements collected from patients in a Japanese hospital and these measurements have continuous values. NeuroLinear generates piece-wise linear discriminant functions for this dataset. The continuous measurements have previously been discretized by domain experts. NeuroRule is applied to the discretized dataset to generate symbolic classification rules. We compare the rules generated by the two techniques and find that the rules generated by NeuroLinear from the original continuously valued dataset to be slightly more accurate and more concise than the rules generated by NeuroRule from the discretized dataset.

Records in a medical dataset may be best characterized by M-of-N concepts. For example, a patient showing at least 2 of the 4 symptoms is likely to be diagnosed as having a certain illness. In this paper, we describe how feedforward neural networks can be used to learn such concepts. We train a network where each input in the data can only have one of two possible values, -1 or 1 and apply the hyperbolic tangent function to each connection from the input layer to the hidden layer of the network before the hidden unit activations are computed. By applying this squashing function, the activation values at the hidden units are effectively computed as the hyperbolic tangent (or the sigmoid) of the weighted inputs, where the weights have magnitudes that are near one. By restricting the inputs and the weights to binary ’ values either -1 or 1, the extraction of the M-of-N concepts from networks becomes trivial. We show how this approach can be used to learn concise and accurate the M-of-N concepts for the diagnosis of hepatobiliary disorders.

Before symbolic rules are extracted from a trained neural network, the network is usually pruned so as to obtain more concise rules. Typical pruning algorithms require retraining the network which incurs additional cost. This paper presents FERNN, a fast method for extracting rules from trained neural networks without network retraining. Given a fully connected trained feedforward network with a single hidden layer, FERNN first identifies the relevant hidden units by computing their information gains. For each relevant hidden unit, its activation values is divided into two subintervals such that the information gain is maximized. FERNN finds the set of relevant network connections from the input units to this hidden unit by checking the magnitudes of their weights. The connections with large weights are identified as relevant. Finally, FERNN generates rules that distinguish the two subintervals of the hidden activation values in terms of the network inputs. Experimental results show that the size and the predictive accuracy of the tree generated are comparable to those extracted by another method which prunes and retrains the network.

Neural networks and decision tree methods are two common approaches to pattern classification. While neural networks can achieve high predictive accuracy rates, the decision boundaries they form are highly nonlinear and generally difficult to comprehend. Decision trees, on the other hand, can be readily translated into a set of rules. In this paper, we present a novel algorithm for generating oblique decision trees that capitalizes on the strength of both approaches. Oblique decision trees classify the patterns by testing on linear combinations of the input attributes. As a result, an oblique decision tree is usually much smaller than the univariate tree generated for the same domain. Our algorithm consists of two components: connectionist and symbolic. A three-layer feedforward neural network is constructed and pruned, a decision tree is then built from the hidden unit activation values of the pruned network. An oblique decision tree is obtained by expressing the activation values using the original input attributes. We test our algorithm on a wide range of problems. The oblique decision trees generated by the algorithm preserve the high accuracy of the neural networks, while keeping the explicitness of decision trees. Moreover, they outperform univariate decision trees generated by the symbolic approach and oblique decision trees built by other approaches in accuracy and tree size.

There exist several methods for transforming decision trees to neural networks. These methods typically construct the networks by directly mapping decision nodes or rules to the neural units. As a result, the networks constructed are often larger than necessary. This article describes a pruning-based method for mapping decision trees to neural networks, which can compress the network by removing unimportant and redundant units and connections. In addition, equivalent decision trees extracted from the pruned networks are simpler than those induced by well-known algorithms such as ID3 and C4.5.

Feature selection determines relevant features in the data. It is often applied in pattern classification, data mining, as well as machine learning. A special concern for feature selection nowadays is that the size of a database is normally very large, both vertically and horizontally. In addition, feature sets may grow as the data collection process continues. Effective solutions are needed to accommodate the practical demands. This paper concentrates on three issues: large number of features, large data size, and expanding feature set. For the first issue, we suggest a probabilistic algorithm to select features. For the second issue, we present a scalable probabilistic algorithm that expedites feature selection further and can scale up without sacrificing the quality of selected features. For the third issue, we propose an incremental algorithm that adapts to the newly extended feature set and captures `concept drifts' by removing features from previously selected and newly added ones. We expect that research on scalable feature selection will be extended to distributed and parallel computing and have impact on applications of data mining and machine learning.

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indi...

Feature selection is a problem of finding relevant features. When the number of features of a dataset is large and its number of patterns is huge, an effective method of feature selection can help in dimensionality reduction. An incremental probabilistic algorithm is designed and implemented as an alternative to the exhaustive and heuristic approaches. Theoretical analysis is given to support the idea of the probabilistic algorithm in finding an optimal or near-optimal subset of features. Experimental results suggest that (1) the probabilistic algorithm is effective in obtaining optimal/suboptimal feature subsets; (2) its incremental version expedites feature selection further when the number of patterns is large and can scale up without sacrificing the quality of selected features.

Feature selection is an integral part of most learning algorithms. Due to the existence of irrelevant and redundant attributes, by selecting only the relevant attributes of the data, higher predictive accuracy can be expected from a machine learning method. In this paper, we propose the use of a three-layer feedforward neural network to select those input attributes that are most useful for discriminating classes in a given set of input patterns. A network pruning algorithm is the foundation of the proposed algorithm. By adding a penalty term to the error function of the network, redundant network connections can be distinguished from those relevant ones by their small weights when the network training process has been completed. A simple criterion to remove an attribute based on the accuracy rate of the network is developed. The network is retrained after removal of an attribute, and the selection process is repeated until no attribute meets the criterion for removal. Our experimental results suggest that the proposed method works very well on a wide variety of classification problems.

This article proposes the use of a penalty function for pruning feedforward neural network by weight elimination. The penalty function proposed consists of two terms. The first term is to discourage the use of unnecessary connections, and the second term is to prevent the weights of the connections from taking excessively large values. Simple criteria for eliminating weights from the network are also given. The effectiveness of this penalty function is tested on three well-known problems: the contiguity problem, the parity problems, and the monks problems. The resulting pruned networks obtained for many of these problems have fewer connections than previously reported in the literature.

An algorithm for extracting rules from a standard three-layer feedforward neural network is proposed. The trained network is first pruned not only to remove redundant connections in the network but, more important, to detect the relevant inputs. The algorithm generates rules from the pruned network by considering only a small number of activation values at the hidden units. If the number of inputs connected to a hidden unit is sufficiently small, then rules that describe how each of its activation values is obtained can be readily generated. Otherwise the hidden unit will be split and treated as output units, with each output unit corresponding to an activation value. A hidden layer is inserted and a new subnetwork is formed, trained, and pruned. This process is repeated until every hidden unit in the network has a relatively small number of input units connected to it. Examples on how the proposed algorithm works are shown using real-world data arising from molecular biology and signal processing. Our results show that for these complex problems, the algorithm can extract reasonably compact rule sets that have high predictive accuracy rates.

Discretization can turn numeric attributes into discrete ones. Feature selection can eliminate some irrelevant and/or redundant attributes. Chi2 is a simple and general algorithm that uses the /spl chi//sup 2/ statistic to discretize numeric attributes repeatedly until some inconsistencies are found in the data. It achieves feature selection via discretization. It can handle mixed attributes, work with multiclass data, and remove irrelevant and redundant attributes.

There exist redundant, irrelevant and noisy data. Using proper data to train a network can speed up training, simplify the learned structure, and improve its performance. A two-phase training algorithm is proposed. In the first phase, the number of input units of the network is determined by using an information base method. Only those attributes that meet certain criteria for inclusion will be considered as the input to the network. In the second phase, the number of hidden units of the network is selected automatically based on the performance of the network on the training data. One hidden unit is added at a time only if it is necessary. The experimental results show that this new algorithm can achieve a faster learning time, a simpler network and an improved performance.

Neural networks often surpass decision trees in predicting pattern classifications, but their predictions cannot be explained. This algorithm's symbolic representations make each prediction explicit and understandable. Our approach to understanding a neural network uses symbolic rules to represent the network decision process. The algorithm, NeuroRule, extracts these rules from a neural network. The network can be interpreted by the rules which, in general, preserve network accuracy and explain the prediction process. We based NeuroRule on a standard three layer feed forward network. NeuroRule consists of four phases. First, it builds a weight decay backpropagation network so that weights reflect the importance of the network's connections. Second, it prunes the network to remove irrelevant connections and units while maintaining the network's predictive accuracy. Third, it discretizes the hidden unit activation values by clustering. Finally, it extracts rules from the network with discretized hidden unit activation values.

The existence of numeric data and large numbers of records in a database present a challenging task in terms of explicit concepts extraction from the raw data. The paper introduces a method that reduces data vertically and horizontally, keeps the discriminating power of the original data, and paves the way for extracting concepts. The method is based on discretization (vertical reduction) and feature selection (horizontal reduction). The experimental results show that (a) the data can be effectively reduced by the proposed method; (b) the predictive accuracy of a classifier (C4.5) can be improved after data and dimensionality reduction; and (c) the classification rules learned are simpler.

Classification is one of the data mining problems receiving great attention recently in the database community. The paper presents an approach to discover symbolic classification rules using neural networks. Neural networks have not been thought suited for data mining because how the classifications were made is not explicitly stated as symbolic rules that are suitable for verification or interpretation by humans. With the proposed approach, concise symbolic rules with high accuracy can be extracted from a neural network. The network is first trained to achieve the required accuracy rate. Redundant connections of the network are then removed by a network pruning algorithm. The activation values of the hidden units in the network are analyzed, and classification rules are generated using the result of this analysis. The effectiveness of the proposed approach is clearly demonstrated by the experimental results on a set of standard data mining test problems.

Neural networks are widely used for classification as they achieve good predictive accuracy. When the class labels are determined by complex interactions of the input variables, neural networks can be expected to provide better predictions than methods that test on the values of one variable at a time such as univariate decision tree classifiers. On the other hand, when no or relatively simple interaction between variables determines the class membership, the neural network may over fit the data and the input-to-output relationship in the data is represented by a function that is more complex than it should be. In this paper, we propose adding discretized values of the continuous variables in the data as input when training the neural networks. Finding out whether the discretized values or the original continuous values of the variables are useful is achieved by pruning. By having only the relevant inputs left in the pruned networks, we are able to extract classification rules from these networks that are accurate, concise and interpretable.

A new method for constructing a feedforward neural network is proposed. The method starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that c...

An efficient implementation of a quasi-Newton algorithm for training feed-forward neural network on a Cray Y-MP is presented. The most time-consuming step of a neural network training using the quasi-Newton algorithm is the computation of the error function and its gradient. Parallelization embedded in these computations can be exploited through vectorization in a Cray Y-MP supercomputer. We show how they can be carried out such that the overall performance of the neural network training process can be enhanced substantially.

This paper describes an algorithm for constructing a single hidden layer feedforward neural network. A distinguishing feature of this algorithm is that it uses the quasi-Newton method to minimize the sequence of error functions associated with the growing network. Experimental results indicate that the algorithm is very efficient and robust. The algorithm was tested on two test problems. The first was the n-bit parity problem and the second was the breast cancer diagnosis problem from the University of Wisconsin Hospitals. For the n-bit parity problem, the algorithm was able to construct neural network having less than n hidden units that solved the problem for n=4,/spl middotspl middotspl middot/,7. For the cancer diagnosis problem, the neural networks constructed by the algorithm had small number of hidden units and high accuracy rates on both the training data and the testing data. >

Keyword extraction is vital for Knowledge Management System, Information Retrieval System, and Digital Libraries as well as for general browsing of the web. Keywords are often the basis of document processing methods such as clustering and retrieval since processing all the words in the document can be slow. Common models for automating the process of keyword extraction are usually done by using several statistics-based methods such as Bayesian, K-Nearest Neighbor, and Expectation-Maximization. These models are limited by word-related features that can be used since adding more features will make the models more complex and difficult to comprehend. In this research, a Neural Network, specifically a backpropagation network, will be used in generalizing the relationship of the title and the content of articles in the archive by following word features other than TF-IDF, such as position of word in the sentence, paragraph, or in the entire document, and formats such as heading, and other attributes defined beforehand. In order to explain how the backpropagation network works, a rule extraction method will be used to extract symbolic data from the resulting backpropagation network. The rules extracted can then be transformed into decision trees performing almost as accurate as the network plus the benefit of being in an easily comprehensible format.

In this chapter we describe how comprehensible rules can be extracted from artificial neural networks (ANN) and support vector machines (SVM). ANN and SVM are two very popular techniques for pattern classification. In the business intelligence application domain of credit scoring, they have been shown to be effective tools for distinguishing between good credit risks and bad credit risks. The accuracy obtained by these two techniques is often higher than that from decision tree methods. Unlike decision tree methods, however, the classifications made by ANN and SVM are difficult to understand by the end-users as outputs from ANN and SVM are computed as nonlinear mapping of the input data attributes. We describe two rule extraction methods that we have developed to overcome this difficulty. These rule extraction methods enable the users to obtain comprehensible propositional rules from ANN and SVM. Such rules can be easily verified by the domain experts and would lead to a better understanding about the data in hand.

In this paper, we present an approach for sample selection using an ensemble of neural networks for credit scoring. The ensemble determines samples that can be considered outliers by checking the classification accuracy of the neural networks on the original training data samples. Those samples that are consistently misclassified by the neural networks in the ensemble are removed from the training dataset. The remaining data samples are then used to train and prune another neural network for rule extraction. Our experimental results on publicly available benchmark credit scoring datasets show that by eliminating the outliers, we obtain neural networks with higher predictive accuracy and simpler in structure compared to the networks that are trained with the original dataset. A rule extraction algorithm is applied to generate comprehensible rules from the neural networks. The extracted rules are more concise than the rules generated from networks that have been trained using the original datasets.

This paper presents the results from a neural network rule extraction algorithm applied to the LED display recognition problem. We show that pruned neural networks with small number of hidden nodes and connections are able to recognize all the 10 digits from 0 to 9. Earlier work by other researchers demonstrated how symbolic fuzzy rules can be extracted from trained neural networks to solve this problem. Our rules in contrast are crisp rules, and they are obtained from smaller networks. As a result, simpler and easier to understand rules are obtained. These rules give us an insight of how neural networks differentiate one digit from the rest in LED display recognition problem.

Neural networks are often selected as tool for software effort prediction because of their capability to approximate any continuous function with arbitrary accuracy. A major drawback of neural networks is the complex mapping between inputs and output, which is not easily understood by a user. This paper describes a rule extraction technique that derives a set of comprehensible IF-THEN rules from a trained neural network applied to the domain of software effort prediction. The suitability of this technique is tested on the ISBSG R11 data set by a comparison with linear regression, radial basis function networks, and CART. It is found that the most accurate results are obtained by CART, though the large number of rules limits comprehensibility. Considering comprehensible models only, the concise set of extracted rules outperform the pruned CART tree, making neural network rule extraction the most suitable technique for software effort prediction when comprehensibility is important. © 2010 IEEE. ispartof: pages:45-52 ispartof: Proceedings 22nd International Conference on Tools with Artificial Intelligence ICTAI 2010 vol:2 pages:45-52 ispartof: International Conference on Tools with Artificial Intelligence (IEEE-ICTAI) location:Arras (France) date:27 Oct - 29 Oct 2010 status: published

Even in an increasingly globalized market, the knowledge about individual local markets could still be invaluable. In this cross-national study of brand image perception of cars, survey data from buyers in the top 20 automobile markets have been collected, where each respondent has been asked to associate a car brand with individual brand images and corporate brand images. These data can be used as tool for decision making at the enterprise level. We describe an algorithm for constructing auto-associative neural networks which can be used as a tool for knowledge discovery from this data. It automatically determines the network topology by adding hidden units as they are needed to improve accuracy and by removing irrelevant input attributes. Two market research applications are presented, the first is for classification, and the second is for measuring similarities in the perceptions of the respondents from the different markets. In the first application, the constructed networks are shown to be more accurate than a decision tree. In the second application, the constructed networks are able to reproduce the training data very accurately and could be used to identify country-level (i.e. local) markets which share similar perceptions about the car brands being studied.

Various benchmarking studies have shown that artificial neural networks and support vector machines often have superior performance when compared to more traditional machine learning techniques. The main resistance against these newer techniques is based on their lack of interpretability: it is difficult for the human analyst to understand the reasoning behind these models' decisions. Various rule extraction (RE) techniques have been proposed to overcome this opacity restriction. These techniques are able to represent the behavior of the complex model with a set of easily understandable rules. However, most of the existing RE techniques can only be applied under limited circumstances, e.g., they assume that all inputs are categorical or can only be applied if the black-box model is a neural network. In this paper, we present Minerva, which is a new algorithm for RE. The main advantage of Minerva is its ability to extract a set of rules from any type of black-box model. Experiments show that the extracted models perform well in comparison with various other rule and decision tree learners.

In this paper, we present a recursive algorithm for extracting classification rules from feedforward neural networks (NNs) that have been trained on data sets having both discrete and continuous attributes. The novelty of this algorithm lies in the conditions of the extracted rules: the rule conditions involving discrete attributes are disjoint from those involving continuous attributes. The algorithm starts by first generating rules with discrete attributes only to explain the classification process of the NN. If the accuracy of a rule with only discrete attributes is not satisfactory, the algorithm refines this rule by recursively generating more rules with discrete attributes not already present in the rule condition, or by generating a hyperplane involving only the continuous attributes. We show that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods.

An effective algorithm for extracting M-of-N rules from trained feedforward neural networks is proposed. Two components of the algorithm distinguish our method from previously proposed algorithms which extract symbolic rules from neural networks. First, we train a network where each input of the data can only have one of the two possible values, -1 or one. Second, we apply the hyperbolic tangent function to each connection from the input layer to the hidden layer of the network. By applying this squashing function, the activation values at the hidden units are effectively computed as the hyperbolic tangent (or the sigmoid) of the weighted inputs, where the weights have magnitudes that are equal one. By restricting the inputs and the weights to binary values either -1 or one, the extraction of M-of-N rules from the networks becomes trivial. We demonstrate the effectiveness of the proposed algorithm on several widely tested datasets. For datasets consisting of thousands of patterns with many attributes, the rules extracted by the algorithm are surprisingly simple and accurate.

Innovative storage technology and the rising popularity of the Internet have generated an ever-growing amount of data. In this vast amount of data much valuable knowledge is available, yet it is hidden. The Support Vector Machine (SVM) is a state-of-the-art classification technique that generally provides accurate models, as it is able to capture non-linearities in the data. However, this strength is also its main weakness, as the generated non-linear models are typically regarded as incomprehensible black-box models. By extracting rules that mimic the black box as closely as possible, we can provide some insight into the logics of the SVM model. This explanation capability is of crucial importance in any domain where the model needs to be validated before being implemented, such as in credit scoring (loan default prediction) and medical diagnosis. If the SVM is regarded as the current state-of-the-art, SVM rule extraction can be the state-of-the-art of the (near) future. This chapter provides an overview of recently proposed SVM rule extraction techniques, complemented with the pedagogical Artificial Neural Network (ANN) rule extraction techniques which are also suitable for SVMs. Issues related to this topic are the different rule outputs and corresponding rule expressiveness; the focus on high dimensional data as SVM models typically perform well on such data; and the requirement that the extracted rules are in line with existing domain knowledge. These issues are explained and further illustrated with a credit scoring case, where we extract a Trepan tree and a RIPPER rule set from the generated SVM model. The benefit of decision tables in a rule extraction context is also demonstrated. Finally, some interesting alternatives for SVM rule extraction are listed.