Search

Convolutional neural networks (CNNs) have achieved superior accuracy in many visual-related tasks. However, the inference process through a CNN's intermediate layers is opaque, making it difficult to interpret such networks or develop trust in their operation. In this paper, we introduce SIGN method for modeling the network's hidden layer activity using probabilistic models. The activity patterns in layers of interest are modeled as Gaussian mixture models, and transition probabilities between clusters in consecutive modeled layers are estimated to identify paths of inference. For fully connected networks, the entire layer activity is clustered, and the resulting model is a hidden Markov model. For convolutional layers, spatial columns of activity are clustered, and a maximum likelihood model is developed for mining an explanatory inference graph. The graph describes the hierarchy of activity clusters most relevant for network prediction. We show that such inference graphs are useful for understanding the general inference process of a class, as well as explaining the (correct or incorrect) decisions the network makes about specific images. In addition, SIGN provide interesting observations regarding hidden layer activity in general, including the concentration of memorization in a single middle layer in fully connected networks, and a highly local nature of column activities in the top CNN layers.

During the process of rehabilitation after stroke, it is important that patients know how well they perform their exercise, so they can improve their performance in future repetitions. Standard clinical rating conducted by human observation is the prevailing way today to monitor motor recovery of the patient. Therefore, patients cannot know whether they are performing a movement properly while exercising by themselves. Adhering to the exercise regime makes the rehabilitation process more effective and efficient, and thus a system that can give the patients feedback on their performance is of great value. Here, we built a machine-learning-based automated model that gives patients accurate information on the compensatory (undesirable) movements that they make. To construct the model, we recorded movements from 30 stroke patients, who each performed 18 movements, used to identify the presence of six types of compensatory movements in stroke patients’ movement trajectories. We used the random-forest algorithm for training this multi-label classification model. We achieved 85 percent average precision across the six movement compensations. This is the first study to automatically identify movement compensations based on stroke patients’ data. This model can be adapted for use in in-clinic and at-home exercise programs for patients after stroke.

In recent years, technological advances, specifically new sensing and communication technologies, have brought new opportunities for a less expensive, dynamic, and more accurate mapping of social land use in cities. However, most research has featured complex methodologies that integrate several data resources or require much prior knowledge about the examined city. We offer a methodology that requires little prior knowledge and mainly relies on call detail records, which is an inexpensive available data resource of mobile phone signals. We introduce the Semi-supervised Self-labeled K-nearest neighbor (SSK) algorithm that combines distance-weighted k-nearest neighbors (DKNN) with a self-labeled iterative technique designed for training classifiers with only a small number of labeled samples. In each iteration, the samples (small land units) that we are most confident of their classification by DKNN are added to the training set of the next iteration. We perform neighbor smoothing to the land-use classification by considering feature-space neighbors as in the regular KNN but also geographical space neighbors, and thereby leverage the tendency of approximate land areas to share similar social land use. Based only on a few labeled examples, the SSK algorithm achieves a high accuracy rate, between 74% without neighbor smoothing, and 80% with it.

Responsiveness to levodopa varies greatly among patients with Parkinson's disease (PD). The factors that affect it are ill defined. The aim of the study was to identify factors predictive of long-term response to levodopa. The medical records of 296 patients with PD (mean age of onset, 62.2 ± 9.7 years) were screened for demographics, previous treatments, and clinical phenotypes. All patients were assessed with the Unified PD Rating Scale (UPDRS)-III before and 3 months after levodopa initiation. Regression and machine-learning analyses were used to determine factors that are associated with levodopa responsiveness and might identify patients who will benefit from treatment. The UPDRS-III score improved by ≥ 30% (good response) in 128 patients (43%). On regression analysis, female gender, young age at onset, and early use of dopamine agonists predicted a good response. Time to initiation of levodopa treatment had no effect on responsiveness except in patients older than 72 years, who were less responsive. Machine-learning analysis validated these factors and added several others: symptoms of rigidity and bradykinesia, disease onset in the legs and on the left side, and fewer white vascular ischemic changes, comorbidities, and pre-non-motor symptoms. The main determinants of variations in levodopa responsiveness are gender, age, and clinical phenotype. Early use of dopamine agonists does not hamper levodopa responsiveness. In addition to validating the regression analysis results, machine-learning methods helped to determine the specific clinical phenotype of patients who may benefit from levodopa in terms of comorbidities and pre-motor and non-motor symptoms.

Although recent studies have shown that a Bayesian network classifier (BNC) that maximizes the classification accuracy (i.e., minimizes the 0/1 loss function) is a powerful tool in both knowledge representation and classification, this classifier: (1) focuses on the majority class and, therefore, misclassifies minority classes; (2) is usually uninformative about the distribution of misclassifications; and (3) is insensitive to error severity (making no distinction between misclassification types). In this study, we propose to learn the structure of a BNC using an information measure (IM) that jointly maximizes the classification accuracy and information, motivate this measure theoretically, and evaluate it compared with six common measures using various datasets. Using synthesized confusion matrices, twenty-three artificial datasets, seventeen UCI datasets, and different performance measures, we show that an IM-based BNC is superior to BNCs learned using the other measures—especially for ordinal classification (for which accounting for the error severity is important) and/or imbalanced problems (which are most real-life classification problems)—and that it does not fall behind state-of-the-art classifiers with respect to accuracy and amount of information provided. To further demonstrate its ability, we tested the IM-based BNC in predicting the severity of motorcycle accidents of young drivers and the disease state of ALS patients—two class-imbalance ordinal classification problems—and show that the IM-based BNC is accurate also for the minority classes (fatal accidents and severe patients) and not only for the majority class (mild accidents and mild patients) as are other classifiers, providing more informative and practical classification results. Based on the many experiments we report on here, we expect these advantages to exist for other problems in which both accuracy and information should be maximized, the data is imbalanced, and/or the problem is ordinal, whether the classifier is a BNC or not. Our code, datasets, and results are publicly available http://www.ee.bgu.ac.il/~boaz/software .

Understanding land use in urban areas, from the perspective of social function, is beneficial for a variety of fields, including urban and highway planning, advertising, and business. However, big cities with complex social dynamics and rapid development complicate the task of understanding these social functions. In this paper, we analyze and interpret human social function in urban areas as reflected in cellular communication usage patterns. We base our analysis on digital traces left by mobile phone users, and from this raw data, we derive a varied collection of features that illuminate the social behavior of each land use. We divide space and time into basic spatiotemporal units and classify them according to their land use. We categorize land uses with a leveled hierarchy of semantic categories that include different levels of detail resolution. We apply the above methodology to a dataset consisting of 62 days of cellular data recorded in nine cities in the Tel Aviv district. The methodology proved beneficial with an accuracy rate ranging from 84 to 91%, dependent on land use label resolution. In addition, analyzing the results sheds light on some of the limitations of relying solely on cellular communication as a data resource. We discuss some of these problems and offer applicable solutions.

Amyotrophic lateral sclerosis (ALS) is a devastating and incurable disease affecting motor neurons, leading to progressive paralysis and death on average within three to five years from onset. The disease is characterized by highly variable patterns and rates of progression, which pose challenges to developing reliable and accurate ALS disease state prediction models to be used on a daily basis in clinics with little data. To meet these challenges, we suggest domain adaptation from a large, but unfortunately biased, clinical trials database to that of a tertiary care ALS clinic. To evaluate the reliability and accuracy of the suggested paradigm, we examine a naive approach by which training is based only on the clinical trials data compared with a domain adaptation approach of an initial training using this same data followed by fine-tuning training using the clinic data. We also allow summarization of the clinical longitudinal data to evaluate non-temporal models, e.g., random forest (RF), XGBoost (XGB), and multilayer perceptron (MLP), partially exploiting the dynamic information hidden in patient clinical records, in comparison to the long short-term memory (LSTM) recurrent neural network, fully exploiting the temporal information in the data. First, we notice the XGB outperformance in terms of the ALS disease state prediction error to the RF and MLP, but surprisingly also to the LSTM regardless of prediction time (up to 24 months ahead). We contribute the inferiority of the highly parametrized neural network to the impact of the curse of dimensionality. Second, we show that this error does not significantly increase when the model is trained using only the clinical trials data, especially for LSTM in long prediction times. Finally, we demonstrate that fine-tuning of the clinical trials-based pre-trained model using the clinic data improves the LSTM and MLP performance compared to using solely the clinical trials or clinic data.

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disease, lasting from the day of onset until death. Factors such as the progression rate and pattern of the disease vary greatly among patients, making it difficult to achieve accurate predictions about ALS. To accurately predict ALS disease state and deterioration, we propose a novel approach that combines: a) sequence clustering based on dynamic time warping for separation among patients with diverse ALS deterioration patterns, b) sequential pattern mining for discovery of deterioration changes that patients of the same type may have in common, and c) deterioration-based patient next-state prediction. Using a clinical dataset, we demonstrate the advantage of the proposed approach in terms of classification accuracy and deterioration detection compared to other classification methods and temporal models such as long short-term memory.

Human mobility patterns reflect many aspects of life, from the global spread of infectious diseases to urban planning and daily commute patterns. In recent years, the prevalence of positioning methods and technologies, such as the global positioning system, cellular radio tower geo-positioning, and WiFi positioning systems, has driven efforts to collect human mobility data and to mine patterns of interest within these data in order to promote the development of location-based services and applications. The efforts to mine significant patterns within large-scale, high-dimensional mobility data have solicited use of advanced analysis techniques, usually based on machine learning methods, and therefore, in this paper, we survey and assess different approaches and models that analyze and learn human mobility patterns using mainly machine learning methods. We categorize these approaches and models in a taxonomy based on their positioning characteristics, the scale of analysis, the properties of the modeling approach, and the class of applications they can serve. We find that these applications can be categorized into three classes: user modeling, place modeling, and trajectory modeling, each class with its characteristics. Finally, we analyze the short-term trends and future challenges of human mobility analysis.

Studies reveal that the false alarm rate (FAR) demonstrated by intensive care unit (ICU) vital signs monitors ranges from 0.72 to 0.99. We applied machine learning (ML) to ICU multi-sensor information to imitate a medical specialist in diagnosing patient condition. We hypothesized that applying this data-driven approach to medical monitors will help reduce the FAR even when data from sensors are missing. An expert-based rules algorithm identified and tagged in our dataset seven clinical alarm scenarios. We compared a random forest (RF) ML model trained using the tagged data, where parameters (e.g., heart rate or blood pressure) were (deliberately) removed, in detecting ICU signals with the full expert-based rules (FER), our ground truth, and partial expert-based rules (PER), missing these parameters. When all alarm scenarios were examined, RF and FER were almost identical. However, in the absence of one to three parameters, RF maintained its values of the Youden index (0.94-0.97) and positive predictive value (PPV) (0.98-0.99), whereas PER lost its value (0.54-0.8 and 0.76-0.88, respectively). While the FAR for PER with missing parameters was 0.17-0.39, it was only 0.01-0.02 for RF. When scenarios were examined separately, RF showed clear superiority in almost all combinations of scenarios and numbers of missing parameters. When sensor data are missing, specialist performance worsens with the number of missing parameters, whereas the RF model attains high accuracy and low FAR due to its ability to fuse information from available sensors, compensating for missing parameters.

While young drivers (YDs) constitute ∼10% of the driver population, their fatality rate in motorcycle accidents is up to three times higher. Thus, we are interested in predicting fatal motorcycle accidents (FMAs), and in identifying their key factors and possible causes. Accurate prediction of YD FMAs from data by risk minimization using the 0/1 loss function (i.e., the ordinary classification accuracy) cannot be guaranteed because these accidents are only ∼1% of all YD motorcycle accidents, and classifiers tend to focus on the majority class of minor accidents at the expense of the minority class of fatal ones. Also, classifiers are usually uninformative (providing no information about the distribution of misclassifications), insensitive to error severity (making no distinction between misclassification of fatal accidents as severe or minor), and limited in identifying key factors. We propose to use an information measure (IM) that jointly maximizes accuracy and information and is sensitive to the error distribution and severity. Using a database of ∼3600 motorcycle accidents, a Bayesian network classifier optimized by IM predicted FMAs better than classifiers maximizing accuracy or other predictive or information measures, and identified fatal accident key factors and causal relations.

We provide a novel algorithm for the discovery of mobility patterns and prediction of users’ destination locations, both in terms of geographic coordinates and semantic meaning. We did not use any semantic data voluntarily provided by a user, and there was no sharing of data among the users. An advantage of our algorithm is that it allows a trade-off between prediction accuracy and information. Experimental validation was conducted on a GPS dataset collected in the Microsoft Research Asia GeoLife project by 168 users in a period of over five years.

We develop methodologies and apply machine-learning algorithms to a database of ALS patients to expose and model underlying mechanisms and relations in the disease. We view the disease state as an ordinal variable (with values between 4 for normal function and 0 for complete loss of function), and show that ordinal classification applied to the data has an advantage over classification that does not utilize the ordinal nature of the domain. To identify important physiological and lab test variables in relation to patient functionality, we rank variables with a decision tree that predicts future disease state using current and past variable instantiations. In addition, we cluster data of patient functionalities in performing daily tasks into higher level groupings of body segments and show how certain variables relate more concretely to certain groupings than to others, thus reducing the dimensionality of the disease state representation in a natural manner that was found to be medically interpretable. Finally, we learn Bayesian networks to detect predictors within the Markov blanket of the disease-state variable and to expose relations among the predictors and with the disease state, as well as to identify value combinations of the predictors that distinguish severe and mild patients.

Objective: Amyotrophic lateral sclerosis (ALS) disease state prediction usually assumes linear progression and uses a classifier evaluated by its accuracy. Since disease progression is not linear, and the accuracy measurement cannot tell large from small prediction errors, we dispense with the linearity assumption and apply ordinal classification that accounts for error severity. In addition, we identify the most influential variables in predicting and explaining the disease. Furthermore, in contrast to conventional modeling of the patient&rsquo, s total functionality, we also model separate patient functionalities (e.g., in walking or speaking). Methods: Using data from 3772 patients from the Pooled Resource Open-Access ALS Clinical Trials (PRO-ACT) database, we introduce and train ordinal classifiers to predict patients&rsquo, disease state in their last clinic visit, while accounting differently for different error severities. We use feature-selection methods and the classifiers themselves to determine the most influential variables in predicting the disease from demographic, clinical, and laboratory data collected in either the first, last, or both clinic visits, and the Bayesian network classifier to identify interrelations among these variables and their relations with the disease state. We apply these methods to model each of the patient functionalities. Results: We show the error distribution in ALS state prediction and demonstrate that ordinal classifiers outperform classifiers that do not account for error severity. We identify clinical and lab test variables influential to prediction of different ALS functionalities and their interrelations, and specific value combinations of these variables that occur more frequently in patients with severe deterioration than in patients with mild deterioration and vice versa. Conclusions: Ordinal classification of ALS state is superior to conventional classification. Identification of influential ALS variables and their interrelations help explain disease mechanism. Modeling of patient functionalities separately allows relation of variables and their connections to different aspects of the disease as may be expressed in different body segments.

Bayesian networks (BNs) provide a powerful graphical model for encoding the probabilistic relationships among a set of variables, and hence can naturally be used for classification. However, Bayesian network classifiers (BNCs) learned in the common way using likelihood scores usually tend to achieve only mediocre classification accuracy because these scores are less specific to classification, but rather suit a general inference problem. We propose risk minimization by cross validation (RMCV) using the 0/1 loss function, which is a classification-oriented score for unrestricted BNCs. RMCV is an extension of classification-oriented scores commonly used in learning restricted BNCs and non-BN classifiers. Using small real and synthetic problems, allowing for learning all possible graphs, we empirically demonstrate RMCV superiority to marginal and class-conditional likelihood-based scores with respect to classification accuracy. Experiments using twenty-two real-world datasets show that BNCs learned using an RMCV-based algorithm significantly outperform the naive Bayesian classifier (NBC), tree augmented NBC (TAN), and other BNCs learned using marginal or conditional likelihood scores and are on par with non-BN state of the art classifiers, such as support vector machine, neural network, and classification tree. These experiments also show that an optimized version of RMCV is faster than all unrestricted BNCs and comparable with the neural network with respect to run-time. The main conclusion from our experiments is that unrestricted BNCs, when learned properly, can be a good alternative to restricted BNCs and traditional machine-learning classifiers with respect to both accuracy and efficiency.

Within the complex and competitive semiconductor manufacturing industry, lot cycle time (CT) remains one of the key performance indicators. Its reduction is of strategic importance as it contributes to cost decreasing, time-to-market shortening, faster fault detection, achieving throughput targets, and improving production-resource scheduling. To reduce CT, we suggest and investigate a data-driven approach that identifies key factors and predicts their impact on CT. In our novel approach, we first identify the most influential factors using conditional mutual information maximization, and then apply the selective naive Bayesian classifier (SNBC) for further selection of a minimal, most discriminative key-factor set for CT prediction. Applied to a data set representing a simulated fab, our SNBC-based approach improves the accuracy of CT prediction in nearly 40% while narrowing the list of factors from 182 to 20. It shows comparable accuracy to those of other machine learning and statistical models, such as a decision tree, a neural network, and multinomial logistic regression. Compared to them, our approach also demonstrates simplicity and interpretability, as well as speedy and efficient model training. This approach could be implemented relatively easily in the fab promoting new insights to the process of wafer fabrication.

The identification of catalytic residues is an essential step in functional characterization of enzymes. We present a purely structural approach to this problem, which is motivated by the difficulty of evolution-based methods to annotate structural genomics targets that have few or no homologs in the databases. Our approach combines a state-of-the-art support vector machine (SVM) classifier with novel structural features that augment structural clues by spatial averaging and Z scoring. Special attention is paid to the class imbalance problem that stems from the overwhelming number of non-catalytic residues in enzymes compared to catalytic residues. This problem is tackled by: (1) optimizing the classifier to maximize a performance criterion that considers both Type I and Type II errors in the classification of catalytic and non-catalytic residues; (2) under-sampling non-catalytic residues before SVM training; and (3) during SVM training, penalizing errors in learning catalytic residues more than errors in learning non-catalytic residues. Tested on four enzyme datasets, one specifically designed by us to mimic the structural genomics scenario and three previously evaluated datasets, our structure-based classifier is never inferior to similar structure-based classifiers and comparable to classifiers that use both structural and evolutionary features. In addition to the evaluation of the performance of catalytic residue identification, we also present detailed case studies on three proteins. This analysis suggests that many false positive predictions may correspond to binding sites and other functional residues. A web server that implements the method, our own-designed database, and the source code of the programs are publicly available at http://www.cs.bgu.ac.il/∼meshi/functionPrediction. Proteins 2011; © 2011 Wiley-Liss, Inc.

We experimentally study the K2 algorithm in learning a Bayesian network (BN) classifier for image detection of cytogenetic abnormalities. Starting from an initial BN structure, the K2 algorithm searches the BN structure space and selects the structure maximizing the K2 metric. To improve the accuracy of the K2-based BN classifier, we investigate the K2 algorithm initial ordering, search procedure, and metric. We find that BN structures learned using random initial orderings, orderings based on expert knowledge, or a scatter criterion are comparable and lead to similar classification accuracies. Replacing the K2 search with hill-climbing search improves the accuracy as does the inclusion of hidden nodes in the BN structure. Also, we demonstrate that though the maximization of the K2 metric solicits structures providing improved inference, these structures contribute to only limited classification accuracy.

We suggest a data-driven methodology to identify key factors of the cycle time (CT) in a semiconductor manufacturing plant and to predict its value. We first extract a data set from a simulated fab and describe each operation in the set using 182 features (factors). Then, we apply conditional mutual information maximization for feature selection and the selective naive Bayesian classifier for further selection and CT prediction. Prediction accuracy of 72.6% is achieved by employing no more than 20 features. Similar results are obtained by neural networks and the C5.0 decision tree.

In this paper, we modify the fuzzy ARTMAP (FA) neural network (NN) using the Bayesian framework in order to improve its classification accuracy while simultaneously reduce its category proliferation. The proposed algorithm, called Bayesian ARTMAP (BA), preserves the FA advantages and also enhances its performance by the following: (1) representing a category using a multidimensional Gaussian distribution, (2) allowing a category to grow or shrink, (3) limiting a category hypervolume, (4) using Bayes' decision theory for learning and inference, and (5) employing the probabilistic association between every category and a class in order to predict the class. In addition, the BA estimates the class posterior probability and thereby enables the introduction of loss and classification according to the minimum expected loss. Based on these characteristics and using synthetic and 20 real-world databases, we show that the BA outperformes the FA, either trained for one epoch or until completion, with respect to classification accuracy, sensitivity to statistical overlapping, learning curves, expected loss, and category proliferation.

In many applications, the detection of a visually obscured magnetic target is followed by the characterization of the target, i.e. localization and magnetic moment estimation. Effective target characterization may reduce the detection false alarm rate as well as direct the searcher toward the target. We address the characterization of a static magnetic target by a three-axis fluxgate magnetometer installed on a stabilized mobile platform. The magnetometer readings are contaminated by magnetic noise, which results in a low signal-to-noise ratio. We formulate the problem as an over-determined nonlinear equation set using a magnetic dipole model for the target and use simulated annealing (SA) in order to rapidly find a good approximation to the global optimum of this equation set. Computer simulations demonstrate high accuracy of the SA method in localizing the target and estimating its magnetic moment in the presence of high-level noise. The high accuracy of the SA method is also exemplified in tests employing real-world magnetic signals. In addition to its high accuracy, the SA method is very rapid, making it appropriate for real-time practical applications.