Search

Reinforcement learning (RL) is a learning paradigm that can be useful in a wide variety of real-world applications. However, its applicability to complex problems remains problematic due to different causes. Particularly important among these are the high quantity of data required by the agent to learn useful policies and the poor scalability to high-dimensional problems due to the use of local approximators. This paper presents a novel RL algorithm, called online fitted policy iteration (OFPI), that steps forward in both directions. OFPI is based on a semi-batch scheme that increases the convergence speed by reusing data and enables the use of global approximators by reformulating the value function approximation as a standard supervised problem. The proposed method has been empirically evaluated in three benchmark problems. During the experiments, OFPI has employed a neural network trained with the extreme learning machine algorithm to approximate the value functions. Results have demonstrated the stability of OFPI using a global function approximator and also performance improvements over two baseline algorithms (SARSA and Q-learning) combined with eligibility traces and a radial basis function network.

Clustering techniques and classification trees are two of the main techniques used in data mining but, at present, there is still a lack of visualization methods for these tools. Many graphs associated with clustering, also with hierarchical clustering, do not give any information about the values of the centroids’ attributes and the relationships among them. In classification trees, graphical procedures can also be developed to help simplify their interpretation and to obtain a better understanding, but more visualization methods to support this tool are needed. This paper presents a novel visualization technique called sectors on sectors (SonS), and an extended version called multidimensional sectors on sectors (MDSonS), for improving the interpretation of several data mining algorithms. These methods are applied for visualizing the results of: (a) hierarchical clustering, which makes it possible to extract all the existing relationships among centroids’ attributes at any hierarchy level; (b) growing hierarchical self-organizing maps (GHSOM), a variant of the well-known self-organizing maps (SOM), by means of which it is possible to visualize, simultaneously, the data information at each hierarchy level compactly and extract relationships among variables; (c) classification trees, in which the SonS is used for representing the input data information for each class presented in each terminal node of a classification tree providing extra information for a better understanding of the problem. These methods are tested by means of several data sets (real and synthetic). The achieved results show the suitability and usefulness of the proposed approaches.

The comfort in footwear is essential because the foot is one of the structures of the human body that supports more weight. Moreover, consumers are demanding ever higher and higher levels of comfort and functionality in shoes. Hence, the analysis of the comfort in the footwear industry is of great interest. This paper proposes the use of SOMs to qualitatively evaluate data related to comfort in footwear provided by the Spanish Technological Institute for Footwear and Related Industries. This work also studies which factors can be decisive when buying footwear, revealing interesting hidden relationships and qualitative patterns. For this purpose, the features that may play a relevant role in this framework, and the crucial relationships between them are studied for the comfort of a given shoe. This study tries to find the way of jointly representing comfort valuations for different areas of the foot with different variables (related to physical characteristics of the testers and characteristics or physical measures of foot-footwear) in order to find out if there is a difference between buying/not buying groups.

Chronic Kidney Disease (CKD) anemia is one of the main common comorbidities in patients undergoing End Stage Renal Disease (ESRD). Iron supplement and especially Erythropoiesis Stimulating Agents (ESA) have become the treatment of choice for that anemia. However, it is very complicated to find an adequate treatment for every patient in each particular situation since dosage guidelines are based on average behaviors, and thus, they do not take into account the particular response to those drugs by different patients, although that response may vary enormously from one patient to another and even for the same patient in different stages of the anemia. This work proposes an advance with respect to previous works that have faced this problem using different methodologies (Machine Learning (ML), among others), since the diversity of the CKD population has been explicitly taken into account in order to produce a general and reliable model for the prediction of ESA/Iron therapy response. Furthermore, the ML model makes use of both human physiology and drug pharmacology to produce a model that outperforms previous approaches, yielding Mean Absolute Errors (MAE) of the Hemoglobin (Hb) prediction around or lower than 0.6g/dl in the three countries analyzed in the study, namely, Spain, Italy and Portugal. HighlightsPrediction algorithm trained and tested on a large sample of real clinical data.Prediction improvement based on red blood cell dynamics and drug kinetics.There is still room for improvement of anemia management in dialysis.The model presented is suitable for the application in a daily clinical practice.

Progress in biomechanical modelling of human soft tissue is the basis for the development of new clinical applications capable of improving the diagnosis and treatment of some diseases (e.g. cancer), as well as the surgical planning and guidance of some interventions. The finite element method (FEM) is one of the most popular techniques used to predict the deformation of the human soft tissue due to its high accuracy. However, FEM has an associated high computational cost, which makes it difficult its integration in real-time computer-aided surgery systems. An alternative for simulating the mechanical behaviour of human organs in real time comes from the use of machine learning (ML) techniques, which are much faster than FEM. This paper assesses the feasibility of ML methods for modelling the biomechanical behaviour of the human liver during the breathing process, which is crucial for guiding surgeons during interventions where it is critical to track this deformation (e.g. some specific kind of biopsies) or for the accurate application of radiotherapy dose to liver tumours. For this purpose, different ML regression models were investigated, including three tree-based methods (decision trees, random forests and extremely randomised trees) and other two simpler regression techniques (dummy model and linear regression). In order to build and validate the ML models, a labelled data set was constructed from modelling the deformation of eight ex-vivo human livers using FEM. The best prediction performance was obtained using extremely randomised trees, with a mean error of 0.07 mm and all the samples with an error under 1 mm. The achieved results lay the foundation for the future development of some real-time software capable of simulating the human liver deformation during the breathing process during clinical interventions., This work has been funded by the Spanish Ministry of Economy and Competitiveness (MINECO) through research projects TIN2014-52033-R and DPI2013-40859-R, both also supported by European FEDER funds. The authors acknowledge the kind collaboration of the personnel from the hospital involved in the research.

We propose a quantum machine learning algorithm for efficiently solving a class of problems encoded in quantum controlled unitary operations. The central physical mechanism of the protocol is the iteration of a quantum time-delayed equation that introduces feedback in the dynamics and eliminates the necessity of intermediate measurements. The performance of the quantum algorithm is analyzed by comparing the results obtained in numerical simulations with the outcome of classical machine learning methods for the same problem. The use of time-delayed equations enhances the toolbox of the field of quantum machine learning, which may enable unprecedented applications in quantum technologies. The authors acknowledge support from Basque Government grants BFI-2012-322 and IT986-16, Spanish MINECO/FEDER FIS2015-69983-P, Ramon y Cajal Grant RYC-2012-11391, and UPV/EHU UFI 11/55.

HighlightsDifferent prediction algorithms were used to predict Hb levels in CRF patients.Prediction errors in the validation cohorts of patients were around 0.6g/dl.Difficulty to obtain lower errors due to the measuring machine precision (0.2g/dl).Relevance analysis of features have been applied for each predictor. Patients who suffer from chronic renal failure (CRF) tend to suffer from an associated anemia as well. Therefore, it is essential to know the hemoglobin (Hb) levels in these patients. The aim of this paper is to predict the hemoglobin (Hb) value using a database of European hemodialysis patients provided by Fresenius Medical Care (FMC) for improving the treatment of this kind of patients. For the prediction of Hb, both analytical measurements and medication dosage of patients suffering from chronic renal failure (CRF) are used. Two kinds of models were trained, global and local models. In the case of local models, clustering techniques based on hierarchical approaches and the adaptive resonance theory (ART) were used as a first step, and then, a different predictor was used for each obtained cluster. Different global models have been applied to the dataset such as Linear Models, Artificial Neural Networks (ANNs), Support Vector Machines (SVM) and Regression Trees among others. Also a relevance analysis has been carried out for each predictor model, thus finding those features that are most relevant for the given prediction.

Detection of ventricular fibrillation (VF) at an early stage is being deeply studied in order to lower the risk of sudden death and allows the specialist to have greater reaction time to give the patient a good recovering therapy. Some works are focusing on detecting VF based on numerical analysis of time-frequency distributions, but in general the methods used do not provide insight into the problem. However, this study proposes a new methodology in order to obtain information about this problem. This work uses a supervised self-organising map (SOM) to obtain visually information among four important groups of patients: VF (ventricular fibrillation), VT (ventricular tachycardia), HP (healthy patients) and AHR (other anomalous heart rates and noise). A total number of 27 variables were obtained from continuous surface ECG recordings in standard databases (MIT and AHA), providing information in the time, frequency, and time-frequency domains. self-organising maps (SOMs), trained with 11 of the 27 variables, were used to extract knowledge about the variable values for each group of patients. Results show that the SOM technique allows to determine the profile of each group of patients, assisting in gaining a deeper understanding of this clinical problem. Additionally, information about the most relevant variables is given by the SOM analysis.

Neural Networks (NN) are one of the most used machine learning techniques in different areas of knowledge. This has led to the emergence of a large number of courses of Neural Networks around the world and in areas where the users of this technique do not have a lot of programming skills. Current software that implements these elements, such as Matlab®, has a number of important limitations in teaching field. In some cases, the implementation of a MLP requires a thorough knowledge of the software and of the instructions that train and validate these systems. In other cases, the architecture of the model is fixed and they do not allow an automatic sweep of the parameters that determine the architecture of the network. This chapter presents a teaching tool for the its use in courses about neural models that solves some of the above-mentioned limitations. This tool is based on Matlab® software.

Objective: Anemia is a frequent comorbidity in hemodialysis patients that can be successfully treated by administering erythropoiesis-stimulating agents (ESAs). ESAs dosing is currently based on clinical protocols that often do not account for the high inter- and intra-individual variability in the patient's response. As a result, the hemoglobin level of some patients oscillates around the target range, which is associated with multiple risks and side-effects. This work proposes a methodology based on reinforcement learning (RL) to optimize ESA therapy. Methods: RL is a data-driven approach for solving sequential decision-making problems that are formulated as Markov decision processes (MDPs). Computing optimal drug administration strategies for chronic diseases is a sequential decision-making problem in which the goal is to find the best sequence of drug doses. MDPs are particularly suitable for modeling these problems due to their ability to capture the uncertainty associated with the outcome of the treatment and the stochastic nature of the underlying process. The RL algorithm employed in the proposed methodology is fitted Q iteration, which stands out for its ability to make an efficient use of data. Results: The experiments reported here are based on a computational model that describes the effect of ESAs on the hemoglobin level. The performance of the proposed method is evaluated and compared with the well-known Q-learning algorithm and with a standard protocol. Simulation results show that the performance of Q-learning is substantially lower than FQI and the protocol. Conclusion: Although prospective validation is required, promising results demonstrate the potential of RL to become an alternative to current protocols., Comment: 17 pages, 10 figures

The development of systems for automatically detecting decay in citrus fruit during quality control is still a challenge for the citrus industry. The feasibility of reflectance spectroscopy in the visible and near infrared (NIR) regions was evaluated for the automatic detection of the early symptoms of decay caused by Penicillium digitatum fungus in citrus fruit. Reflectance spectra of sound and decaying surface parts of mandarins cv. ‘Clemenvilla’ were acquired in two different spectral regions, from 650 nm to 1050 nm (visible–NIR) and from 1000 nm to 1700 nm (NIR), pointing to significant differences in spectra between sound and decaying skin for both spectral ranges. Three different manifold learning methods (principal component analysis, factor analysis and Sammon mapping) were investigated to transform the high-dimensional spectral data into meaningful representations of reduced dimensionality containing the essential information. The low-dimensional data representations were used as input feature vectors to discriminate between sound and decaying skin using a supervised classifier based on linear discriminant analysis. The best classification results were achieved by employing factor analysis on the NIR spectra, yielding a maximum overall classification accuracy of 97.8%, with a percentage of well-classified sound and decaying samples of 100% and 94.4%, respectively. These results lay the foundation for the future implementation of reflectance spectroscopy technology on a commercial fruit sorter for the purpose of detecting decay in citrus fruit.

[EN] Green mold (Penicillium digitatum) and blue mold (Penicillium italicum) are important sources of postharvest decay affecting the commercialization of mandarins. These fungi infections produce enormous economic losses in mandarin production if early detection is not carried out. Nowadays, this detection is performed manually in dark chambers, where the fruit is illuminated by ultraviolet light to produce fluorescence, which is potentially dangerous for humans. This paper documents a new methodology based on hyperspectral imaging and advanced machine-learning techniques (artificial neural networks and classification and regression trees) for the segmentation and classification of images of citrus free of damage and affected by green mold and blue mold. Feature selection methods are used in order to reduce the dimensionality of the hyperspectral images and determine the 10 most relevant. Neural Networks were used to segment the hyperspectral images. Result's achieved using classifiers based on decision trees show an accuracy of around 93% in the problem of decay classification., This work was partially funded by the University of Valencia through project UV-INV-AE11-41271 and by the Instituto Nacional de Investigacion y Tecnologia Agraria y Alimentaria de Espana (INIA) through research projects RTA2012-00062-C04-01 and RTA2012-00062-C04-03

Sparse Manifold Clustering and Embedding (SMCE) algorithm has been recently proposed for simultaneous clustering and dimensionality reduction of data on nonlinear manifolds using sparse representation techniques. In this work, SMCE algorithm is applied to the differential discrimination of Glioblastoma and Meningioma Tumors by means of their Gene Expression Profiles. Our purpose was to evaluate the robustness of this nonlinear manifold to classify gene expression profiles, characterized by the high-dimensionality of their representations and the low discrimination power of most of the genes. For this objective, we used SMCE to reduce the dimensionality of a preprocessed dataset of 35 single-labeling cDNA microarrays with 11500 original clones. Afterwards, supervised and unsupervised methodologies were applied to obtain the classification model: the former was based on linear discriminant analysis, the later on clustering using the SMCE embedding data. The results obtained using both approaches showed that all (100%) the samples could be correctly classified and the results of all repetitions but one formed a compatible cluster of predictive labels. Finally, the embedding dimensionality of the dataset extracted by SMCE revealed large discrimination margins between both classes. (c) 2013 Elsevier Ltd. All rights reserved., This work was supported by the University of Valencia through project UV-INV-AE11-41271.

Feature selection methods in machine learning models are a powerful tool to knowledge extraction. In this work they are used to analyse the intrinsic modifications of cardiac response during ventricular fibrillation due to physical exercise. The data used are two sets of registers from isolated rabbit hearts: control (G1: without physical training), and trained (G2). Four parameters were extracted (dominant frequency, normalized energy, regularity index and number of occurrences). From them, 18 features were extracted. This work analyses the relevance of each feature to classify the records in G1 and G2 using Logistic Regression, Multilayer Perceptron and Extreme Learning Machine. Three feature selection methods are presented: one based on the output variation, other on the classification results and, finally, another method based in the variation in ROC curve. Although we obtained different sorting of features for each used classifier, the features related to the mean value and standard deviation of dominant frequency and regularity index were the most relevant, stating that the modifications in VF response produced by physical exercise are related to the cardiac activation rate, as to the regularity of that activation.

The citrus industry is nowadays an important part of the Spanish agricultural sector. One of the main problems present in the citrus industry is decay caused by Penicillium digitatum and Penicillium italicum fungi. Early detection of decay produced by fungi in citrus is especially important for the citrus industry of distribution. This chapter presents a hyperspectral computer vision system and a set of machine learning techniques in order to detect decay caused by Penicillium digitatum and Penicillium italicum fungi that produce more economic losses to the sector. More specifically, the authors employ a hyperspectral system and artificial neural networks. Nowadays, inspection and removal of damaged citrus is done manually by workers using dangerous ultraviolet light. The proposed system constitutes a feasible and implementable solution for the citrus industry; this has been proven by the fact that several machinery enterprises have shown their interest in the implementation and patent of the system.

The basis of all clinical science developments is the analysis of the data obtained from a particular problem. In recent decades, however, the capacity of computers to process data has been increasing exponentially, which has created the possibility of applying more powerful methods of data analysis. Among these methods, the multidimensional visual data mining methods are outstanding. These methods show all the variables of one particular problem on the whole allowing to the clinical specialist to extract his own conclusions. In this chapter, a neural approximation to this kind of data mining is shown by means of the valuation analysis of the knee in athletes in the pre- and post-surgery of the anterior cruciate ligament, studying variables of force and measurements at different distances of the knee.

The aim of this work is to study the applicability of reinforcement learning methods to design adaptive treatment strategies that optimize, in the long-term, the dosage of erythropoiesis-stimulating agents (ESAs) in the management of anemia in patients undergoing hemodialysis. Adaptive treatment strategies are recently emerging as a new paradigm for the treatment and long-term management of the chronic disease. Reinforcement Learning (RL) can be useful to extract such strategies from clinical data, taking into account delayed effects and without requiring any mathematical model. In this work, we focus on the so-called Fitted Q Iteration algorithm, a RL approach that deals with the data very efficiently. Achieved results show the suitability of the proposed RL policies that can improve the performance of the treatment followed in the clinics. The methodology can be easily extended to other problems of drug dosage optimization.

Clustering techniques have been widely applied to extract information from high-dimensional data structures in the last few years. Graphs are especially relevant for clustering, but many graphs associated with hierarchical clustering do not give any information about the values of the centroids' attributes and the relationships among them. In this paper, we propose a new visualization approach for hierarchical cluster analysis in which the above-mentioned information is available. The method is based on pie charts. The pie charts are divided into several pie segments or sectors corresponding to each cluster. The radius of each pie segment is proportional to the number of patterns included in each cluster. By means of new divisions in each pie sector and a color bar with as many labels as attributes, we can extract all the existing relationships among centroids' attributes at any hierarchy level. The methodology is tested in one synthetic data set and one real data set. Achieved results show the suitability and usefulness of the proposed approach.