Search

Clustering of users underlies many of the personalisation algorithms that are in use nowadays. Such clustering is mostly performed in an offline fashion. For a health and wellbeing setting, offline clustering might however not be suitable, as limited data is often available and patient states can also quickly evolve over time. Existing online clustering algorithms are not suitable for the health domain due to the type of data that involves multiple time series evolving over time. In this paper we propose a new online clustering algorithm called CluStream-GT that is suitable for health applications. By using both artificial and real datasets, we show that the approach is far more efficient compared to regular clustering, with an average speedup of 93%, while only losing 12% in the accuracy of the clustering with artificial data and 3% with real data.CCS CONCEPTS• Computing methodologies$\rightarrow$ Cluster analysis.

Language systems have been of great interest to the research community and have recently reached the mass market through various assistant platforms on the web. Reinforcement Learning methods that optimize dialogue policies have seen successes in past years and have recently been extended into methods that personalizethe dialogue, e.g. take the personal context of users into account. These works, however, are limited to personalization to a single user with whom they require multiple interactions and do not generalize the usage of context across users. This work introduces a problem where a generalized usage of context is relevant and proposes two Reinforcement Learning (RL)-based approaches to this problem. The first approach uses a single learner and extends the traditional POMDP formulation of dialogue state with features that describe the user context. The second approach segments users by context and then employs a learner per context. We compare these approaches in a benchmark of existing non-RL and RL-based methods in three established and one novel application domain of financial product recommendation. We compare the influence of context and training experiences on performance and find that learning approaches generally outperform a handcrafted gold standard. CCS CONCEPTS • Computing methodologies → Reinforcement learning; Discourse, dialogue and pragmatics; • Human-centered computing → Natural language interfaces.

Online fraud poses a relatively new threat to the revenues of companies. A way to detect and prevent fraudulent behavior is with the use of specific machine learning (ML) techniques. These anomaly detection techniques have been thoroughly studied, but the level of employment is not as high. The airline industry suffers from fraud by parties such as online travel agencies (OTAs). These agencies are commissioned by an airline carrier to sell its travel tickets. Through policy violations, they can illegitimately claim some of the airline’s revenue by offering cheaper fares to customers.This research applies several anomaly detection techniques to detect fraudulent behavior by OTAs and assesses their strengths and weaknesses. Since the data is not labeled, it is not known whether fraud has actually occurred. Therefore, unsupervised ML is used. The contributions of this paper are, firstly, to show how to shape the online booking data and how to engineer new and relevant features. Secondly, this research includes a case study in which domain experts evaluate the detection performance of the considered ML methods by classifying a set of 75 bookings. According to the experts’ analysis, the techniques are able to discover previously unknown fraudulent bookings, which will not have been found otherwise. This demonstrates that anomaly detection is a valuable tool for the airline industry to discover fraudulent behavior.

The huge wealth of data in the health domain can be exploited to create models that predict development of health states over time. Temporal learning algorithms are well suited to learn relationships between health states and make predictions about their future developments. However, these algorithms: (1) either focus on learning one generic model for all patients, providing general insights but often with limited predictive performance, or (2) learn individualized models from which it is hard to derive generic concepts. In this paper, we present a middle ground, namely parameterized dynamical systems models that are generated from data using a Genetic Programming (GP) framework. A fitness function suitable for the health domain is exploited. An evaluation of the approach in the mental health domain shows that performance of the model generated by the GP is on par with a dynamical systems model developed based on domain knowledge, significantly outperforms a generic Long Term Short Term Memory (LSTM) model and in some cases also outperforms an individualized LSTM model.

Personalisation has become omnipresent in society. For the domain of health and wellbeing such personalisation can contribute to better interventions and improved health states of users. In order for personalisation to be effective in this domain, it needs to be performed quickly and with minimal impact on the users. Reinforcement learning is one of the techniques that can be used to establish such personalisation, but it is not known to be very fast at learning. Cluster-based reinforcement learning has been proposed to improve the learning speed. Here, users who show similar behaviour are clustered and one policy is learned for each individual cluster. An important factor in this effort is the method used for clustering, which has the potential to influence the benefit of such an approach. In this paper, we propose three distance metrics based on the state of the users (Euclidean distance, Dynamic Time Warping, and high-level features) and apply different clustering techniques given these distance metrics to study their impact on the overall performance. We evaluate the different methods in a simulator with users spawned from very distinct user profiles as well as overlapping user profiles. The results show that clustering configurations using high-level features significantly outperform regular reinforcement learning without clustering (which either learn one policy for all or one policy per individual).

Trace clustering has increasingly been applied to find homogenous process executions. However, current techniques have difficulties in finding a meaningful and insightful clustering of patients on the basis of healthcare data. The resulting clusters are often not in line with those of medical experts, nor do the clusters guarantee to help return meaningful process maps of patients’ clinical pathways. After all, a single hospital may conduct thousands of distinct activities and generate millions of events per year. In this paper, we propose a novel trace clustering approach by using sample sets of patients provided by medical experts. More specifically, we learn frequent sequence patterns on a sample set, rank each patient based on the patterns, and use an automated approach to determine the corresponding cluster. We find each cluster separately, while the frequent sequence patterns are used to discover a process map. The approach is implemented in ProM and evaluated using a large data set obtained from a university medical center. The evaluation shows F1-scores of 0.7 for grouping kidney injury, 0.9 for diabetes, and 0.64 for head/neck tumor, while the process maps show meaningful behavioral patterns of the clinical pathways of these groups, according to the domain experts.

Personalization of support in health and wellbeing settings is challenging. While personalization has shown to be highly beneficial to maximize the success of interventions, often only very limited experiences are available to personalize support strategies. Because of its focus on finding suitable actions/interventions that lead to long term rewards, reinforcement learning is very suitable for personalization but requires a substantial learning period. To overcome this so-called cold start problem, we propose a novel approach called narrowing reinforcement learning. The approach exploits experiences of the nearest neighbors around a user to generate a suitable policy, expressing which action to perform in what state. Using a narrowing function, the size of the neighborhood is reduced as more experiences are collected, allowing for the most personalized experience that is possible given the amount of collected experiences. An evaluation of the approach in a realistic simulator shows that it significantly outperforms the current state-of-the-art approaches for personalization in health and wellbeing using reinforcement learning.

Research has shown that personalization of health interventions can contribute to an improved effectiveness. Reinforcement learning algorithms can be used to perform such tailoring. In this paper, we present a cluster-based reinforcement learning approach which learns optimal policies for groups of users. Such an approach can speed up the learning process while still giving a level of personalization. We apply both online and batch learning to learn policies over the clusters and introduce a publicly available simulator which we have developed to evaluate the approach. The results show batch learning significantly outperforms online learning. Furthermore, near-optimal clustering is found which proves to be beneficial in learning significantly better policies compared to learning per user and learning across all users.

Simulation environments have proven to be very useful as testbeds for reinforcement learning (RL) algorithms. For settings where an actual human user is involved, these simulation environments allow one to test out the suitability of new RL approaches without having to include real users at first. It obviously does require the simulator to have a certain degree of realism, however, realistic simulators for the behavior of humans in the health domain are rarely seen. To generate realistic behavior, the simulator could be driven by data from real users, but this might lead to privacy issues. In this paper, we propose to use Generative Adversarial Networks (GANs) for generating realistic simulation environments. In this first step, we use an existing simulator that simulates daily activities of users and the GANs are used to generate realistic sensory data that accompanies such activities. After training, the original (potentially privacy sensitive) data can be thrown away and the simulator can simply be driven by the GAN models. Results show that a model trained on real data shows similar performance on the data artificially generated by the GAN.

In this paper we present the design of an agent system for medicine usage management. Part of it is an intelligent ambient agent which incorporates an explicit representation of a dynamical system model to estimate the medicine level in the patient’s body. By simulation the ambient agent is able to analyse whether the patient intends to take the medicine too early or too late, and can take measures to prevent this.