Search

Background: Reinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients. Methods: With data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations. Results: We created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation. Conclusion: Cross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond. [ABSTRACT FROM AUTHOR]

Key summary points: Aim: To investigate the prevalence of non-pharmacological interventions (NPIs) in older homecare recipients in six European countries. Findings: The prevalence of NPIs varied considerably between homecare users in different European countries. Interventions with a potential positive impact on health outcomes show a relatively low prevalence. Message: Further research into better implementation of potentially beneficial interventions in treatment guidelines might be needed, in order to optimize functioning and quality of life of older homecare recipients. Purpose: Non-pharmacological interventions (NPIs) play an important role in the management of older people receiving homecare. However, little is known about how often specific NPIs are being used and to what extent usage varies between countries. The aim of the current study was to investigate the prevalence of NPIs in older homecare recipients in six European countries. Methods: This is a cross-sectional study of older homecare recipients (65+) using baseline data from the longitudinal cohort study 'Identifying best practices for care-dependent elderly by Benchmarking Costs and outcomes of community care' (IBenC). The analyzed NPIs are based on the interRAI Home Care instrument, a comprehensive geriatric assessment instrument. The prevalence of 24 NPIs was analyzed in Belgium, Germany, Finland, Iceland, Italy and the Netherlands. NPIs from seven groups were considered: psychosocial interventions, physical activity, regular care interventions, special therapies, preventive measures, special aids and environmental interventions. Results: A total of 2884 homecare recipients were included. The mean age at baseline was 82.9 years and of all participants, 66.9% were female. The intervention with the highest prevalence in the study sample was 'emergency assistance available' (74%). Two other highly prevalent interventions were 'physical activity' (69%) and 'home nurse' (62%). Large differences between countries in the use of NPIs were observed and included, for example, 'going outside' (range 7–82%), 'home health aids' (range 12–93%), and 'physician visit' (range 24–94%). Conclusions: The use of NPIs varied considerably between homecare users in different European countries. It is important to better understand the barriers and facilitators of use of these potentially beneficial interventions in order to design successful uptake strategies. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) has shown remarkable success in artificial domains and in some real-world applications. However, substantial challenges remain such as learning efficiently under safety constraints. Adherence to safety constraints is a hard requirement in many high-impact application domains such as healthcare and finance. These constraints are preferably represented symbolically to ensure clear semantics at a suitable level of abstraction. Existing approaches to safe DRL assume that being unsafe leads to low rewards. We show that this is a special case of symbolically constrained RL and analyze a generic setting in which total reward and being safe may or may not be correlated. We analyze the impact of symbolic constraints and identify a connection between expected future reward and distance towards a goal in an automaton representation of the constraints. We use this connection in an algorithm for learning complex behaviors safely and efficiently. This algorithm relies on symbolic reasoning over safety constraints to improve the efficiency of a subsymbolic learner with a symbolically obtained measure of progress. We measure sample efficiency on a grid world and a conversational product recommender with real-world constraints. The so-called Planning for Potential algorithm converges quickly and significantly outperforms all baselines. Specifically, we find that symbolic reasoning is necessary for safety during and after learning and can be effectively used to guide a neural learner towards promising areas of the solution space. We conclude that RL can be applied both safely and efficiently when combined with symbolic reasoning. [ABSTRACT FROM AUTHOR]

Purpose: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis.Methods: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance.Results: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68-0.99 in the ICU, to 0.96-0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance.Conclusion: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside. [ABSTRACT FROM AUTHOR]

Using a pure machine learning approach to enable the generation of behavior for agents in serious gaming applications can be problematic, because such applications often require human-like behavior for agents that interact with human players. Such human-like behavior is not guaranteed with e.g. basic reinforcement learning schemes. Cognitive models can be very useful to establish human-like behavior in an agent. However, they require ample domain knowledge that might be difficult to obtain. In this paper, a cognitive model is taken as a basis, and the addition of scenario specific information is for a large part automated by means of machine learning techniques. The performance of the approach of automatically adding scenario specific information is rigorously evaluated using a case study in the domain of fighter air combat. An evolutionary algorithm is proposed for automatically tailoring a cognitive model for situation awareness of fighter pilots. The standard algorithm and several extensions are evaluated with respect to performance in air combat. The results show that it is possible to apply the algorithm to optimize belief networks for cognitive models of intelligent agents (adversarial fighters) in the aforementioned domain, thereby reducing the effort required to elicit knowledge from experts, while retaining the required 'human-like' behavior. [ABSTRACT FROM AUTHOR]

When developing intelligent agents, approaches that allow the anticipation of other agents is of utmost importance. For humans, this has also been shown to be crucial to establish good interactions. In this paper, a design for an agent that is equipped with theory of mind based reasoning capabilities is presented. The approach moves beyond the state of the art from several angles: first, it allows for the expression of certainties with respect to the predicted states of the other agents. Second, it allows the prediction during a substantial number of time steps in the future, thereby utilizing the theory of mind model multiple times. The approach has been applied to the domain of fighter pilots whereby intelligent opponents are developed to facilitate dedicated training for F16 fighter pilots. [ABSTRACT FROM AUTHOR]

Collective decision making involves on the one hand individual mental states such as beliefs, emotions and intentions, and on the other hand interaction with others with possibly different mental states. Achieving a satisfactory common group decision on which all agree requires that such mental states are adapted to each other by social interaction. Recent developments in social neuroscience have revealed neural mechanisms by which such mutual adaptation can be realised. These mechanisms not only enable intentions to converge to an emerging common decision, but at the same time enable to achieve shared underlying individual beliefs and emotions. This paper presents a computational model for such processes. As an application of the model, an agent-based analysis was made of patterns in crowd behaviour, in particular to simulate a real-life incident that took place on May 4, 2010 in Amsterdam. From available video material and witness reports, useful empirical data were extracted. Similar patterns were achieved in simulations, whereby some of the parameters of the model were tuned to the case addressed, and most parameters were assigned default values. The results show the inclusion of contagion of belief, emotion, and intention states of agents results in better reproduction of the incident than non-inclusion. [ABSTRACT FROM AUTHOR]

To support human functioning, ambient intelligent agents require knowledge about the tasks executed by the human. This knowledge includes design-time information like: (i) the goal of a task and (ii) the alternative ways for a human to achieve that goal, as well as run-time information such as the choices made by a human during task execution. In order to provide effective support, the agent must know exactly what steps the human is following. However, if not all steps along the path can be observed, it is possible that the agent cannot uniquely derive which path the human is following. Furthermore, in order to provide timely support, the agent must observe, reason, conclude and support within a limited period of time. To deal with these problems, this paper presents a generic focused reasoning mechanism to enable a guided selection of the path which is most likely followed by the human. This mechanism is based upon knowledge about the human and the workflow to perform the task. In order to come to such an approach, a reasoning mechanism is adopted in combination with the introduction of a new workflow representation, which is utilized to focus the reasoning process in an appropriate manner. The approach is evaluated by means of an extensive case study. [ABSTRACT FROM AUTHOR]

ificial Intelligence has contributed (formal) design models and software support tools to application areas such as architecture, engineering and software design. This paper explores the effectiveness of applying design models to the area of organization (re)design. To that purpose a component-based model for (re)design of organizations is presented as a specialization of an existing generic design model. Using recently developed formalizations within Organization Theory organization models are described as design object descriptions, and organization goals as design requirements. A formal design process description is presented that models the redesign process for an organization that adapts to changes in the environment. The formally specified and implemented approach to organization redesign thus obtained has been tested for a well-known historical case study from the Organization Theory literature. [ABSTRACT FROM AUTHOR]

Organizations change with the dynamics of the world. To enable organizations to change, certain structures and capabilities are needed. As all processes, a change process has an organization of its own. In this paper it is shown how within a formal organization modeling approach also organizational change processes can be modeled. A generic organization model (covering both organization structure and behavior) for organizational change is presented and formally evaluated for a case study. This model takes into account different phases in a change process considered in Organization Theory literature, such as unfreezing, movement and refreezing. Moreover, at the level of individuals, the internal beliefs and their changes are incorporated in the model. In addition, an internal mental model for (reflective) reasoning about expected role behavior is included in the organization model. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editor discusses various topics within the issue including analysis of human behavior, the field of robotics, and artificial intelligence techniques.