Search

Background: Reinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients. Methods: With data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations. Results: We created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation. Conclusion: Cross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond. [ABSTRACT FROM AUTHOR]

Background: The identification of risk factors for adverse outcomes and prolonged intensive care unit (ICU) stay in COVID-19 patients is essential for prognostication, determining treatment intensity, and resource allocation. Previous studies have determined risk factors on admission only, and included a limited number of predictors. Therefore, using data from the highly granular and multicenter Dutch Data Warehouse, we developed machine learning models to identify risk factors for ICU mortality, ventilator-free days and ICU-free days during the course of invasive mechanical ventilation (IMV) in COVID-19 patients., Methods: The DDW is a growing electronic health record database of critically ill COVID-19 patients in the Netherlands. All adult ICU patients on IMV were eligible for inclusion. Transfers, patients admitted for less than 24 h, and patients still admitted at time of data extraction were excluded. Predictors were selected based on the literature, and included medication dosage and fluid balance. Multiple algorithms were trained and validated on up to three sets of observations per patient on day 1, 7, and 14 using fivefold nested cross-validation, keeping observations from an individual patient in the same split., Results: A total of 1152 patients were included in the model. XGBoost models performed best for all outcomes and were used to calculate predictor importance. Using Shapley additive explanations (SHAP), age was the most important demographic risk factor for the outcomes upon start of IMV and throughout its course. The relative probability of death across age values is visualized in Partial Dependence Plots (PDPs), with an increase starting at 54 years. Besides age, acidaemia, low P/F-ratios and high driving pressures demonstrated a higher probability of death. The PDP for driving pressure showed a relative probability increase starting at 12 cmH 2 O., Conclusion: Age is the most important demographic risk factor of ICU mortality, ICU-free days and ventilator-free days throughout the course of invasive mechanical ventilation in critically ill COVID-19 patients. pH, P/F ratio, and driving pressure should be monitored closely over the course of mechanical ventilation as risk factors predictive of these outcomes.