Towards Real-World Applications of Personalized Anesthesia Using Policy Constraint Q Learning for Propofol Infusion Control

Automated anesthesia promises to enable more personalized and precise anesthetic administration and free anesthesiologists from repetitive tasks, allowing them to focus on the most critical aspects of surgical care for patients. Current research has typically focused on creating simulated environments from which agents can learn. These approaches have demonstrated good experimental results, but are still far from clinical application. In this paper, Policy Constraint Q-Learning (PCQL), a data-driven reinforcement learning algorithm for solving the problem of learning strategies on real world anesthesia data, is proposed. Conservative Q-Learning is first introduced to alleviate the problem of Q function overestimation in an offline context. A policy constraint term is then added to agent training to keep the action distribution of the agent and the anesthesiologist consistent, ensuring the agent makes safer decisions in anesthesia scenarios. The effectiveness of PCQL was validated by extensive experiments on a real clinical anesthesia dataset we collected. Experimental results show that PCQL is predicted to achieve higher gains than the baseline approaches while maintaining good agreement with the reference dose given by the anesthesiologist, using less total dose, and being more responsive to the patient's vital signs. In addition, the confidence intervals of the agent were investigated and were founded to cover most of the clinical decisions of the anesthesiologist. Finally, an interpretable method, SHAP, was used to analyze the contributing components of the model predictions, increasing the transparency of the model.