Continuous action deep reinforcement learning for propofol dosing during general anesthesia

Comparison of time-series models for predicting physiological metrics under sedation

This study presents a comprehensive comparison of multiple time-series models applied to physiological metric predictions. It aims to explore the effectiveness of both statistical prediction models and pharmacokinetic-pharmacodynamic prediction model and modern deep learning approaches. Specifically, the study focuses on predicting the bispectral index (BIS), a vital metric in anesthesia used to assess the depth of sedation during surgery, using datasets collected from real-life surgeries. The goal is to evaluate and compare model performance considering both univariate and multivariate schemes. Accurate BIS prediction is essential for avoiding under- or over-sedation, which can lead to adverse outcomes. The study investigates a range of models: The traditional mathematical models include the pharmacokinetic-pharmacodynamic model and statistical models such as autoregressive integrated moving average (ARIMA) and vector autoregression (VAR). The deep learning models encompass recurrent neural networks (RNNs), specifically Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), as well as Temporal Convolutional Networks (TCNs) and Transformer models. The analysis focuses on evaluating model performance in predicting the BIS using two distinct datasets of physiological metrics collected from actual surgical procedures. It explores both univariate and multivariate prediction schemes and investigates how different combinations of features and input sequence lengths impact model accuracy. The experimental findings reveal significant performance differences among the models: In univariate prediction scenarios for predicting BIS, the LSTM model demonstrates a 2.88% improvement over the second-best performing model. For multivariate predictions, the LSTM model outperforms others by 6.67% compared to the next best model. Furthermore, the addition of Electromyography (EMG) and Mean Arterial Pressure (MAP) brings significant accuracy improvement when predicting BIS. The study emphasizes the importance of selecting and building appropriate time-series models to achieve accurate predictions in biomedical applications. This research provides insights to guide future efforts in improving vital sign prediction methodologies for clinical and research purposes. Clinically, with improvements in the prediction of physiological parameters, clinicians can be informed of interventions if an anomaly is detected or predicted.

Model-Informed Reinforcement Learning for Enabling Precision Dosing Via Adaptive Dosing

Precision dosing, the tailoring of drug doses to optimize therapeutic benefits and minimize risks in each patient, is essential for drugs with a narrow therapeutic window and severe adverse effects. Adaptive dosing strategies extend the precision dosing concept to time-varying treatments which require sequential dose adjustments based on evolving patient conditions. Reinforcement learning (RL) naturally fits this paradigm: it perfectly mimics the sequential decision-making process where clinicians adapt dose administration based on patient response and evolution monitoring. This paper aims to investigate the potentiality of coupling RL with population PK/PD models to develop precision dosing algorithms, reviewing the most relevant works in the field. Case studies in which PK/PD models were integrated within RL algorithms as simulation engine to predict consequences of any dosing action have been considered and discussed. They mainly concern propofol-induced anesthesia, anticoagulant therapy with warfarin and a variety of anticancer treatments differing for administered agents and/or monitored biomarkers. The resulted picture highlights a certain heterogeneity in terms of precision dosing approaches, applied methodologies, and degree of adherence to the clinical domain. In addition, a tutorial on how a precision dosing problem should be formulated in terms of the key elements composing the RL framework (i.e., system state, agent actions and reward function), and on how PK/PD models could enhance RL approaches is proposed for readers interested in delving in this field. Overall, the integration of PK/PD models into a RL-framework holds great promise for precision dosing, but further investigations and advancements are still needed to address current limitations and extend the applicability of this methodology to drugs requiring adaptive dosing strategies.

Application of Machine Learning in Predicting Perioperative Outcomes in Patients with Cancer: A Narrative Review for Clinicians

This narrative review explores the utilization of machine learning (ML) and artificial intelligence (AI) models to enhance perioperative cancer care. ML and AI models offer significant potential to improve perioperative cancer care by predicting outcomes and supporting clinical decision-making. Tailored for perioperative professionals including anesthesiologists, surgeons, critical care physicians, nurse anesthetists, and perioperative nurses, this review provides a comprehensive framework for the integration of ML and AI models to enhance patient care delivery throughout the perioperative continuum.

Use of Artificial Intelligence in Improving Outcomes in Heart Disease: A Scientific Statement From the American Heart Association

A major focus of academia, industry, and global governmental agencies is to develop and apply artificial intelligence and other advanced analytical tools to transform health care delivery. The American Heart Association supports the creation of tools and services that would further the science and practice of precision medicine by enabling more precise approaches to cardiovascular and stroke research, prevention, and care of individuals and populations. Nevertheless, several challenges exist, and few artificial intelligence tools have been shown to improve cardiovascular and stroke care sufficiently to be widely adopted. This scientific statement outlines the current state of the art on the use of artificial intelligence algorithms and data science in the diagnosis, classification, and treatment of cardiovascular disease. It also sets out to advance this mission, focusing on how digital tools and, in particular, artificial intelligence may provide clinical and mechanistic insights, address bias in clinical studies, and facilitate education and implementation science to improve cardiovascular and stroke outcomes. Last, a key objective of this scientific statement is to further the field by identifying best practices, gaps, and challenges for interested stakeholders.

Clinical knowledge-guided deep reinforcement learning for sepsis antibiotic dosing recommendations

Sepsis is the third leading cause of death worldwide. Antibiotics are an important component in the treatment of sepsis. The use of antibiotics is currently facing the challenge of increasing antibiotic resistance (Evans et al., 2021). Sepsis medication prediction can be modeled as a Markov decision process, but existing methods fail to integrate with medical knowledge, making the decision process potentially deviate from medical common sense and leading to underperformance. (Wang et al., 2021). In this paper, we use Deep Q-Network (DQN) to construct a Sepsis Anti-infection DQN (SAI-DQN) model to address the challenge of determining the optimal combination and duration of antibiotics in sepsis treatment. By setting sepsis clinical knowledge as reward functions to guide DQN complying with medical guidelines, we formed personalized treatment recommendations for antibiotic combinations. The results showed that our model had a higher average value for decision-making than clinical decisions. For the test set of patients, our model predicts that 79.07% of patients will achieve a favorable prognosis with the recommended combination of antibiotics. By statistically analyzing decision trajectories and drug action selection, our model was able to provide reasonable medication recommendations that comply with clinical practices. Our model was able to improve patient outcomes by recommending appropriate antibiotic combinations in line with certain clinical knowledge.

Applications of artificial intelligence in anesthesia: A systematic review

This review article examines the utility of artificial intelligence (AI) in anesthesia, with a focus on recent developments and future directions in the field. A total of 19,300 articles were available on the given topic after searching in the above mentioned databases, and after choosing the custom range of years from 2015 to 2023 as an inclusion component, only 12,100 remained. 5,720 articles remained after eliminating non-full text. Eighteen papers were identified to meet the inclusion criteria for the review after applying the inclusion and exclusion criteria. The applications of AI in anesthesia after studying the articles were in favor of the use of AI as it enhanced or equaled human judgment in drug dose decision and reduced mortality by early detection. Two studies tried to formulate prediction models, current techniques, and limitations of AI; ten studies are mainly focused on pain and complications such as hypotension, with a P value of <0.05; three studies tried to formulate patient outcomes with the help of AI; and three studies are mainly focusing on how drug dose delivery is calculated (median: 1.1% ± 0.5) safely and given to the patients with applications of AI. In conclusion, the use of AI in anesthesia has the potential to revolutionize the field and improve patient outcomes. AI algorithms can accurately predict patient outcomes and anesthesia dosing, as well as monitor patients during surgery in real time. These technologies can help anesthesiologists make more informed decisions, increase efficiency, and reduce costs. However, the implementation of AI in anesthesia also presents challenges, such as the need to address issues of bias and privacy. As the field continues to evolve, it will be important to carefully consider the ethical implications of AI in anesthesia and ensure that these technologies are used in a responsible and transparent manner.

Value function assessment to different RL algorithms for heparin treatment policy of patients with sepsis in ICU

Heparin is a critical aspect of managing sepsis after abdominal surgery, which can improve microcirculation, protect organ function, and reduce mortality. However, there is no clinical evidence to support decision-making for heparin dosage. This paper proposes a model called SOFA-MDP, which utilizes SOFA scores as states of MDP, to investigate clinic policies. Different algorithms provide different value functions, making it challenging to determine which value function is more reliable. Due to ethical restrictions, we cannot test all policies on patients. To address this issue, we proposed two value function assessment methods: action similarity rate and relative gain. We experimented with heparin treatment policies for sepsis patients after abdominal surgery using MIMIC-IV. In the experiments, TD(0) shows the most reliable performance. Using the action similarity rate and relative gain to assess AI policy from TD(0), the agreement rates between AI policy and "good" physician's actual treatment are 64.6% and 73.2%, while the agreement rates between AI policy and "bad" physician's actual treatment are 44.1% and 35.8%, the gaps are 20.5% and 37.4%, respectively. External validation using action similarity rate and relative gain based on eICU resulted in agreement rates of 61.5% and 69.1% with the "good" physician's treatment, and 45.2% and 38.3% with the "bad" physician's treatment, with gaps of 16.3% and 30.8%, respectively. In conclusion, the model provides instructive support for clinical decisions, and the evaluation methods accurately distinguish reliable and unreasonable outcomes.

Development of a Bispectral index score prediction model based on an interpretable deep learning algorithm

BACKGROUND

Proper maintenance of hypnosis is crucial for ensuring the safety of patients undergoing surgery. Accordingly, indicators, such as the Bispectral index (BIS), have been developed to monitor hypnotic levels. However, the black-box nature of the algorithm coupled with the hardware makes it challenging to understand the underlying mechanisms of the algorithms and integrate them with other monitoring systems, thereby limiting their use.

OBJECTIVE

We propose an interpretable deep learning model that forecasts BIS values 25 s in advance using 30 s electroencephalogram (EEG) data.

MATERIAL AND METHODS

The proposed model utilized EEG data as a predictor, which is then decomposed into amplitude and phase components using fast Fourier Transform. An attention mechanism was applied to interpret the importance of these components in predicting BIS. The predictability of the model was evaluated on both regression and binary classification tasks, where the former involved predicting a continuous BIS value, and the latter involved classifying a dichotomous status at a BIS value of 60. To evaluate the interpretability of the model, we analyzed the attention values expressed in the amplitude and phase components according to five ranges of BIS values. The proposed model was trained and evaluated using datasets collected from two separate medical institutions.

RESULTS AND CONCLUSION

The proposed model achieved excellent performance on both the internal and external validation datasets. The model achieved a root-mean-square error of 6.614 for the regression task, and an area under the receiver operating characteristic curve of 0.937 for the binary classification task. Interpretability analysis provided insight into the relationship between EEG frequency components and BIS values. Specifically, the attention mechanism revealed that higher BIS values were associated with increased amplitude attention values in high-frequency bands and increased phase attention values in various frequency bands. This finding is expected to facilitate a more profound understanding of the BIS prediction mechanism, thereby contributing to the advancement of anesthesia technologies.

Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia

Ventilation should be assisted without asynchrony or cardiorespiratory instability during anesthesia emergence until sufficient spontaneous ventilation is recovered. In this multicenter cohort study, we develop and validate a reinforcement learning-based Artificial Intelligence model for Ventilation control during Emergence (AIVE) from general anesthesia. Ventilatory and hemodynamic parameters from 14,306 surgical cases at an academic hospital between 2016 and 2019 are used for training and internal testing of the model. The model's performance is also evaluated on the external validation cohort, which includes 406 cases from another academic hospital in 2022. The estimated reward of the model's policy is higher than that of the clinicians' policy in the internal (0.185, the 95% lower bound for best AIVE policy vs. -0.406, the 95% upper bound for clinicians' policy) and external validation (0.506, the 95% lower bound for best AIVE policy vs. 0.154, the 95% upper bound for clinicians' policy). Cardiorespiratory instability is minimized as the clinicians' ventilation matches the model's ventilation. Regarding feature importance, airway pressure is the most critical factor for ventilation control. In conclusion, the AIVE model achieves higher estimated rewards with fewer complications than clinicians' ventilation control policy during anesthesia emergence.

Deep reinforcement learning-based propofol infusion control for anesthesia: A feasibility study with a 3000-subject dataset

In this work, we present a deep reinforcement learning-based approach as a baseline system for autonomous propofol infusion control. Specifically, design an environment for simulating the possible conditions of a target patient based on input demographic data and design our reinforcement learning model-based system so that it effectively makes predictions on the proper level of propofol infusion to maintain stable anesthesia even under dynamic conditions that can affect the decision-making process, such as the manual control of remifentanil by anesthesiologists and the varying patient conditions under anesthesia. Through an extensive set of evaluations using patient data from 3000 subjects, we show that the proposed method results in stabilization in the anesthesia state, by managing the bispectral index (BIS) and effect-site concentration for a patient showing varying conditions.

A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis

Deep Reinforcement Learning (DRL) has been increasingly attempted in assisting clinicians for real-time treatment of sepsis. While a value function quantifies the performance of policies in such decision-making processes, most value-based DRL algorithms cannot evaluate the target value function precisely and are not as safe as clinical experts. In this study, we propose a Weighted Dueling Double Deep Q-Network with embedded human Expertise (WD3QNE). A target Q value function with adaptive dynamic weight is designed to improve the estimate accuracy and human expertise in decision-making is leveraged. In addition, the random forest algorithm is employed for feature selection to improve model interpretability. We test our algorithm against state-of-the-art value function methods in terms of expected return, survival rate, action distribution and external validation. The results demonstrate that WD3QNE obtains the highest survival rate of 97.81% in MIMIC-III dataset. Our proposed method is capable of providing reliable treatment decisions with embedded clinician expertise.

Artificial intelligence in perioperative medicine - a narrative review

Recent advancements in artificial intelligence (AI) techniques have enabled the development of accurate prediction models using clinical big data. AI models for perioperative risk stratification, intraoperative event prediction, biosignal analyses, and intensive care medicine have been developed in the field of perioperative medicine. Some of these models have been validated using external datasets and randomized controlled trials. Once these models are implemented in electronic health record systems or software medical devices, they could help anesthesiologists improve clinical outcomes by accurately predicting complications and suggesting optimal treatment strategies in real-time. This review provides an overview of the AI techniques used in perioperative medicine and a summary of the studies that have been published using these techniques. Understanding these techniques will aid in their appropriate application in clinical practice.