Reinforcement Learning for Clinical Decision Support in Critical Care: Comprehensive Review

Improving sepsis classification performance with artificial intelligence algorithms: A comprehensive overview of healthcare applications

PURPOSE

This study investigates the potential of machine learning (ML) algorithms in improving sepsis diagnosis and prediction, focusing on their relevance in healthcare decision-making. The primary objective is to contribute to healthcare decision-making by evaluating the performance of various supervised and unsupervised models.

MATERIALS AND METHODS

Through an extensive literature review, optimal ML models used in sepsis research were identified. Diverse datasets from relevant sources were employed, and rigorous evaluation metrics, including accuracy, specificity, and sensitivity, were applied. Innovative techniques were introduced, such as a Stacked Blended Ensemble Model and Skopt Optimization with Blended Ensemble, incorporating Bayesian optimization for hyperparameter tuning.

RESULTS

ML algorithms demonstrate efficacy in sepsis diagnosis, presenting an improved balance between specificity and sensitivity, critical for effective clinical decision-making. Classifier ensemble models show enhanced accuracy and efficiency, with novel optimization techniques contributing to improved adaptability.

CONCLUSION

The study emphasizes the potential benefits of ML algorithms in sepsis management, advocating for ongoing research to optimize performance and ensure ethical utilization in healthcare decision-making. Ethical considerations, interpretability, and transparency are crucial factors in implementing these algorithms in clinical practice.

A review of reinforcement learning for natural language processing and applications in healthcare

IMPORTANCE

Reinforcement learning (RL) represents a pivotal avenue within natural language processing (NLP), offering a potent mechanism for acquiring optimal strategies in task completion. This literature review studies various NLP applications where RL has demonstrated efficacy, with notable applications in healthcare settings.

OBJECTIVES

To systematically explore the applications of RL in NLP, focusing on its effectiveness in acquiring optimal strategies, particularly in healthcare settings, and provide a comprehensive understanding of RL's potential in NLP tasks.

MATERIALS AND METHODS

Adhering to the PRISMA guidelines, an exhaustive literature review was conducted to identify instances where RL has exhibited success in NLP applications, encompassing dialogue systems, machine translation, question-answering, text summarization, and information extraction. Our methodological approach involves closely examining the technical aspects of RL methodologies employed in these applications, analyzing algorithms, states, rewards, actions, datasets, and encoder-decoder architectures.

RESULTS

The review of 93 papers yields insights into RL algorithms, prevalent techniques, emergent trends, and the fusion of RL methods in NLP healthcare applications. It clarifies the strategic approaches employed, datasets utilized, and the dynamic terrain of RL-NLP systems, thereby offering a roadmap for research and development in RL and machine learning techniques in healthcare. The review also addresses ethical concerns to ensure equity, transparency, and accountability in the evolution and application of RL-based NLP technologies, particularly within sensitive domains such as healthcare.

DISCUSSION

The findings underscore the promising role of RL in advancing NLP applications, particularly in healthcare, where its potential to optimize decision-making and enhance patient outcomes is significant. However, the ethical challenges and technical complexities associated with RL demand careful consideration and ongoing research to ensure responsible and effective implementation.

CONCLUSIONS

By systematically exploring RL's applications in NLP and providing insights into technical analysis, ethical implications, and potential advancements, this review contributes to a deeper understanding of RL's role for language processing.

Reinforcement Learning to Optimize Ventilator Settings for Patients on Invasive Mechanical Ventilation: A Retrospective Study

BACKGROUND

One of the significant changes in intensive care medicine over the past two decades is the acknowledgement that improper mechanical ventilation settings substantially contribute to pulmonary injury in critically ill patients. Artificial intelligence (AI) solutions can be used to optimize mechanical ventilation settings in intensive care units (ICUs) and to improve patient outcomes. Specifically, machine learning algorithms can be trained on large datasets of patient information and mechanical ventilation settings. These algorithms can then predict patient responses to different ventilation strategies and suggest personalized ventilation settings for individual patients.

OBJECTIVE

In this study, we aimed to design and evaluate an AI solution that could tailor an optimal ventilator strategy for each critically ill patient who requires mechanical ventilation.

METHODS

We proposed a reinforcement learning-based AI solution using observational data from multiple ICUs in the US. The primary outcome was hospital mortality. Secondary outcomes were the proportion of optimal oxygen saturation and the proportion of optimal mean arterial blood pressure. We trained our AI agent to recommend low/medium/high levels of three ventilator settings - positive end-expiratory pressure (PEEP), fraction of inspired oxygen (FiO2) and ideal body weight-adjusted tidal volume - according to patients' health conditions. We defined a policy as a set of rules guiding ventilator setting changes given specific clinical scenarios. Off-policy evaluation metrics were applied to evaluate the AI policy.

RESULTS

We studied 21,595 and 5,105 patients' ICU stays from the eICU Collaborative Research (eICU) and Medical Information Mart for Intensive Care IV (MIMIC-IV) databases respectively. Using the learnt AI policy, we estimated the hospital mortality rate (eICU 12.1±3.1%; MIMIC-IV 29.1±0.9%), proportion of optimal oxygen saturation (eICU 58.7±4.7%; MIMIC-IV 49.0±1.0%), and proportion of optimal mean arterial blood pressure (eICU 31.1±4.5%; MIMIC-IV 41.2±1.0%). Based on multiple quantitative and qualitative evaluation metrics, our proposed AI solution outperformed observed clinical practice.We studied 21,595 5105 and 5,105 21595 patients' ' ICU stays from the Medical Information Mart for Intensive Care -IV (MIMIC-IV)(1) and eICU Collaborative Research (eICU) and Medical Information Mart for Intensive Care IV (MIMIC-IV) databases respectively. Observed hospital mortality rates were 18.2% (eICU) and 31.1% (MIMIC-IV). UsingFor the learnt AI policy, we estimated the hospital mortality rate (eICU 14.7±0.7%; MIMIC-IV 29.1±0.9%), proportion of optimal oxygen saturation (eICU 57.8±1.0%; MIMIC-IV 49.0±1.0%), and proportion of optimal mean arterial blood pressure (eICU 34.7 ± 1.0%; MIMIC-IV 41.2±1.0%). Based on multiple quantitative and qualitative evaluation metrics, our proposed AI solution has potential to outperformed observed clinical practice.

CONCLUSIONS

Our study found that customizing ventilation settings for individual patients led to lower estimated hospital mortality rates compared to actual rates. This highlights the potential effectiveness of using RL methodology to develop AI models that analyze complex clinical data for optimizing treatment parameters. Additionally, our findings suggest the integration of this model into a clinical decision support system for refining ventilation settings, supporting the need for prospective validation trials.

CLINICALTRIAL

Artificial Intelligence and Machine Learning in Cancer Pain: A Systematic Review

BACKGROUND/OBJECTIVES

Pain is a challenging multifaceted symptom reported by most cancer patients. This systematic review aims to explore applications of artificial intelligence/machine learning (AI/ML) in predicting pain-related outcomes and pain management in cancer.

METHODS

A comprehensive search of Ovid MEDLINE, EMBASE and Web of Science databases was conducted using terms: "Cancer," "Pain," "Pain Management," "Analgesics," "Artificial Intelligence," "Machine Learning," and "Neural Networks" published up to September 7, 2023. AI/ML models, their validation and performance were summarized. Quality assessment was conducted using PROBAST risk-of-bias andadherence to TRIPOD guidelines.

RESULTS

Forty four studies from 2006 to 2023 were included. Nineteen studies used AI/ML for classifying pain after cancer therapy [median AUC 0.80 (range 0.76-0.94)]. Eighteen studies focused on cancer pain research [median AUC 0.86 (range 0.50-0.99)], and 7 focused on applying AI/ML for cancer pain management, [median AUC 0.71 (range 0.47-0.89)]. Median AUC (0.77) of models across all studies. Random forest models demonstrated the highest performance (median AUC 0.81), lasso models had the highest median sensitivity (1), while Support Vector Machine had the highest median specificity (0.74). Overall adherence to TRIPOD guidelines was 70.7%. Overall, high risk-of-bias (77.3%), lack of external validation (14%) and clinical application (23%) was detected. Reporting of model calibration was also missing (5%).

CONCLUSION

Implementation of AI/ML tools promises significant advances in the classification, risk stratification, and management decisions for cancer pain. Further research focusing on quality improvement, model calibration, rigorous external clinical validation in real healthcare settings is imperative for ensuring its practical and reliable application in clinical practice.

Artificial intelligence and machine learning: Definition of terms and current concepts in critical care research

With increasing computing power, artificial intelligence (AI) and machine learning (ML) have prospered, which facilitate the analysis of large datasets, especially those found in critical care. It is important to define these terminologies, to inform a standardized approach to critical care research. This manuscript hopes to clarify these terms with examples from medical literature. Three major components that are required for a successful ML implementation: (i) reliable dataset, (ii) ML algorithm, and (iii) unbiased model evaluation, are discussed. A reliable dataset can be structured or unstructured with limited noise, outliers, and missing values. ML, a subset of AI, is typically focused on supervised or unsupervised learning tasks in which the output is based on inputs and derived from iterative pattern recognition algorithms, while AI is the overall ability of a machine to "think" or mimic human behavior; and to analyze data free from human influence. Even with successful implementation, advanced AI and ML algorithms have faced challenges in adoption into practice, mainly due to their lack of interpretability, which hinders trust, buy-in, and engagement from clinicians. Consequently, traditional algorithms, such as linear and logistic regression, that may have reduced predictive power but are highly interpretable, continue to be widely used.

Systematic literature review on reinforcement learning in non-communicable disease interventions

There is evidence that reducing modifiable risk factors and strengthening medical and health interventions can reduce early mortality and economic losses from non-communicable diseases (NCDs). Machine learning (ML) algorithms have been successfully applied to preventing and controlling NCDs. Reinforcement learning (RL) is the most promising of these approaches because of its ability to dynamically adapt interventions to NCD disease progression and its commitment to achieving long-term intervention goals. This paper reviews the preferred algorithms, data sources, design details, and obstacles to clinical application in existing studies to facilitate the early application of RL algorithms in clinical practice research for NCD interventions. We screened 40 relevant papers for quantitative and qualitative analysis using the PRISMA review flow diagram. The results show that researchers tend to use Deep Q-Network (DQN) and Actor-Critic as well as their improved or hybrid algorithms to train and validate RL models on retrospective datasets. Often, the patient's physical condition is the main defining parameter of the state space, while interventions are the main defining parameter of the action space. Mostly, changes in the patient's physical condition are used as a basis for immediate rewards to the agent. Various attempts have been made to address the challenges to clinical application, and several approaches have been proposed from existing research. However, as there is currently no universally accepted solution, the use of RL algorithms in clinical practice for NCD interventions necessitates more comprehensive responses to the issues addressed in this paper, which are safety, interpretability, training efficiency, and the technical aspect of exploitation and exploration in RL algorithms.

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Energy landscape analysis and time-series clustering analysis of patient state multistability related to rheumatoid arthritis drug treatment: The KURAMA cohort study

Rheumatoid arthritis causes joint inflammation due to immune abnormalities, resulting in joint pain and swelling. In recent years, there have been considerable advancements in the treatment of this disease. However, only approximately 60% of patients achieve remission. Patients with multifactorial diseases shift between states from day to day. Patients may remain in a good or poor state with few or no transitions, or they may switch between states frequently. The visualization of time-dependent state transitions, based on the evaluation axis of stable/unstable states, may provide useful information for achieving rheumatoid arthritis treatment goals. Energy landscape analysis can be used to quantitatively determine the stability/instability of each state in terms of energy. Time-series clustering is another method used to classify transitions into different groups to identify potential patterns within a time-series dataset. The objective of this study was to utilize energy landscape analysis and time-series clustering to evaluate multidimensional time-series data in terms of multistability. We profiled each patient's state transitions during treatment using energy landscape analysis and time-series clustering. Energy landscape analysis divided state transitions into two patterns: "good stability leading to remission" and "poor stability leading to treatment dead-end." The number of patients whose disease status improved increased markedly until approximately 6 months after treatment initiation and then plateaued after 1 year. Time-series clustering grouped patients into three clusters: "toward good stability," "toward poor stability," and "unstable." Patients in the "unstable" cluster are considered to have clinical courses that are difficult to predict; therefore, these patients should be treated with more care. Early disease detection and treatment initiation are important. The evaluation of state multistability enables us to understand a patient's current state in the context of overall state transitions related to rheumatoid arthritis drug treatment and to predict future state transitions.

Exploring the Promise and Challenges of Artificial Intelligence in Biomedical Research and Clinical Practice

ABSTRACT

Artificial intelligence (AI) is poised to revolutionize how science, and biomedical research in particular, are done. With AI, problem-solving and complex tasks using massive data sets can be performed at a much higher rate and dimensionality level compared with humans. With the ability to handle huge data sets and self-learn, AI is already being exploited in drug design, drug repurposing, toxicology, and material identification. AI could also be used in both basic and clinical research in study design, defining outcomes, analyzing data, interpreting findings, and even identifying the most appropriate areas of investigation and funding sources. State-of-the-art AI-based large language models, such as ChatGPT and Perplexity, are positioned to change forever how science is communicated and how scientists interact with one another and their profession, including postpublication appraisal and critique. Like all revolutions, upheaval will follow and not all outcomes can be predicted, necessitating guardrails at the onset, especially to minimize the untoward impact of the many drawbacks of large language models, which include lack of confidentiality, risk of hallucinations, and propagation of mainstream albeit potentially mistaken opinions and perspectives. In this review, we highlight areas of biomedical research that are already being reshaped by AI and how AI is likely to affect it further in the near future. We discuss the potential benefits of AI in biomedical research and address possible risks, some surrounding the creative process, that warrant further reflection.

Integrating clinical pharmacology and artificial intelligence: potential benefits, challenges, and role of clinical pharmacologists

INTRODUCTION

The integration of artificial intelligence (AI) into clinical pharmacology could be a potential approach for accelerating drug discovery and development, improving patient care, and streamlining medical research processes.

AREAS COVERED

We reviewed the current state of AI applications in clinical pharmacology, focusing on drug discovery and development, precision medicine, pharmacovigilance, and other ventures. Key AI applications in clinical pharmacology are examined, including machine learning, natural language processing, deep learning, and reinforcement learning etc. Additionally, the evolving role of clinical pharmacologists, ethical considerations, and challenges in implementing AI in clinical pharmacology are discussed.

EXPERT OPINION

The AI could be instrumental in accelerating drug discovery, predicting drug safety and efficacy, and optimizing clinical trial designs. It can play a vital role in precision medicine by helping in personalized drug dosing, treatment selection, and predicting drug response based on genetic, clinical, and environmental factors. The role of AI in pharmacovigilance, such as signal detection and adverse event prediction, is also promising. The collaboration between clinical pharmacologists and AI experts also poses certain ethical and practical challenges. Clinical pharmacologists can be instrumental in shaping the future of AI-driven clinical pharmacology and contribute to the improvement of healthcare systems.

Reinforcement Learning and PK-PD Models Integration to Personalize the Adaptive Dosing Protocol of Erdafitinib in Patients with Metastatic Urothelial Carcinoma

The integration of pharmacokinetic-pharmacodynamic (PK-PD) modeling and simulations with artificial intelligence/machine learning algorithms is one of the most attractive areas of the pharmacometric research. These hybrid techniques are currently under investigation to perform several tasks, among which precision dosing. In this scenario, this paper presents and evaluates a new framework embedding PK-PD models into a reinforcement learning (RL) algorithm, Q-learning (QL), to personalize pharmacological treatment. Each patient is represented with a set of PK-PD parameters and has a personal QL agent which optimizes the individual treatment. In the training phase, leveraging PK-PD simulations, the QL agent assesses different actions, defined consistently with the clinical knowledge to consider only plausible dose-adjustments, in order to find the optimal rules. The proposed framework was evaluated to optimize the erdafitinib treatment in patients with metastatic urothelial carcinoma. This drug was approved by the US Food and Drug Administration (FDA) with a dose-adaptive protocol based on monitoring the levels of serum phosphate, which represent a biomarker of both treatment efficacy and toxicity. To evaluate the flexibility of the methodology, a heterogeneous virtual population of 141 patients was generated using an erdafitinib population PK (PopPK)-PD literature model. For each patient, treatment response was simulated by using both QL-optimized protocol and the clinical one. QL agents outperform the approved dose-adaptive rules, increasing more than 10% the efficacy and the safety of treatment at each end point. Results confirm the great potentialities of the integration of PopPK-PD models and RL algorithms to optimize precision dosing tasks.

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

OBJECTIVE

Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients.

DATA SOURCES

A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking.

DATA EXTRACTION

Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician's policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed.

DATA SYNTHESIS

A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure.

CONCLUSION

In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation.

Development and Validation of a Decision Tree Analysis Model for Predicting Home Discharge in a Convalescent Ward: A Single Institution Study

OBJECTIVES

Accurately predicting the likelihood of inpatients' home discharge in a convalescent ward is crucial for assisting patients and families in decision-making. While logistic regression analysis has been commonly used, its complexity limits practicality in clinical settings. We focused on decision tree analysis, which is visually straightforward. This study aimed to develop and validate the accuracy of a prediction model for home discharge for inpatients in a convalescent ward using a decision tree analysis.

METHODS

The cohort consisted of 651 patients admitted to our convalescent ward from 2018 to 2020. We collected data from medical records, including disease classification, sex, age, duration of acute hospitalization, discharge destination (home or nonhome), and Functional Independence Measure (FIM) subitems at admission. We divided the cohort data into training and validation sets and developed a prediction model using decision tree analysis with discharge destination as the target and other variables as predictors. The model's accuracy was validated using the validation data set.

RESULTS

The decision tree model identified FIM grooming as the first single discriminator of home discharge, diverging at four points and identifying subsequent branching for the duration of acute hospitalization. The model's accuracy was 86.7%, with a sensitivity of 0.96, specificity of 0.52, positive predictive accuracy of 0.88, and negative predictive accuracy of 0.80. The area under the receiver operating characteristic curve was 0.75.

CONCLUSION

The predictive model demonstrated more than moderate predictive accuracy, suggesting its utility in clinical practice. Grooming emerged as a variable with the highest explanatory power for determining home discharge.

The use of probabilistic graphical models in pediatric sepsis: a feasibility and scoping review

Background

Recent research has demonstrated that machine learning (ML) has the potential to improve several aspects of medical application for critical illness, including sepsis. This scoping review aims to evaluate the feasibility of probabilistic graphical model (PGM) methods in pediatric sepsis application and describe the use of pediatric sepsis definition in these studies.

Methods

Literature searches were conducted in PubMed, Scopus, Cumulative Index to Nursing and Allied Health Literature (CINAHL+), and Web of Sciences from 2000-2023. Keywords included "pediatric", "neonates", "infants", "machine learning", "probabilistic graphical model", and "sepsis".

Results

A total of 3,244 studies were screened, and 72 were included in this scoping review. Sepsis was defined using positive microbiology cultures in 19 studies (26.4%), followed by the 2005's international pediatric sepsis consensus definition in 11 studies (15.3%), and Sepsis-3 definition in seven studies (9.7%). Other sepsis definitions included: bacterial infection, the international classification of diseases, clinicians' assessment, and antibiotic administration time. Among the most common ML approaches used were logistic regression (n=27), random forest (n=24), and Neural Network (n=18). PGMs were used in 13 studies (18.1%), including Bayesian classifiers (n=10), and the Markov Model (n=3). When applied on the same dataset, PGMs show a relatively inferior performance to other ML models in most cases. Other aspects of explainability and transparency were not examined in these studies.

Conclusions

Current studies suggest that the performance of probabilistic graphic models is relatively inferior to other ML methods. However, its explainability and transparency advantages make it a potentially viable method for several pediatric sepsis studies and applications.

Causal inference using observational intensive care unit data: a scoping review and recommendations for future practice

This scoping review focuses on the essential role of models for causal inference in shaping actionable artificial intelligence (AI) designed to aid clinicians in decision-making. The objective was to identify and evaluate the reporting quality of studies introducing models for causal inference in intensive care units (ICUs), and to provide recommendations to improve the future landscape of research practices in this domain. To achieve this, we searched various databases including Embase, MEDLINE ALL, Web of Science Core Collection, Google Scholar, medRxiv, bioRxiv, arXiv, and the ACM Digital Library. Studies involving models for causal inference addressing time-varying treatments in the adult ICU were reviewed. Data extraction encompassed the study settings and methodologies applied. Furthermore, we assessed reporting quality of target trial components (i.e., eligibility criteria, treatment strategies, follow-up period, outcome, and analysis plan) and main causal assumptions (i.e., conditional exchangeability, positivity, and consistency). Among the 2184 titles screened, 79 studies met the inclusion criteria. The methodologies used were G methods (61%) and reinforcement learning methods (39%). Studies considered both static (51%) and dynamic treatment regimes (49%). Only 30 (38%) of the studies reported all five target trial components, and only seven (9%) studies mentioned all three causal assumptions. To achieve actionable AI in the ICU, we advocate careful consideration of the causal question of interest, describing this research question as a target trial emulation, usage of appropriate causal inference methods, and acknowledgement (and examination of potential violations of) the causal assumptions.

Implementing Artificial Intelligence: Assessing the Cost and Benefits of Algorithmic Decision-Making in Critical Care

This article provides an overview of the most useful artificial intelligence algorithms developed in critical care, followed by a comprehensive outline of the benefits and limitations. We begin by describing how nurses and physicians might be aided by these new technologies. We then move to the possible changes in clinical guidelines with personalized medicine that will allow tailored therapies and probably will increase the quality of the care provided to patients. Finally, we describe how artificial intelligence models can unleash researchers' minds by proposing new strategies, by increasing the quality of clinical practice, and by questioning current knowledge and understanding.

AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges

Aim

Artificial intelligence (AI) and machine learning (ML) are important areas of computer science that have recently attracted attention for their application to medicine. However, as techniques continue to advance and become more complex, it is increasingly challenging for clinicians to stay abreast of the latest research. This overview aims to translate research concepts and potential concerns to healthcare professionals interested in applying AI and ML to resuscitation research but who are not experts in the field.

Main text

We present various research including prediction models using structured and unstructured data, exploring treatment heterogeneity, reinforcement learning, language processing, and large-scale language models. These studies potentially offer valuable insights for optimizing treatment strategies and clinical workflows. However, implementing AI and ML in clinical settings presents its own set of challenges. The availability of high-quality and reliable data is crucial for developing accurate ML models. A rigorous validation process and the integration of ML into clinical practice is essential for practical implementation. We furthermore highlight the potential risks associated with self-fulfilling prophecies and feedback loops, emphasizing the importance of transparency, interpretability, and trustworthiness in AI and ML models. These issues need to be addressed in order to establish reliable and trustworthy AI and ML models.

Conclusion

In this article, we overview concepts and examples of AI and ML research in the resuscitation field. Moving forward, appropriate understanding of ML and collaboration with relevant experts will be essential for researchers and clinicians to overcome the challenges and harness the full potential of AI and ML in resuscitation.

The role of artificial intelligence in hypertensive disorders of pregnancy: towards personalized healthcare

INTRODUCTION

Guidelines advise ongoing follow-up of patients after hypertensive disorders of pregnancy (HDP) to assess cardiovascular risk and manage future patient-specific pregnancy conditions. However, there are limited tools available to monitor patients, with those available tending to be simple risk assessments that lack personalization. A promising approach could be the emerging artificial intelligence (AI)-based techniques, developed from big patient datasets to provide personalized recommendations for preventive advice.

AREAS COVERED

In this narrative review, we discuss the impact of integrating AI and big data analysis for personalized cardiovascular care, focusing on the management of HDP.

EXPERT OPINION

The pathophysiological response of women to pregnancy varies, and deeper insight into each response can be gained through a deeper analysis of the medical history of pregnant women based on clinical records and imaging data. Further research is required to be able to implement AI for clinical cases using multi-modality and multi-organ assessment, and this could expand both knowledge on pregnancy-related disorders and personalized treatment planning.

Bottom-up and top-down paradigms of artificial intelligence research approaches to healthcare data science using growing real-world big data

OBJECTIVES

As the real-world electronic health record (EHR) data continue to grow exponentially, novel methodologies involving artificial intelligence (AI) are becoming increasingly applied to enable efficient data-driven learning and, ultimately, to advance healthcare. Our objective is to provide readers with an understanding of evolving computational methods and help in deciding on methods to pursue.

TARGET AUDIENCE

The sheer diversity of existing methods presents a challenge for health scientists who are beginning to apply computational methods to their research. Therefore, this tutorial is aimed at scientists working with EHR data who are early entrants into the field of applying AI methodologies.

SCOPE

This manuscript describes the diverse and growing AI research approaches in healthcare data science and categorizes them into 2 distinct paradigms, the bottom-up and top-down paradigms to provide health scientists venturing into artificial intelligent research with an understanding of the evolving computational methods and help in deciding on methods to pursue through the lens of real-world healthcare data.

Development of a Reinforcement Learning Algorithm to Optimize Corticosteroid Therapy in Critically Ill Patients with Sepsis

BACKGROUND

The optimal indication, dose, and timing of corticosteroids in sepsis is controversial. Here, we used reinforcement learning to derive the optimal steroid policy in septic patients based on data on 3051 ICU admissions from the AmsterdamUMCdb intensive care database.

METHODS

We identified septic patients according to the 2016 consensus definition. An actor-critic RL algorithm using ICU mortality as a reward signal was developed to determine the optimal treatment policy from time-series data on 277 clinical parameters. We performed off-policy evaluation and testing in independent subsets to assess the algorithm's performance.

RESULTS

Agreement between the RL agent's policy and the actual documented treatment reached 59%. Our RL agent's treatment policy was more restrictive compared to the actual clinician behavior: our algorithm suggested withholding corticosteroids in 62% of the patient states, versus 52% according to the physicians' policy. The 95% lower bound of the expected reward was higher for the RL agent than clinicians' historical decisions. ICU mortality after concordant action in the testing dataset was lower both when corticosteroids had been withheld and when corticosteroids had been prescribed by the virtual agent. The most relevant variables were vital parameters and laboratory values, such as blood pressure, heart rate, leucocyte count, and glycemia.

CONCLUSIONS

Individualized use of corticosteroids in sepsis may result in a mortality benefit, but optimal treatment policy may be more restrictive than the routine clinical practice. Whilst external validation is needed, our study motivates a 'precision-medicine' approach to future prospective controlled trials and practice.

A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis

Deep Reinforcement Learning (DRL) has been increasingly attempted in assisting clinicians for real-time treatment of sepsis. While a value function quantifies the performance of policies in such decision-making processes, most value-based DRL algorithms cannot evaluate the target value function precisely and are not as safe as clinical experts. In this study, we propose a Weighted Dueling Double Deep Q-Network with embedded human Expertise (WD3QNE). A target Q value function with adaptive dynamic weight is designed to improve the estimate accuracy and human expertise in decision-making is leveraged. In addition, the random forest algorithm is employed for feature selection to improve model interpretability. We test our algorithm against state-of-the-art value function methods in terms of expected return, survival rate, action distribution and external validation. The results demonstrate that WD3QNE obtains the highest survival rate of 97.81% in MIMIC-III dataset. Our proposed method is capable of providing reliable treatment decisions with embedded clinician expertise.

An Insight of Deep Learning Based Demand Forecasting in Smart Grids

Smart grids are able to forecast customers' consumption patterns, i.e., their energy demand, and consequently electricity can be transmitted after taking into account the expected demand. To face today's demand forecasting challenges, where the data generated by smart grids is huge, modern data-driven techniques need to be used. In this scenario, Deep Learning models are a good alternative to learn patterns from customer data and then forecast demand for different forecasting horizons. Among the commonly used Artificial Neural Networks, Long Short-Term Memory networks-based on Recurrent Neural Networks-are playing a prominent role. This paper provides an insight into the importance of the demand forecasting issue, and other related factors, in the context of smart grids, and collects some experiences of the use of Deep Learning techniques, for demand forecasting purposes. To have an efficient power system, a balance between supply and demand is necessary. Therefore, industry stakeholders and researchers should make a special effort in load forecasting, especially in the short term, which is critical for demand response.

Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Toward automatic motivator selection for autism behavior intervention therapy

Children with autism spectrum disorder (ASD) usually show little interest in academic activities and may display disruptive behavior when presented with assignments. Research indicates that incorporating motivational variables during interventions results in improvements in behavior and academic performance. However, the impact of such motivational variables varies between children. In this paper, we aim to address the problem of selecting the right motivator for children with ASD using reinforcement learning by adapting to the most influential factors impacting the effectiveness of the contingent motivator used. We model the task of selecting a motivator as a Markov decision process problem. The states, actions and rewards design consider the factors that impact the effectiveness of a motivator based on applied behavior analysis as well as learners' individual preferences. We use a Q-learning algorithm to solve the modeled problem. Our proposed solution is then implemented as a mobile application developed for special education plans coordination. To evaluate the motivator selection feature, we conduct a study involving a group of teachers and therapists and assess how the added feature aids the participants in their decision-making process of selecting a motivator. Preliminary results indicated that the motivator selection feature improved the usability of the mobile app. Analysis of the algorithm performance showed promising results and indicated improvement of the recommendations over time.

Deep reinforcement learning for personalized treatment recommendation

In precision medicine, the ultimate goal is to recommend the most effective treatment to an individual patient based on patient-specific molecular and clinical profiles, possibly high-dimensional. To advance cancer treatment, large-scale screenings of cancer cell lines against chemical compounds have been performed to help better understand the relationship between genomic features and drug response; existing machine learning approaches use exclusively supervised learning, including penalized regression and recommender systems. However, it would be more efficient to apply reinforcement learning to sequentially learn as data accrue, including selecting the most promising therapy for a patient given individual molecular and clinical features and then collecting and learning from the corresponding data. In this article, we propose a novel personalized ranking system called Proximal Policy Optimization Ranking (PPORank), which ranks the drugs based on their predicted effects per cell line (or patient) in the framework of deep reinforcement learning (DRL). Modeled as a Markov decision process, the proposed method learns to recommend the most suitable drugs sequentially and continuously over time. As a proof-of-concept, we conduct experiments on two large-scale cancer cell line data sets in addition to simulated data. The results demonstrate that the proposed DRL-based PPORank outperforms the state-of-the-art competitors based on supervised learning. Taken together, we conclude that novel methods in the framework of DRL have great potential for precision medicine and should be further studied.

Machine Learning for Acute Kidney Injury Prediction in the Intensive Care Unit

Machine learning is the field of artificial intelligence in which computers are trained to make predictions or to identify patterns in data through complex mathematical algorithms. It has great potential in critical care to predict outcomes, such as acute kidney injury, and can be used for prognosis and to suggest management strategies. Machine learning can also be used as a research tool to advance our clinical and biochemical understanding of acute kidney injury. In this review, we introduce basic concepts in machine learning and review recent research in each of these domains.

Challenges in translational machine learning

Machine learning (ML) algorithms are increasingly being used to help implement clinical decision support systems. In this new field, we define as "translational machine learning", joint efforts and strong communication between data scientists and clinicians help to span the gap between ML and its adoption in the clinic. These collaborations also improve interpretability and trust in translational ML methods and ultimately aim to result in generalizable and reproducible models. To help clinicians and bioinformaticians refine their translational ML pipelines, we review the steps from model building to the use of ML in the clinic. We discuss experimental setup, computational analysis, interpretability and reproducibility, and emphasize the challenges involved. We highly advise collaboration and data sharing between consortia and institutes to build multi-centric cohorts that facilitate ML methodologies that generalize across centers. In the end, we hope that this review provides a way to streamline translational ML and helps to tackle the challenges that come with it.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

A Reinforcement Learning Application for Optimal Fluid and Vasopressor Interventions in Septic ICU Patients

Sepsis is one of the leading causes of death in ICU and its timely recognition and management are of primary importance. Resuscitation from hypotension in patients with sepsis is one of the first challenges that require fluid and/or vasopressor administrations. Unfortunately, clinical guidelines provide only indications of the strategy that should be adopted in this critical population but personalized strategies are still missing. In this study, we propose a comparative analysis of reinforcement learning applications on ICU data collected in the electronic health records and publicly available within the MIMIC-III database. The ultimate goal of the study is to estimate the optimal fluid and vasopressor administrations. Results show that, after the use of principal component analysis for reducing feature space dimensionality, model performances increased, thus suggesting that additional preprocessing strategies might be used for both reducing the computational cost and refining model performances. Clinical relevance In a context where clinical guidelines are not able to provide the best treatment strategies at a patient level, reinforcement learning applications trained on retrospectively collected data may be used for developing models able to suggest to clinicians the optimal dosage of fluids and/or vasopressors in order to improve 90-day patients' survival.

Supervised Optimal Chemotherapy Regimen Based on Offline Reinforcement Learning

In recent years, reinforcement learning (RL) has achieved a remarkable achievement and it has attracted researchers' attention in modeling real-life scenarios by expanding its research beyond conventional complex games. Prediction of optimal treatment regimens from observational real clinical data is being popularized, and more advanced versions of RL algorithms are being implemented in the literature. However, RL-generated medications still need careful supervision of expertise parties or doctors in healthcare. Hence, in this paper, a Supervised Optimal Chemotherapy Regimen (SOCR) approach to investigate optimal chemotherapy-dosing schedule for cancer patients was presented by using Offline Reinforcement Learning. The optimal policy suggested by the RL approach was supervised by incorporating previous treatment decisions of oncologists, which could add clinical expertise knowledge on algorithmic results. Presented SOCR approach followed a model-based architecture using conservative Q-Learning (CQL) algorithm. The developed model was tested using a manually constructed database of forty Stage-IV colon cancer patients, receiving line-1 chemotherapy treatments, who were clinically classified as 'Bevacizumab based patient' and 'Cetuximab based patient'. Experimental results revealed that the supervision from the oncologists has considered the effect to stabilize chemotherapy regimen and it was suggested that the proposed framework could be successfully used as a supportive model for oncologists in deciding their treatment decisions.

Machine learning methods for functional recovery prediction and prognosis in post-stroke rehabilitation: a systematic review

BACKGROUND

Rehabilitation medicine is facing a new development phase thanks to a recent wave of rigorous clinical trials aimed at improving the scientific evidence of protocols. This phenomenon, combined with new trends in personalised medical therapies, is expected to change clinical practice dramatically. The emerging field of Rehabilomics is only possible if methodologies are based on biomedical data collection and analysis. In this framework, the objective of this work is to develop a systematic review of machine learning algorithms as solutions to predict motor functional recovery of post-stroke patients after treatment.

METHODS

We conducted a comprehensive search of five electronic databases using the Patient, Intervention, Comparison and Outcome (PICO) format. We extracted health conditions, population characteristics, outcome assessed, the method for feature extraction and selection, the algorithm used, and the validation approach. The methodological quality of included studies was assessed using the prediction model risk of bias assessment tool (PROBAST). A qualitative description of the characteristics of the included studies as well as a narrative data synthesis was performed.

RESULTS

A total of 19 primary studies were included. The predictors most frequently used belonged to the areas of demographic characteristics and stroke assessment through clinical examination. Regarding the methods, linear and logistic regressions were the most frequently used and cross-validation was the preferred validation approach.

CONCLUSIONS

We identified several methodological limitations: small sample sizes, a limited number of external validation approaches, and high heterogeneity among input and output variables. Although these elements prevented a quantitative comparison across models, we defined the most frequently used models given a specific outcome, providing useful indications for the application of more complex machine learning algorithms in rehabilitation medicine.

A clinician's guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML)

Developing functional machine learning (ML)-based models to address unmet clinical needs requires unique considerations for optimal clinical utility. Recent debates about the rigours, transparency, explainability, and reproducibility of ML models, terms which are defined in this article, have raised concerns about their clinical utility and suitability for integration in current evidence-based practice paradigms. This featured article focuses on increasing the literacy of ML among clinicians by providing them with the knowledge and tools needed to understand and critically appraise clinical studies focused on ML. A checklist is provided for evaluating the rigour and reproducibility of the four ML building blocks: data curation, feature engineering, model development, and clinical deployment. Checklists like this are important for quality assurance and to ensure that ML studies are rigourously and confidently reviewed by clinicians and are guided by domain knowledge of the setting in which the findings will be applied. Bridging the gap between clinicians, healthcare scientists, and ML engineers can address many shortcomings and pitfalls of ML-based solutions and their potential deployment at the bedside.

Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare

Machine learning (ML) and artificial intelligence (AI) algorithms have the potential to derive insights from clinical data and improve patient outcomes. However, these highly complex systems are sensitive to changes in the environment and liable to performance decay. Even after their successful integration into clinical practice, ML/AI algorithms should be continuously monitored and updated to ensure their long-term safety and effectiveness. To bring AI into maturity in clinical care, we advocate for the creation of hospital units responsible for quality assurance and improvement of these algorithms, which we refer to as “AI-QI” units. We discuss how tools that have long been used in hospital quality assurance and quality improvement can be adapted to monitor static ML algorithms. On the other hand, procedures for continual model updating are still nascent. We highlight key considerations when choosing between existing methods and opportunities for methodological innovation.

VacSIM: Learning effective strategies for COVID-19 vaccine distribution using reinforcement learning

A COVID-19 vaccine is our best bet for mitigating the ongoing onslaught of the pandemic. However, vaccine is also expected to be a limited resource. An optimal allocation strategy, especially in countries with access inequities and temporal separation of hot-spots, might be an effective way of halting the disease spread. We approach this problem by proposing a novel pipeline VacSIM that dovetails Deep Reinforcement Learning models into a Contextual Bandits approach for optimizing the distribution of COVID-19 vaccine. Whereas the Reinforcement Learning models suggest better actions and rewards, Contextual Bandits allow online modifications that may need to be implemented on a day-to-day basis in the real world scenario. We evaluate this framework against a naive allocation approach of distributing vaccine proportional to the incidence of COVID-19 cases in five different States across India (Assam, Delhi, Jharkhand, Maharashtra and Nagaland) and demonstrate up to 9039 potential infections prevented and a significant increase in the efficacy of limiting the spread over a period of 45 days through the VacSIM approach. Our models and the platform are extensible to all states of India and potentially across the globe. We also propose novel evaluation strategies including standard compartmental model-based projections and a causality-preserving evaluation of our model. Since all models carry assumptions that may need to be tested in various contexts, we open source our model VacSIM and contribute a new reinforcement learning environment compatible with OpenAI gym to make it extensible for real-world applications across the globe. 2

AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening

Advances in artificial intelligence in healthcare are frequently promoted as ‘solutions’ to improve the accuracy, safety, and quality of clinical decisions, treatments, and care. Despite some diagnostic success, however, AI systems rely on forms of reductive reasoning and computational determinism that embed problematic assumptions about clinical decision-making and clinical practice. Clinician autonomy, experience, and judgement are reduced to inputs and outputs framed as binary or multi-class classification problems benchmarked against a clinician’s capacity to identify or predict disease states. This paper examines this reductive reasoning in AI systems for colorectal cancer (CRC) to highlight their limitations and risks: (1) in AI systems themselves due to inherent biases in (a) retrospective training datasets and (b) embedded assumptions in underlying AI architectures and algorithms; (2) in the problematic and limited evaluations being conducted on AI systems prior to system integration in clinical practice; and (3) in marginalising socio-technical factors in the context-dependent interactions between clinicians, their patients, and the broader health system. The paper argues that to optimise benefits from AI systems and to avoid negative unintended consequences for clinical decision-making and patient care, there is a need for more nuanced and balanced approaches to AI system deployment and evaluation in CRC.

Reinforcement learning evaluation of treatment policies for patients with hepatitis C virus

BACKGROUND

Evaluation of new treatment policies is often costly and challenging in complex conditions, such as hepatitis C virus (HCV) treatment, or in limited-resource settings. We sought to identify hypothetical policies for HCV treatment that could best balance the prevention of cirrhosis while preserving resources (financial or otherwise).

METHODS

The cohort consisted of 3792 HCV-infected patients without a history of cirrhosis or hepatocellular carcinoma at baseline from the national Veterans Health Administration from 2015 to 2019. To estimate the efficacy of hypothetical treatment policies, we utilized historical data and reinforcement learning to allow for greater flexibility when constructing new HCV treatment strategies. We tested and compared four new treatment policies: a simple stepwise policy based on Aspartate Aminotransferase to Platelet Ratio Index (APRI), a logistic regression based on APRI, a logistic regression on multiple longitudinal and demographic indicators that were prespecified for clinical significance, and a treatment policy based on a risk model developed for HCV infection.

RESULTS

The risk-based hypothetical treatment policy achieved the lowest overall risk with a score of 0.016 (90% CI 0.016, 0.019) while treating the most high-risk (346.4 ± 1.4) and the fewest low-risk (361.0 ± 20.1) patients. Compared to hypothetical treatment policies that treated approximately the same number of patients (1843.7 vs. 1914.4 patients), the risk-based policy had more untreated time per patient (7968.4 vs. 7742.9 patient visits), signaling cost reduction for the healthcare system.

CONCLUSIONS

Off-policy evaluation strategies are useful to evaluate hypothetical treatment policies without implementation. If a quality risk model is available, risk-based treatment strategies can reduce overall risk and prioritize patients while reducing healthcare system costs.

Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment

Sepsis is a potentially life-threatening inflammatory response to infection or severe tissue damage. It has a highly variable clinical course, requiring constant monitoring of the patient's state to guide the management of intravenous fluids and vasopressors, among other interventions. Despite decades of research, there's still debate among experts on optimal treatment. Here, we combine for the first time, distributional deep reinforcement learning with mechanistic physiological models to find personalized sepsis treatment strategies. Our method handles partial observability by leveraging known cardiovascular physiology, introducing a novel physiology-driven recurrent autoencoder, and quantifies the uncertainty of its own results. Moreover, we introduce a framework for uncertainty-aware decision support with humans in the loop. We show that our method learns physiologically explainable, robust policies, that are consistent with clinical knowledge. Further our method consistently identifies high-risk states that lead to death, which could potentially benefit from more frequent vasopressor administration, providing valuable guidance for future research.

Learning dynamic treatment strategies for coronary heart diseases by artificial intelligence: real-world data-driven study

BACKGROUND

Coronary heart disease (CHD) has become the leading cause of death and one of the most serious epidemic diseases worldwide. CHD is characterized by urgency, danger and severity, and dynamic treatment strategies for CHD patients are needed. We aimed to build and validate an AI model for dynamic treatment recommendations for CHD patients with the goal of improving patient outcomes and learning best practices from clinicians to help clinical decision support for treating CHD patients.

METHODS

We formed the treatment strategy as a sequential decision problem, and applied an AI supervised reinforcement learning-long short-term memory (SRL-LSTM) framework that combined supervised learning (SL) and reinforcement learning (RL) with an LSTM network to track patients' states to learn a recommendation model that took a patient's diagnosis and evolving health status as input and provided a treatment recommendation in the form of whether to take specific drugs. The experiments were conducted by leveraging a real-world intensive care unit (ICU) database with 13,762 admitted patients diagnosed with CHD. We compared the performance of the applied SRL-LSTM model and several state-of-the-art SL and RL models in reducing the estimated in-hospital mortality and the Jaccard similarity with clinicians' decisions. We used a random forest algorithm to calculate the feature importance of both the clinician policy and the AI policy to illustrate the interpretability of the AI model.

RESULTS

Our experimental study demonstrated that the AI model could help reduce the estimated in-hospital mortality through its RL function and learn the best practice from clinicians through its SL function. The similarity between the clinician policy and the AI policy regarding the surviving patients was high, while for the expired patients, it was much lower. The dynamic treatment strategies made by the AI model were clinically interpretable and relied on sensible clinical features extracted according to monitoring indexes and risk factors for CHD patients.

CONCLUSIONS

We proposed a pipeline for constructing an AI model to learn dynamic treatment strategies for CHD patients that could improve patient outcomes and mimic the best practices of clinicians. And a lot of further studies and efforts are needed to make it practical.

Predicting SARS-CoV-2 infection duration at hospital admission:a deep learning solution

COVID-19 cases are increasing around the globe with almost 5 million of deaths. We propose here a deep learning model capable of predicting the duration of the infection by means of information available at hospital admission. A total of 222 patients were enrolled in our observational study. Anagraphical and anamnestic data, COVID-19 signs and symptoms, COVID-19 therapy, hematochemical test results, and prior therapies administered to patients are used as predictors. A set of 55 features, all of which can be taken in the first hours of the patient's hospitalization, was considered. Different solutions were compared achieving the best performance with a sequential convolutional neural network-based model merged in an ensemble with two different meta-learners linked in cascade. We obtained a median absolute error of 2.7 days (IQR = 3.0) in predicting the duration of the infection; the error was equally distributed in the infection duration range. This tool could preemptively give an outlook of the COVID-19 patients' expected path and the associated hospitalization effort. The proposed solution could be viable in tackling the huge burden and the logistics complexity of hospitals or rehabilitation centers during the pandemic waves. With data taken ad admission, entering a PCA-based feature selection, a k-fold cross-validated CNN-based model was implemented. After external texting, a median absolute error of 2.7 days [IQR = 3 days].

Enhancing Diagnosis Through Technology: Decision Support, Artificial Intelligence, and Beyond

Patient care in intensive care environments is complex, time-sensitive, and data-rich, factors that make these settings particularly well-suited to clinical decision support (CDS). A wide range of CDS interventions have been used in intensive care unit environments. The field needs well-designed studies to identify the most effective CDS approaches. Evolving artificial intelligence and machine learning models may reduce information-overload and enable teams to take better advantage of the large volume of patient data available to them. It is vital to effectively integrate new CDS into clinical workflows and to align closely with the cognitive processes of frontline clinicians.

Reinforcement learning assisted oxygen therapy for COVID-19 patients under intensive care

BACKGROUND

Patients with severe Coronavirus disease 19 (COVID-19) typically require supplemental oxygen as an essential treatment. We developed a machine learning algorithm, based on deep Reinforcement Learning (RL), for continuous management of oxygen flow rate for critically ill patients under intensive care, which can identify the optimal personalized oxygen flow rate with strong potentials to reduce mortality rate relative to the current clinical practice.

METHODS

We modeled the oxygen flow trajectory of COVID-19 patients and their health outcomes as a Markov decision process. Based on individual patient characteristics and health status, an optimal oxygen control policy is learned by using deep deterministic policy gradient (DDPG) and real-time recommends the oxygen flow rate to reduce the mortality rate. We assessed the performance of proposed methods through cross validation by using a retrospective cohort of 1372 critically ill patients with COVID-19 from New York University Langone Health ambulatory care with electronic health records from April 2020 to January 2021.

RESULTS

The mean mortality rate under the RL algorithm is lower than the standard of care by 2.57% (95% CI: 2.08-3.06) reduction (P < 0.001) from 7.94% under the standard of care to 5.37% under our proposed algorithm. The averaged recommended oxygen flow rate is 1.28 L/min (95% CI: 1.14-1.42) lower than the rate delivered to patients. Thus, the RL algorithm could potentially lead to better intensive care treatment that can reduce the mortality rate, while saving the oxygen scarce resources. It can reduce the oxygen shortage issue and improve public health during the COVID-19 pandemic.

CONCLUSIONS

A personalized reinforcement learning oxygen flow control algorithm for COVID-19 patients under intensive care showed a substantial reduction in 7-day mortality rate as compared to the standard of care. In the overall cross validation cohort independent of the training data, mortality was lowest in patients for whom intensivists' actual flow rate matched the RL decisions.

Overcoming Challenges of Applying Reinforcement Learning for Intelligent Vehicle Control

Reinforcement learning (RL) is a booming area in artificial intelligence. The applications of RL are endless nowadays, ranging from fields such as medicine or finance to manufacturing or the gaming industry. Although multiple works argue that RL can be key to a great part of intelligent vehicle control related problems, there are many practical problems that need to be addressed, such as safety related problems that can result from non-optimal training in RL. For instance, for an RL agent to be effective it should first cover all the situations during training that it may face later. This is often difficult when applied to the real-world. In this work we investigate the impact of RL applied to the context of intelligent vehicle control. We analyse the implications of RL in path planning tasks and we discuss two possible approaches to overcome the gap between the theorical developments of RL and its practical applications. Specifically, firstly this paper discusses the role of Curriculum Learning (CL) to structure the learning process of intelligent vehicle control in a gradual way. The results show how CL can play an important role in training agents in such context. Secondly, we discuss a method of transferring RL policies from simulation to reality in order to make the agent experience situations in simulation, so it knows how to react to them in reality. For that, we use Arduino Yún controlled robots as our platforms. The results enhance the effectiveness of the presented approach and show how RL policies can be transferred from simulation to reality even when the platforms are resource limited.

Systematic review of machine learning models for personalised dosing of heparin

AIM

To identify and critically appraise studies of prediction models, developed using machine learning (ML) methods, for determining the optimal dosing of unfractionated heparin (UFH).

METHODS

Embase, PubMed, CINAHL, Web of Science, International Pharmaceutical Abstracts and IEEE Xplore databases were searched from inception to 31 January 2020 to identify relevant studies using key search terms synonymous with artificial intelligence or ML, 'prediction', 'dose', 'activated partial thromboplastin time (aPTT)' and 'UFH.' Studies had to have used ML methods for developing models that predicted optimal dose of UFH or target therapeutic aPTT levels in the hospital setting. The CHARMS Checklist was used to assess quality and risk of bias of included studies.

RESULTS

Of 8393 retrieved abstracts, 61 underwent full text review and eight studies met inclusion criteria. Four studies described models for predicting aPTT, three studies described models predicting optimal dose of heparin during dialysis and one study described a model that used surrogate outcomes of clotting and bleeding to predict a therapeutic aPTT. Studies varied widely in reporting of study participants, feature characterisation and selection, handling of missing data, sample size calculations and the intended clinical application of the model. Only one study conducted an external validation and no studies evaluated model impacts in clinical practice.

CONCLUSION

Studies of ML models for UFH dosing are few and none report a model ready for routine clinical use. Existing studies are limited by low methodological quality, inadequate reporting of study factors and absence of external validation and impact analysis.

Accessing Artificial Intelligence for Clinical Decision-Making

Advancements in computing and data from the near universal acceptance and implementation of electronic health records has been formative for the growth of personalized, automated, and immediate patient care models that were not previously possible. Artificial intelligence (AI) and its subfields of machine learning, reinforcement learning, and deep learning are well-suited to deal with such data. The authors in this paper review current applications of AI in clinical medicine and discuss the most likely future contributions that AI will provide to the healthcare industry. For instance, in response to the need to risk stratify patients, appropriately cultivated and curated data can assist decision-makers in stratifying preoperative patients into risk categories, as well as categorizing the severity of ailments and health for non-operative patients admitted to hospitals. Previous overt, traditional vital signs and laboratory values that are used to signal alarms for an acutely decompensating patient may be replaced by continuously monitoring and updating AI tools that can pick up early imperceptible patterns predicting subtle health deterioration. Furthermore, AI may help overcome challenges with multiple outcome optimization limitations or sequential decision-making protocols that limit individualized patient care. Despite these tremendously helpful advancements, the data sets that AI models train on and develop have the potential for misapplication and thereby create concerns for application bias. Subsequently, the mechanisms governing this disruptive innovation must be understood by clinical decision-makers to prevent unnecessary harm. This need will force physicians to change their educational infrastructure to facilitate understanding AI platforms, modeling, and limitations to best acclimate practice in the age of AI. By performing a thorough narrative review, this paper examines these specific AI applications, limitations, and requisites while reviewing a few examples of major data sets that are being cultivated and curated in the US.

A Clinical Prediction Model to Predict Heparin Treatment Outcomes and Provide Dosage Recommendations: Development and Validation Study

BACKGROUND

Unfractionated heparin is widely used in the intensive care unit as an anticoagulant. However, weight-based heparin dosing has been shown to be suboptimal and may place patients at unnecessary risk during their intensive care unit stay.

OBJECTIVE

In this study, we intended to develop and validate a machine learning-based model to predict heparin treatment outcomes and to provide dosage recommendations to clinicians.

METHODS

A shallow neural network model was adopted in a retrospective cohort of patients from the Multiparameter Intelligent Monitoring in Intensive Care III (MIMIC III) database and patients admitted to the Peking Union Medical College Hospital (PUMCH). We modeled the subtherapeutic, normal, and supratherapeutic activated partial thromboplastin time (aPTT) as the outcomes of heparin treatment and used a group of clinical features for modeling. Our model classifies patients into 3 different therapeutic states. We tested the prediction ability of our model and evaluated its performance by using accuracy, the kappa coefficient, precision, recall, and the F1 score. Furthermore, a dosage recommendation module was designed and evaluated for clinical decision support.

RESULTS

A total of 3607 patients selected from MIMIC III and 1549 patients admitted to the PUMCH who met our criteria were included in this study. The shallow neural network model showed results of F1 scores 0.887 (MIMIC III) and 0.925 (PUMCH). When compared with the actual dosage prescribed, our model recommended increasing the dosage for 72.2% (MIMIC III, 1240/1718) and 64.7% (PUMCH, 281/434) of the subtherapeutic patients and decreasing the dosage for 80.9% (MIMIC III, 504/623) and 76.7% (PUMCH, 277/361) of the supratherapeutic patients, suggesting that the recommendations can contribute to clinical improvements and that they may effectively reduce the time to optimal dosage in the clinical setting.

CONCLUSIONS

The evaluation of our model for predicting heparin treatment outcomes demonstrated that the developed model is potentially applicable for reducing the misdosage of heparin and for providing appropriate decision recommendations to clinicians.

Reinforcement Learning Assisted Oxygen Therapy for COVID-19 Patients Under Intensive Care

Background

Patients with severe Coronavirus disease 19 (COVID-19) typically require supplemental oxygen as an essential treatment. We developed a machine learning algorithm, based on deep Reinforcement Learning (RL), for continuous management of oxygen flow rate for critically ill patients under intensive care, which can identify the optimal personalized oxygen flow rate with strong potentials to reduce mortality rate relative to the current clinical practice.

Methods

We modeled the oxygen flow trajectory of COVID-19 patients and their health outcomes as a Markov decision process. Based on individual patient characteristics and health status, an optimal oxygen control policy is learned by using deep deterministic policy gradient (DDPG) and real-time recommends the oxygen flow rate to reduce the mortality rate. We assessed the performance of proposed methods through cross validation by using a retrospective cohort of 1,372 critically ill patients with COVID-19 from New York University Langone Health ambulatory care with electronic health records from April 2020 to January 2021.

Results

The mean mortality rate under the RL algorithm is lower than the standard of care by 2.57% (95% CI: 2.08-3.06) reduction (P<0.001) from 7.94% under the standard of care to 5.37 % under our proposed algorithm. The averaged recommended oxygen flow rate is 1.28 L/min (95% CI: 1.14-1.42) lower than the rate delivered to patients. Thus, the RL algorithm could potentially lead to better intensive care treatment that can reduce the mortality rate, while saving the oxygen scarce resources. It can reduce the oxygen shortage issue and improve public health during the COVID-19 pandemic.

Conclusions

A personalized reinforcement learning oxygen flow control algorithm for COVID-19 patients under intensive care showed a substantial reduction in 7-day mortality rate as compared to the standard of care. In the overall cross validation cohort independent of the training data, mortality was lowest in patients for whom intensivists' actual flow rate matched the RL decisions.

Patient-Specific Sedation Management via Deep Reinforcement Learning

Introduction: Developing reliable medication dosing guidelines is challenging because individual dose-response relationships are mitigated by both static (e. g., demographic) and dynamic factors (e.g., kidney function). In recent years, several data-driven medication dosing models have been proposed for sedatives, but these approaches have been limited in their ability to assess interindividual differences and compute individualized doses. Objective: The primary objective of this study is to develop an individualized framework for sedative-hypnotics dosing. Method: Using publicly available data (1,757 patients) from the MIMIC IV intensive care unit database, we developed a sedation management agent using deep reinforcement learning. More specifically, we modeled the sedative dosing problem as a Markov Decision Process and developed an RL agent based on a deep deterministic policy gradient approach with a prioritized experience replay buffer to find the optimal policy. We assessed our method's ability to jointly learn an optimal personalized policy for propofol and fentanyl, which are among commonly prescribed sedative-hypnotics for intensive care unit sedation. We compared our model's medication performance against the recorded behavior of clinicians on unseen data. Results: Experimental results demonstrate that our proposed model would assist clinicians in making the right decision based on patients' evolving clinical phenotype. The RL agent was 8% better at managing sedation and 26% better at managing mean arterial compared to the clinicians' policy; a two-sample t-test validated that these performance improvements were statistically significant (p < 0.05). Conclusion: The results validate that our model had better performance in maintaining control variables within their target range, thereby jointly maintaining patients' health conditions and managing their sedation.

Reinforcement learning in surgery

Patients and physicians make essential decisions regarding diagnostic and therapeutic interventions. These actions should be performed or deferred under time constraints and uncertainty regarding patients' diagnoses and predicted response to treatment. This may lead to cognitive and judgment errors. Reinforcement learning is a subfield of machine learning that identifies a sequence of actions to increase the probability of achieving a predetermined goal. Reinforcement learning has the potential to assist in surgical decision making by recommending actions at predefined intervals and its ability to utilize complex input data, including text, image, and temporal data, in the decision-making process. The algorithm mimics a human trial-and-error learning process to calculate optimum recommendation policies. The article provides insight regarding challenges in the development and application of reinforcement learning in the medical field, with an emphasis on surgical decision making. The review focuses on challenges in formulating reward function describing the ultimate goal and determination of patient states derived from electronic health records, along with the lack of resources to simulate the potential benefits of suggested actions in response to changing physiological states during and after surgery. Although clinical implementation would require secure, interoperable, livestreaming electronic health record data for use by virtual model, development and validation of personalized reinforcement learning models in surgery can contribute to improving care by helping patients and clinicians make better decisions.