Individualized sepsis treatment using reinforcement learning

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care

Medical treatments often involve a sequence of decisions, each informed by previous outcomes. This process closely aligns with reinforcement learning (RL), a framework for optimizing sequential decisions to maximize cumulative rewards under unknown dynamics. While RL shows promise for creating data-driven treatment plans, its application in medical contexts is challenging due to the frequent need to use sparse rewards, primarily defined based on mortality outcomes. This sparsity can reduce the stability of offline estimates, posing a significant hurdle in fully utilizing RL for medical decision-making. We introduce a deep Q-learning approach to obtain more reliable critical care policies by integrating relevant but noisy frequently measured biomarker signals into the reward specification without compromising the optimization of the main outcome. Our method prunes the action space based on all available rewards before training a final model on the sparse main reward. This approach minimizes potential distortions of the main objective while extracting valuable information from intermediate signals to guide learning. We evaluate our method in off-policy and offline settings using simulated environments and real health records from intensive care units. Our empirical results demonstrate that our method outperforms common offline RL methods such as conservative Q-learning and batch-constrained deep Q-learning. By disentangling sparse rewards and frequently measured reward proxies through action pruning, our work represents a step towards developing reliable policies that effectively harness the wealth of available information in data-intensive critical care environments.

Review of reinforcement learning applications in segmentation, chemotherapy, and radiotherapy of cancer

Owing to early diagnosis and treatment of cancer as a prerequisite in recent times, the role of machine learning has been increased substantially. The mathematically powerful and optimized solutions for the detection and cure of cancer are constantly being explored and novel models based upon standard algorithms are also being developed. Leveraging one such solution is Reinforcement Learning (RL), which is a semi-supervised type of learning. The paper presents a detailed discussion on the various RL techniques, algorithms, and open issues, in addition to the review of literature for diagnosis and treatment of cancer. A smaller number of publications for diagnosis and treatment of cancer have been reported before 2011 but now after the success of Deep Learning (DL) and the advent of Deep Reinforcement Learning (DRL), the publications have grown in number from 2017 onwards. The scope of RL for cancer diagnosis and treatment is also demystified and provides the research community with the insights of how to formulate RL problem as a Cancer diagnostic problem. RL has been found successful for landmark detection in medical images and optimal control of drugs and radiations.

Leveraging physiology and artificial intelligence to deliver advancements in health care

Artificial intelligence in health care has experienced remarkable innovation and progress in the last decade. Significant advancements can be attributed to the utilization of artificial intelligence to transform physiology data to advance health care. In this review, we explore how past work has shaped the field and defined future challenges and directions. In particular, we focus on three areas of development. First, we give an overview of artificial intelligence, with special attention to the most relevant artificial intelligence models. We then detail how physiology data have been harnessed by artificial intelligence to advance the main areas of health care: automating existing health care tasks, increasing access to care, and augmenting health care capabilities. Finally, we discuss emerging concerns surrounding the use of individual physiology data and detail an increasingly important consideration for the field, namely the challenges of deploying artificial intelligence models to achieve meaningful clinical impact.

Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia

Ventilation should be assisted without asynchrony or cardiorespiratory instability during anesthesia emergence until sufficient spontaneous ventilation is recovered. In this multicenter cohort study, we develop and validate a reinforcement learning-based Artificial Intelligence model for Ventilation control during Emergence (AIVE) from general anesthesia. Ventilatory and hemodynamic parameters from 14,306 surgical cases at an academic hospital between 2016 and 2019 are used for training and internal testing of the model. The model's performance is also evaluated on the external validation cohort, which includes 406 cases from another academic hospital in 2022. The estimated reward of the model's policy is higher than that of the clinicians' policy in the internal (0.185, the 95% lower bound for best AIVE policy vs. -0.406, the 95% upper bound for clinicians' policy) and external validation (0.506, the 95% lower bound for best AIVE policy vs. 0.154, the 95% upper bound for clinicians' policy). Cardiorespiratory instability is minimized as the clinicians' ventilation matches the model's ventilation. Regarding feature importance, airway pressure is the most critical factor for ventilation control. In conclusion, the AIVE model achieves higher estimated rewards with fewer complications than clinicians' ventilation control policy during anesthesia emergence.

Reinforcement Learning Model for Managing Noninvasive Ventilation Switching Policy

Noninvasive ventilation (NIV) has been recognized as a first-line treatment for respiratory failure in patients with chronic obstructive pulmonary disease (COPD) and hypercapnia respiratory failure, which can reduce mortality and burden of intubation. However, during the long-term NIV process, failure to respond to NIV may cause overtreatment or delayed intubation, which is associated with increased mortality or costs. Optimal strategies for switching regime in the course of NIV treatment remain to be explored.For the goal of reducing 28-day mortality of the patients undergoing NIV, Double Dueling Deep Q Network (D3QN) of offline-reinforcement learning algorithm was adopted to develop an optimal regime model for making treatment decisions of discontinuing ventilation, continuing NIV, or intubation. The model was trained and tested using the data from Multi-Parameter Intelligent Monitoring in Intensive Care III (MIMIC-III) and evaluated by the practical strategies. Furthermore, the applicability of the model in majority disease subgroups (Catalogued by International Classification of Diseases, ICD) was investigated. Compared with physician's strategies, the proposed model achieved a higher expected return score (4.25 vs. 2.68) and its recommended treatments reduced the expected mortality from 27.82% to 25.44% in all NIV cases. In particular, for these patients finally received intubation in practice, if the model also supported the regime, it would warn of switching to intubation 13.36 hours earlier than clinicians (8.64 vs. 22 hours after the NIV treatment), granting a 21.7% reduction in estimated mortality. In addition, the model was applicable across various disease groups with distinguished achievement in dealing with respiratory disorders. The proposed model is promising to dynamically provide personalized optimal NIV switching regime for patients undergoing NIV with the potential of improving treatment outcomes.

Towards more efficient and robust evaluation of sepsis treatment with deep reinforcement learning

BACKGROUND

In recent years, several studies have applied advanced AI methods, i.e., deep reinforcement learning, in discovering more efficient treatment policies for sepsis. However, due to a paucity of understanding of sepsis itself, the existing approaches still face a severe evaluation challenge, that is, how to properly evaluate the goodness of treatments during the learning process and the effectiveness of the final learned treatment policies.

METHODS

We propose a deep inverse reinforcement learning with mini-tree model that integrates different aspects of factors into the reward formulation, including the critical factors in causing mortality and the key indicators in the existing sepsis treatment guidelines, in order to provide a more comprehensive evaluation of treatments during learning. A new off-policy evaluation method is then proposed to enable more robust evaluation of the learned policies by considering the weighted averaged value functions estimated until the current step.

RESULTS

Results in the MIMIC-III dataset show that the proposed methods can achieve more efficient treatment policies with higher reliability compared to those used by the clinicians.

CONCLUSIONS

A more sound and comprehensive evaluation of treatments of sepsis should consider the most critical factors in infulencing the mortality during treatment as well as those key indicators in the existing sepsis diagnosis guidelines.

Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Closed-Loop Controlled Fluid Administration Systems: A Comprehensive Scoping Review

Physiological Closed-Loop Controlled systems continue to take a growing part in clinical practice, offering possibilities of providing more accurate, goal-directed care while reducing clinicians’ cognitive and task load. These systems also provide a standardized approach for the clinical management of the patient, leading to a reduction in care variability across multiple dimensions. For fluid management and administration, the advantages of closed-loop technology are clear, especially in conditions that require precise care to improve outcomes, such as peri-operative care, trauma, and acute burn care. Controller design varies from simplistic to complex designs, based on detailed physiological models and adaptive properties that account for inter-patient and intra-patient variability; their maturity level ranges from theoretical models tested in silico to commercially available, FDA-approved products. This comprehensive scoping review was conducted in order to assess the current technological landscape of this field, describe the systems currently available or under development, and suggest further advancements that may unfold in the coming years. Ten distinct systems were identified and discussed.

VacSIM: Learning effective strategies for COVID-19 vaccine distribution using reinforcement learning

A COVID-19 vaccine is our best bet for mitigating the ongoing onslaught of the pandemic. However, vaccine is also expected to be a limited resource. An optimal allocation strategy, especially in countries with access inequities and temporal separation of hot-spots, might be an effective way of halting the disease spread. We approach this problem by proposing a novel pipeline VacSIM that dovetails Deep Reinforcement Learning models into a Contextual Bandits approach for optimizing the distribution of COVID-19 vaccine. Whereas the Reinforcement Learning models suggest better actions and rewards, Contextual Bandits allow online modifications that may need to be implemented on a day-to-day basis in the real world scenario. We evaluate this framework against a naive allocation approach of distributing vaccine proportional to the incidence of COVID-19 cases in five different States across India (Assam, Delhi, Jharkhand, Maharashtra and Nagaland) and demonstrate up to 9039 potential infections prevented and a significant increase in the efficacy of limiting the spread over a period of 45 days through the VacSIM approach. Our models and the platform are extensible to all states of India and potentially across the globe. We also propose novel evaluation strategies including standard compartmental model-based projections and a causality-preserving evaluation of our model. Since all models carry assumptions that may need to be tested in various contexts, we open source our model VacSIM and contribute a new reinforcement learning environment compatible with OpenAI gym to make it extensible for real-world applications across the globe. 2

Early detection of sepsis using artificial intelligence: a scoping review protocol

BACKGROUND

Sepsis is a life-threatening organ dysfunction caused by a dysregulated host response to infection. To decrease the high case fatality rates and morbidity for sepsis and septic shock, there is a need to increase the accuracy of early detection of suspected sepsis in prehospital and emergency department settings. This may be achieved by developing risk prediction decision support systems based on artificial intelligence.

METHODS

The overall aim of this scoping review is to summarize the literature on existing methods for early detection of sepsis using artificial intelligence. The review will be performed using the framework formulated by Arksey and O'Malley and further developed by Levac and colleagues. To identify primary studies and reviews that are suitable to answer our research questions, a comprehensive literature collection will be compiled by searching several sources. Constrictions regarding time and language will have to be implemented. Therefore, only studies published between 1 January 1990 and 31 December 2020 will be taken into consideration, and foreign language publications will not be considered, i.e., only papers with full text in English will be included. Databases/web search engines that will be used are PubMed, Web of Science Platform, Scopus, IEEE Xplore, Google Scholar, Cochrane Library, and ACM Digital Library. Furthermore, clinical studies that have completed patient recruitment and reported results found in the database ClinicalTrials.gov will be considered. The term artificial intelligence is viewed broadly, and a wide range of machine learning and mathematical models suitable as base for decision support will be evaluated. Two members of the team will test the framework on a sample of included studies to ensure that the coding framework is suitable and can be consistently applied. Analysis of collected data will provide a descriptive summary and thematic analysis. The reported results will convey knowledge about the state of current research and innovation for using artificial intelligence to detect sepsis in early phases of the medical care chain.

ETHICS AND DISSEMINATION

The methodology used here is based on the use of publicly available information and does not need ethical approval. It aims at aiding further research towards digital solutions for disease detection and health innovation. Results will be extracted into a review report for submission to a peer-reviewed scientific journal. Results will be shared with relevant local and national authorities and disseminated in additional appropriate formats such as conferences, lectures, and press releases.

Trajectories, bifurcations, and pseudo-time in large clinical datasets: applications to myocardial infarction and diabetes data

BACKGROUND

Large observational clinical datasets are becoming increasingly available for mining associations between various disease traits and administered therapy. These datasets can be considered as representations of the landscape of all possible disease conditions, in which a concrete disease state develops through stereotypical routes, characterized by "points of no return" and "final states" (such as lethal or recovery states). Extracting this information directly from the data remains challenging, especially in the case of synchronic (with a short-term follow-up) observations.

RESULTS

Here we suggest a semi-supervised methodology for the analysis of large clinical datasets, characterized by mixed data types and missing values, through modeling the geometrical data structure as a bouquet of bifurcating clinical trajectories. The methodology is based on application of elastic principal graphs, which can address simultaneously the tasks of dimensionality reduction, data visualization, clustering, feature selection, and quantifying the geodesic distances (pseudo-time) in partially ordered sequences of observations. The methodology allows a patient to be positioned on a particular clinical trajectory (pathological scenario) and the degree of progression along it to be characterized with a qualitative estimate of the uncertainty of the prognosis. We developed a tool ClinTrajan for clinical trajectory analysis implemented in the Python programming language. We test the methodology in 2 large publicly available datasets: myocardial infarction complications and readmission of diabetic patients data.

CONCLUSIONS

Our pseudo-time quantification-based approach makes it possible to apply the methods developed for dynamical disease phenotyping and illness trajectory analysis (diachronic data analysis) to synchronic observational data.