The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas

Physiological control for left ventricular assist devices based on deep reinforcement learning

BACKGROUND

The improvement of controllers of left ventricular assist device (LVAD) technology supporting heart failure (HF) patients has enormous impact, given the high prevalence and mortality of HF in the population. The use of reinforcement learning for control applications in LVAD remains minimally explored. This work introduces a preload-based deep reinforcement learning control for LVAD based on the proximal policy optimization algorithm.

METHODS

The deep reinforcement learning control is built upon data derived from a deterministic high-fidelity cardiorespiratory simulator exposed to variations of total blood volume, heart rate, systemic vascular resistance, pulmonary vascular resistance, right ventricular end-systolic elastance, and left ventricular end-systolic elastance, to replicate realistic inter- and intra-patient variability of patients with a severe HF supported by LVAD. The deep reinforcement learning control obtained in this work is trained to avoid ventricular suction and allow aortic valve opening by using left ventricular pressure signals: end-diastolic pressure, maximum pressure in the left ventricle (LV), and maximum pressure in the aorta.

RESULTS

The results show controller obtained in this work, compared to the constant speed LVAD alternative, assures a more stable end-diastolic volume (EDV), with a standard deviation of 5 mL and 9 mL, respectively, and a higher degree of aortic flow, with an average flow of 1.1 L/min and 0.9 L/min, respectively.

CONCLUSION

This work implements a deep reinforcement learning controller in a high-fidelity cardiorespiratory simulator, resulting in increases of flow through the aortic valve and increases of EDV stability, when compared to a constant speed LVAD strategy.

Calibration algorithms for continuous glucose monitoring systems based on interstitial fluid sensing

Continuous glucose monitoring (CGM) is of great importance to the treatment and prevention of diabetes. As a proven commercial technology, electrochemical glucose sensor based on interstitial fluid (ISF) sensing has high sensitivity and wide detection range. Therefore, it has good promotion prospects in noninvasive or minimally-invasive CGM system. However, since there are concentration differences and time lag between glucose in plasma and ISF, the accuracy of this type of sensors are still limited. Typical calibration algorithms rely on simple linear regression which do not account for the variability of the sensitivity of sensors. To enhance the accuracy and stability of CGM based on ISF, optimization of calibration algorithm for sensors is indispensable. While there have been considerable researches on improving calibration algorithms for CGM, they have still received less attention. This article reviews the problem of typical calibration and presents the outstanding calibration algorithms in recent years. Finally, combined with existing research and emerging sensing technologies, this paper makes an outlook on the future calibration algorithms for CGM sensors.

Role of artificial intelligence in gastrointestinal surgery

Artificial intelligence is rapidly evolving and its application is increasing day-by-day in the medical field. The application of artificial intelligence is also valuable in gastrointestinal diseases, by calculating various scoring systems, evaluating radiological images, preoperative and intraoperative assistance, processing pathological slides, prognosticating, and in treatment responses. This field has a promising future and can have an impact on many management algorithms. In this minireview, we aimed to determine the basics of artificial intelligence, the role that artificial intelligence may play in gastrointestinal surgeries and malignancies, and the limitations thereof.

Artificial Intelligence in Pediatric Emergency Medicine: Applications, Challenges, and Future Perspectives

The dawn of Artificial intelligence (AI) in healthcare stands as a milestone in medical innovation. Different medical fields are heavily involved, and pediatric emergency medicine is no exception. We conducted a narrative review structured in two parts. The first part explores the theoretical principles of AI, providing all the necessary background to feel confident with these new state-of-the-art tools. The second part presents an informative analysis of AI models in pediatric emergencies. We examined PubMed and Cochrane Library from inception up to April 2024. Key applications include triage optimization, predictive models for traumatic brain injury assessment, and computerized sepsis prediction systems. In each of these domains, AI models outperformed standard methods. The main barriers to a widespread adoption include technological challenges, but also ethical issues, age-related differences in data interpretation, and the paucity of comprehensive datasets in the pediatric context. Future feasible research directions should address the validation of models through prospective datasets with more numerous sample sizes of patients. Furthermore, our analysis shows that it is essential to tailor AI algorithms to specific medical needs. This requires a close partnership between clinicians and developers. Building a shared knowledge platform is therefore a key step.

Advanced decision support system for individuals with diabetes on multiple daily injections therapy using reinforcement learning and nearest-neighbors: In-silico and clinical results

Many individuals with diabetes on multiple daily insulin injections therapy use carbohydrate ratios (CRs) and correction factors (CFs) to determine mealtime and correction insulin boluses. The CRs and CFs vary over time due to physiological changes in individuals' response to insulin. Errors in insulin dosing can lead to life-threatening abnormal glucose levels, increasing the risk of retinopathy, neuropathy, and nephropathy. Here, we present a novel learning algorithm that uses Q-learning to track optimal CRs and uses nearest-neighbors based Q-learning to track optimal CFs. The learning algorithm was compared with the run-to-run algorithm A and the run-to-run algorithm B, both proposed in the literature, over an 8-week period using a validated simulator with a realistic scenario created with suboptimal CRs and CFs values, carbohydrate counting errors, and random meals sizes at random ingestion times. From Week 1 to Week 8, the learning algorithm increased the percentage of time spent in target glucose range (4.0 to 10.0 mmol/L) from 51 % to 64 % compared to 61 % and 58 % with the run-to-run algorithm A and the run-to-run algorithm B, respectively. The learning algorithm decreased the percentage of time spent below 4.0 mmol/L from 9 % to 1.9 % compared to 3.4 % and 2.3 % with the run-to-run algorithm A and the run-to-run algorithm B, respectively. The algorithm was also assessed by comparing its recommendations with (i) the endocrinologist's recommendations on two type 1 diabetes individuals over a 16-week period and (ii) real-world individuals' therapy settings changes of 23 individuals (19 type 2 and 4 type 1) over an 8-week period using the commercial Bigfoot Unity Diabetes Management System. The full agreements (i) were 89 % and 76 % for CRs and CFs for the type 1 diabetes individuals and (ii) was 62 % for mealtime doses for the individuals on the commercial Bigfoot system. Therefore, the proposed algorithm has the potential to improve glucose control in individuals with type 1 and type 2 diabetes.

Advancements in robotic surgery: innovations, challenges and future prospects

The use of robots has revolutionized healthcare, wherein further innovations have led to improved precision and accuracy. Conceived in the late 1960s, robot-assisted surgeries have evolved to become an integral part of various surgical specialties. Modern robotic surgical systems are equipped with highly dexterous arms and miniaturized instruments that reduce tremors and enable delicate maneuvers. Implementation of advanced materials and designs along with the integration of imaging and visualization technologies have enhanced surgical accuracy and made robots safer and more adaptable to various procedures. Further, the haptic feedback system allows surgeons to determine the consistency of the tissues they are operating upon, without physical contact, thereby preventing injuries due to the application of excess force. With the implementation of teleoperation, surgeons can now overcome geographical limitations and provide specialized healthcare remotely. The use of artificial intelligence (AI) and machine learning (ML) aids in surgical decision-making by improving the recognition of minute and complex anatomical structures. All these advancements have led to faster recovery and fewer complications in patients. However, the substantial cost of robotic systems, their maintenance, the size of the systems and proper surgeon training pose major challenges. Nevertheless, with future advancements such as AI-driven automation, nanorobots, microscopic incision surgeries, semi-automated telerobotic systems, and the impact of 5G connectivity on remote surgery, the growth curve of robotic surgery points to innovation and stands as a testament to the persistent pursuit of progress in healthcare.

Artificial Intelligence and Machine Learning for Improving Glycemic Control in Diabetes: Best Practices, Pitfalls, and Opportunities

OBJECTIVE

Artificial intelligence and machine learning are transforming many fields including medicine. In diabetes, robust biosensing technologies and automated insulin delivery therapies have created a substantial opportunity to improve health. While the number of manuscripts addressing the topic of applying machine learning to diabetes has grown in recent years, there has been a lack of consistency in the methods, metrics, and data used to train and evaluate these algorithms. This manuscript provides consensus guidelines for machine learning practitioners in the field of diabetes, including best practice recommended approaches and warnings about pitfalls to avoid.

METHODS

Algorithmic approaches are reviewed and benefits of different algorithms are discussed including importance of clinical accuracy, explainability, interpretability, and personalization. We review the most common features used in machine learning applications in diabetes glucose control and provide an open-source library of functions for calculating features, as well as a framework for specifying data sets using data sheets. A review of current data sets available for training algorithms is provided as well as an online repository of data sources.

SIGNIFICANCE

These consensus guidelines are designed to improve performance and translatability of new machine learning algorithms developed in the field of diabetes for engineers and data scientists.

Deep reinforcement learning-based control of chemo-drug dose in cancer treatment

BACKGROUND AND OBJECTIVE

Advancement in the treatment of cancer, as a leading cause of death worldwide, has promoted several research activities in various related fields. The development of effective treatment regimens with optimal drug dose administration using a mathematical modeling framework has received extensive research attention during the last decades. However, most of the control techniques presented for cancer chemotherapy are mainly model-based approaches. The available model-free techniques based on Reinforcement Learning (RL), commonly discretize the problem states and variables, which other than demanding expert supervision, cannot model the real-world conditions accurately. The more recent Deep Reinforcement Learning (DRL) methods, which enable modeling the problem in its original continuous space, are rarely applied in cancer chemotherapy.

METHODS

In this paper, we propose an effective and robust DRL-based, model-free method for the closed-loop control of cancer chemotherapy drug dosing. A nonlinear pharmacological cancer model is used for simulating the patient and capturing the cancer dynamics. In contrast to previous work, the state variables and control action are modeled in their original infinite spaces to avoid expert-guided discretization and provide a more realistic solution. The DRL network is trained to automatically adjust the drug dose based on the monitored states of the patient. The proposed method provides an adaptive control technique to respond to the special conditions and diagnosis measurements of different categories of patients.

RESULTS AND CONCLUSIONS

The performance of the proposed DRL-based controller is evaluated by numerical analysis of different diverse simulated patients. Comparison to the state-of-the-art RL-based method, which uses discretized state and action spaces, shows the superiority of the approach in the process and duration of cancer chemotherapy treatment. In the majority of the studied cases, the proposed model decreases the medication period and the total amount of administrated drug, while increasing the rate of reduction in tumor cells.

Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial

The personalized titration and optimization of insulin regimens for treatment of type 2 diabetes (T2D) are resource-demanding healthcare tasks. Here we propose a model-based reinforcement learning (RL) framework (called RL-DITR), which learns the optimal insulin regimen by analyzing glycemic state rewards through patient model interactions. When evaluated during the development phase for managing hospitalized patients with T2D, RL-DITR achieved superior insulin titration optimization (mean absolute error (MAE) of 1.10 ± 0.03 U) compared to other deep learning models and standard clinical methods. We performed a stepwise clinical validation of the artificial intelligence system from simulation to deployment, demonstrating better performance in glycemic control in inpatients compared to junior and intermediate-level physicians through quantitative (MAE of 1.18 ± 0.09 U) and qualitative metrics from a blinded review. Additionally, we conducted a single-arm, patient-blinded, proof-of-concept feasibility trial in 16 patients with T2D. The primary outcome was difference in mean daily capillary blood glucose during the trial, which decreased from 11.1 (±3.6) to 8.6 (±2.4) mmol L-1 (P < 0.01), meeting the pre-specified endpoint. No episodes of severe hypoglycemia or hyperglycemia with ketosis occurred. These preliminary results warrant further investigation in larger, more diverse clinical studies. ClinicalTrials.gov registration: NCT05409391 .

A dosing strategy model of deep deterministic policy gradient algorithm for sepsis patients

BACKGROUND

A growing body of research suggests that the use of computerized decision support systems can better guide disease treatment and reduce the use of social and medical resources. Artificial intelligence (AI) technology is increasingly being used in medical decision-making systems to obtain optimal dosing combinations and improve the survival rate of sepsis patients. To meet the real-world requirements of medical applications and make the training model more robust, we replaced the core algorithm applied in an AI-based medical decision support system developed by research teams at the Massachusetts Institute of Technology (MIT) and IMPERIAL College London (ICL) with the deep deterministic policy gradient (DDPG) algorithm. The main objective of this study was to develop an AI-based medical decision-making system that makes decisions closer to those of professional human clinicians and effectively reduces the mortality rate of sepsis patients.

METHODS

We used the same public intensive care unit (ICU) dataset applied by the research teams at MIT and ICL, i.e., the Multiparameter Intelligent Monitoring in Intensive Care III (MIMIC-III) dataset, which contains information on the hospitalizations of 38,600 adult sepsis patients over the age of 15. We applied the DDPG algorithm as a strategy-based reinforcement learning approach to construct an AI-based medical decision-making system and analyzed the model results within a two-dimensional space to obtain the optimal dosing combination decision for sepsis patients.

RESULTS

The results show that when the clinician administered the exact same dose as that recommended by the AI model, the mortality of the patients reached the lowest rate at 11.59%. At the same time, according to the database, the baseline mortality rate of the patients was calculated as 15.7%. This indicates that the patient mortality rate when difference between the doses administered by clinicians and those determined by the AI model was zero was approximately 4.2% lower than the baseline patient mortality rate found in the dataset. The results also illustrate that when a clinician administered a different dose than that recommended by the AI model, the patient mortality rate increased, and the greater the difference in dose, the higher the patient mortality rate. Furthermore, compared with the medical decision-making system based on the Deep-Q Learning Network (DQN) algorithm developed by the research teams at MIT and ICL, the optimal dosing combination recommended by our model is closer to that given by professional clinicians. Specifically, the number of patient samples administered by clinicians with the exact same dose recommended by our AI model increased by 142.3% compared with the model based on the DQN algorithm, with a reduction in the patient mortality rate of 2.58%.

CONCLUSIONS

The treatment plan generated by our medical decision-making system based on the DDPG algorithm is closer to that of a professional human clinician with a lower mortality rate in hospitalized sepsis patients, which can better help human clinicians deal with complex conditional changes in sepsis patients in an ICU. Our proposed AI-based medical decision-making system has the potential to provide the best reference dosing combinations for additional drugs.

Bolus Insulin calculation without meal information. A reinforcement learning approach

In continuous subcutaneous insulin infusion and multiple daily injections, insulin boluses are usually calculated based on patient-specific parameters, such as carbohydrates-to-insulin ratio (CR), insulin sensitivity-based correction factor (CF), and the estimation of the carbohydrates (CHO) to be ingested. This study aimed to calculate insulin boluses without CR, CF, and CHO content, thereby eliminating the errors caused by misestimating CHO and alleviating the management burden on the patient. A Q-learning-based reinforcement learning algorithm (RL) was developed to optimise bolus insulin doses for in-silico type 1 diabetic patients. A realistic virtual cohort of 68 patients with type 1 diabetes that was previously developed by our research group, was considered for the in-silico trials. The results were compared to those of the standard bolus calculator (SBC) with and without CHO misestimation using open-loop basal insulin therapy. The percentage of the overall duration spent in the target range of 70-180 mg/dL was 73.4% and 72.37%, <70 mg/dL was 1.96 and 0.70%, and >180 mg/dL was 23.40 and 24.63%, respectively, for RL and SBC without CHO misestimation. The results revealed that RL outperformed SBC in the presence of CHO misestimation, and despite not knowing the CHO content of meals, the performance of RL was similar to that of SBC in perfect conditions. This algorithm can be incorporated into artificial pancreas and automatic insulin delivery systems in the future.

A review on modeling tumor dynamics and agent reward functions in reinforcement learning based therapy optimization

Revolutionary changes of deep reinforcement learning are leading to high-performing intelligent solutions in multiple fields, including healthcare. At the moment, chemotherapy and radiotherapy are common types of treatments for cancer, however, both therapies are usually radical procedures with undesirable side effects. There is an increasing number of evidence that patient-based optimal schedule has a significant impact in increasing efficiency and survival, and reducing side effects during both therapies. To apply artificial intelligence in therapy optimization, an adequate model of tumor growth incorporating the effect of the treatment is mandatory. A method on training a controller for dosage and scheduling, reinforcement learning can be applied, where a well-chosen agent rewarding function is key to achieve optimal behavior. In this survey paper, some selected tumor growth models, reinforcement learning based solutions and especially agent reward functions are reviewed and compared, providing a summary on state of the art approaches.

Projection based inverse reinforcement learning for the analysis of dynamic treatment regimes

Dynamic Treatment Regimes (DTRs) are adaptive treatment strategies that allow clinicians to personalize dynamically the treatment for each patient based on their step-by-step response to their treatment. There are a series of predefined alternative treatments for each disease and any patient may associate with one of these treatments according to his/her demographics. DTRs for a certain disease are studied and evaluated by means of statistical approaches where patients are randomized at each step of the treatment and their responses are observed. Recently, the Reinforcement Learning (RL) paradigm has also been applied to determine DTRs. However, such approaches may be limited by the need to design a true reward function, which may be difficult to formalize when the expert knowledge is not well assessed, as when the DTR is in the design phase. To address this limitation, an extension of the RL paradigm, namely Inverse Reinforcement Learning (IRL), has been adopted to learn the reward function from data, such as those derived from DTR trials. In this paper, we define a Projection Based Inverse Reinforcement Learning (PB-IRL) approach to learn the true underlying reward function for given demonstrations (DTR trials). Such a reward function can be used both to evaluate the set of DTRs determined for a certain disease, as well as to enable an RL-based intelligent agent to self-learn the best way and then act as a decision support system for the clinician.

Evaluation of blood glucose level control in type 1 diabetic patients using deep reinforcement learning

Diabetes mellitus is a disease associated with abnormally high levels of blood glucose due to a lack of insulin. Combining an insulin pump and continuous glucose monitor with a control algorithm to deliver insulin is an alternative to patient self-management of insulin doses to control blood glucose levels in diabetes mellitus patients. In this work, we propose a closed-loop control for blood glucose levels based on deep reinforcement learning. We describe the initial evaluation of several alternatives conducted on a realistic simulator of the glucoregulatory system and propose a particular implementation strategy based on reducing the frequency of the observations and rewards passed to the agent, and using a simple reward function. We train agents with that strategy for three groups of patient classes, evaluate and compare it with alternative control baselines. Our results show that our method is able to outperform baselines as well as similar recent proposals, by achieving longer periods of safe glycemic state and low risk.

Reinforcement learning in ophthalmology: potential applications and challenges to implementation

Reinforcement learning is a subtype of machine learning in which a virtual agent, functioning within a set of predefined rules, aims to maximise a specified outcome or reward. This agent can consider multiple variables and many parallel actions at once to optimise its reward, thereby solving complex, sequential problems. Clinical decision making requires physicians to optimise patient outcomes within a set practice framework and, thus, presents considerable opportunity for the implementation of reinforcement learning-driven solutions. We provide an overview of reinforcement learning, and focus on potential applications within ophthalmology. We also explore the challenges associated with development and implementation of reinforcement learning solutions and discuss possible approaches to address them.

A Reinforcement Learning Based System for Blood Glucose Control without Carbohydrate Estimation in Type 1 Diabetes: In Silico Validation

Type 1 Diabetes (T1D) is a chronic autoimmune disease, which requires the use of exogenous insulin for glucose regulation. In current hybrid closed-loop systems, meal entry is manual which adds cognitive burden to the persons living with T1D. In this study, we proposed a control system based on Proximal Policy Optimisation (PPO) that controls both basal and bolus insulin infusion and only requires meal announcement, thus eliminating the need for carbohydrate estimation. We evaluated the system on a challenging meal scenario, using an open-source simulator based on the UVA/Padova 2008 model and achieved a mean Time in Range value of 65% for the adult subject cohort, while maintaining a moderate hypoglycemic and hyperglycemic risk profile. The approach shows promise and welcomes further research towards the translation to a real-life artificial pancreas. Clinical relevance- This was an in-silico analysis towards the development of an autonomous artificial pancreas system for glucose control. The proposed system show promise in eliminating the need for estimating the carbohydrate content in meals.

Artificial intelligence in critically ill diabetic patients: current status and future prospects

Recent years have witnessed increasing numbers of artificial intelligence (AI) based applications and devices being tested and approved for medical care. Diabetes is arguably the most common chronic disorder worldwide and AI is now being used for making an early diagnosis, to predict and diagnose early complications, increase adherence to therapy, and even motivate patients to manage diabetes and maintain glycemic control. However, these AI applications have largely been tested in non-critically ill patients and aid in managing chronic problems. Intensive care units (ICUs) have a dynamic environment generating huge data, which AI can extract and organize simultaneously, thus analysing many variables for diagnostic and/or therapeutic purposes in order to predict outcomes of interest. Even non-diabetic ICU patients are at risk of developing hypo or hyperglycemia, complicating their ICU course and affecting outcomes. In addition, to maintain glycemic control frequent blood sampling and insulin dose adjustments are required, increasing nursing workload and chances of error. AI has the potential to improve glycemic control while reducing the nursing workload and errors. Continuous glucose monitoring (CGM) devices, which are Food and Drug Administration (FDA) approved for use in non-critically ill patients, are now being recommended for use in specific ICU populations with increased accuracy. AI based devices including artificial pancreas and CGM regulated insulin infusion system have shown promise as comprehensive glycemic control solutions in critically ill patients. Even though many of these AI applications have shown potential, these devices need to be tested in larger number of ICU patients, have wider availability, show favorable cost-benefit ratio and be amenable for easy integration into the existing healthcare systems, before they become acceptable to ICU physicians for routine use.

Machine Learning and Smart Devices for Diabetes Management: Systematic Review

(1) Background: The use of smart devices to better manage diabetes has increased significantly in recent years. These technologies have been introduced in order to make life easier for patients with diabetes by allowing better control of the stability of blood sugar levels and anticipating the occurrence of dangerous events (hypo/hyperglycemia), etc. That being said, the main objectives of the self-management of diabetes is to improve the lifestyle and life quality of patients with diabetes; (2) Methods: We performed a systematic review based on articles that focus on the use of smart devices for the monitoring and better management of diabetes. The search was focused on keywords related to the topic, such as “Diabetes”, “Technology”, “Self-management”, “Artificial Intelligence”, etc. This was performed using databases, such as Scopus, Google Scholar, and PubMed; (3) Results: A total of 89 studies, published between 2011 and 2021, were included. The majority of the selected research aims to solve a diabetes management problem (e.g., blood glucose prediction, early detection of risk events, and the automatic adjustment of insulin doses, etc.). In these studies, wearable devices were used in combination with artificial intelligence (AI) techniques; (4) Conclusions: Wearable devices have attracted a great deal of scientific interest in the field of healthcare for people with chronic conditions, such as diabetes. They are capable of assisting in the management of diabetes, as well as preventing complications associated with this condition. Furthermore, the usage of these devices has improved illness management and quality of life.

Integrating Multiple Inputs Into an Artificial Pancreas System: Narrative Literature Review

BACKGROUND

Type 1 diabetes (T1D) is a chronic autoimmune disease in which a deficiency in insulin production impairs the glucose homeostasis of the body. Continuous subcutaneous infusion of insulin is a commonly used treatment method. Artificial pancreas systems (APS) use continuous glucose level monitoring and continuous subcutaneous infusion of insulin in a closed-loop mode incorporating a controller (or control algorithm). However, the operation of APS is challenging because of complexities arising during meals, exercise, stress, sleep, illnesses, glucose sensing and insulin action delays, and the cognitive burden. To overcome these challenges, options to augment APS through integration of additional inputs, creating multi-input APS (MAPS), are being investigated.

OBJECTIVE

The aim of this survey is to identify and analyze input data, control architectures, and validation methods of MAPS to better understand the complexities and current state of such systems. This is expected to be valuable in developing improved systems to enhance the quality of life of people with T1D.

METHODS

A literature survey was conducted using the Scopus, PubMed, and IEEE Xplore databases for the period January 1, 2005, to February 10, 2020. On the basis of the search criteria, 1092 articles were initially shortlisted, of which 11 (1.01%) were selected for an in-depth narrative analysis. In addition, 6 clinical studies associated with the selected studies were also analyzed.

RESULTS

Signals such as heart rate, accelerometer readings, energy expenditure, and galvanic skin response captured by wearable devices were the most frequently used additional inputs. The use of invasive (blood or other body fluid analytes) inputs such as lactate and adrenaline were also simulated. These inputs were incorporated to switch the mode of the controller through activity detection, directly incorporated for decision-making and for the development of intermediate modules for the controller. The validation of the MAPS was carried out through the use of simulators based on different physiological models and clinical trials.

CONCLUSIONS

The integration of additional physiological signals with continuous glucose level monitoring has the potential to optimize glucose control in people with T1D through addressing the identified limitations of APS. Most of the identified additional inputs are related to wearable devices. The rapid growth in wearable technologies can be seen as a key motivator regarding MAPS. However, it is important to further evaluate the practical complexities and psychosocial aspects associated with such systems in real life.

Data-Driven Robust Control Using Reinforcement Learning

This paper proposes a robust control design method using reinforcement learning for controlling partially-unknown dynamical systems under uncertain conditions. The method extends the optimal reinforcement learning algorithm with a new learning technique based on the robust control theory. By learning from the data, the algorithm proposes actions that guarantee the stability of the closed-loop system within the uncertainties estimated also from the data. Control policies are calculated by solving a set of linear matrix inequalities. The controller was evaluated using simulations on a blood glucose model for patients with Type 1 diabetes. Simulation results show that the proposed methodology is capable of safely regulating the blood glucose within a healthy level under the influence of measurement and process noises. The controller has also significantly reduced the post-meal fluctuation of the blood glucose. A comparison between the proposed algorithm and the existing optimal reinforcement learning algorithm shows the improved robustness of the closed-loop system using our method.

A Machine Learning Approach to Minimize Nocturnal Hypoglycemic Events in Type 1 Diabetic Patients under Multiple Doses of Insulin

Nocturnal hypoglycemia (NH) is one of the most challenging events for multiple dose insulin therapy (MDI) in people with type 1 diabetes (T1D). The goal of this study is to design a method to reduce the incidence of NH in people with T1D under MDI therapy, providing a decision-support system and improving confidence toward self-management of the disease considering the dataset used by Bertachi et al. Different machine learning (ML) algorithms, data sources, optimization metrics and mitigation measures to predict and avoid NH events have been studied. In addition, we have designed population and personalized models and studied the generalizability of the models and the influence of physical activity (PA) on them. Obtaining 30 g of rescue carbohydrates (CHO) is the optimal value for preventing NH, so it can be asserted that this is the value with which the time under 70 mg/dL decreases the most, with almost a 35% reduction, while increasing the time in the target range by 1.3%. This study supports the feasibility of using ML techniques to address the prediction of NH in patients with T1D under MDI therapy, using continuous glucose monitoring (CGM) and a PA tracker. The results obtained prove that BG predictions can not only be critical in achieving safer diabetes management, but also assist physicians and patients to make better and safer decisions regarding insulin therapy and their day-to-day lives.

Long-term use of the hybrid artificial pancreas by adjusting carbohydrate ratios and programmed basal rate: A reinforcement learning approach

BACKGROUND AND OBJECTIVES

The hybrid artificial pancreas regulates glucose levels in people with type 1 diabetes. It delivers (i) insulin boluses at meal times based on the meals' carbohydrate content and the carbohydrate ratios (CRs) and (ii) insulin basal, between meals and at night, continuously modulated around individual-specific programmed basal rate. The CRs and programmed basal rate significantly vary between individuals and within the same individual with type 1 diabetes, and using suboptimal values in the hybrid artificial pancreas may degrade glucose control. We propose a reinforcement learning algorithm to adaptively optimize CRs and programmed basal rate to improve the performance of the hybrid artificial pancreas.

METHODS

The proposed reinforcement learning algorithm was designed using the Q-learning approach. The algorithm learns the optimal actions (CRs and programmed basal rate) by applying them to the individual's state (previous day's glucose levels and insulin delivery) based on an exploration and exploitation trade-off. First, outcomes from our simulator were compared to those of a clinical study in 23 individuals with type 1 diabetes and have yielded similar results. Second, the learning algorithm was tested using the simulator with two scenarios. Scenario 1 has fixed meal sizes and ingestion times and scenario 2 has a more realistic eating behavior with random meal sizes, ingestion times, and carbohydrate counting errors.

RESULTS

After about five weeks, the reinforcement learning algorithm improved the percentage of time spent in target range from 67% to 86.7% in scenario 1 and 65.5% to 86% in scenario 2. The percentage of time spent below 4.0 mmol/L decreased from 9% to 0.9% in scenario 1 and 9.5% to 1.1% in scenario 2.

CONCLUSIONS

Results indicate that the proposed algorithm has the potential to improve glucose control in people with type 1 diabetes using the hybrid artificial pancreas. The proposed algorithm is a key in making the hybrid artificial pancreas adaptive for the long-term real life outpatient studies.

Toward a Fully Automated Artificial Pancreas System Using a Bioinspired Reinforcement Learning Design: In Silico Validation

OBJECTIVE

The automation of insulin treatment is the most challenge aspect of glucose management for type 1 diabetes owing to unexpected exogenous events (e.g., meal intake). In this article, we propose a novel reinforcement learning (RL) based artificial intelligence (AI) algorithm for a fully automated artificial pancreas (AP) system.

METHODS

A bioinspired RL designing method was developed for automated insulin infusion. This method has reward functions that imply the temporal homeostatic objective and discount factors that reflect an individual specific pharmacological characteristic. The proposed method was applied to a training method using an RL algorithm and was evaluated in virtual patients from the FDA approved UVA/Padova simulator with unannounced meal intakes.

RESULTS

For a single-meal experiment with preprandial fasting, the trained policy demonstrated fully automated regulation in both the basal and postprandial phases. In the in silico trial with a variation of insulin sensitivity and dawn phenomenon, the policy achieved a mean glucose of 124.72 mg/dL and percentage time in the normal range of 89.56%. The layer-wise relevance propagation provides interpretable information on AI-driven decision for robustness to sensor noise, automated postprandial regulation, and insulin stacking avoidance.

CONCLUSION

The AP algorithm based on the bioinspired RL approach enables fully automated blood glucose control with unannounced meal intake.

SIGNIFICANCE

The proposed framework can be extended to other drug-based treatments for systems with significant uncertainties.

Predicting Incremental and Future Visual Change in Neovascular Age-Related Macular Degeneration Using Deep Learning

PURPOSE

To evaluate the predictive usefulness of quantitative imaging biomarkers, acquired automatically from OCT scans, of cross-sectional and future visual outcomes of patients with neovascular age-related macular degeneration (AMD) starting anti-vascular endothelial growth factor (VEGF) therapy.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Treatment-naive, first-treated eyes of patients with neovascular AMD between 2007 and 2017 at Moorfields Eye Hospital (a large, United Kingdom single center) undergoing anti-VEGF therapy.

METHODS

Automatic segmentation was carried out by applying a deep learning segmentation algorithm to 137 379 OCT scans from 6467 eyes of 3261 patients with neovascular AMD. After applying selection criteria, 926 eyes of 926 patients were analyzed.

MAIN OUTCOME MEASURES

Correlation coefficients (R² values) and mean absolute error (MAE) between quantitative OCT (qOCT) parameters and cross-sectional visual function, as well as the predictive value of these parameters for short-term visual change, that is, incremental visual acuity (VA) resulting from an individual injection, as well as VA at distant time points (up to 12 months after baseline).

RESULTS

Visual acuity at distant time points could be predicted: R² = 0.80 (MAE, 5.0 Early Treatment Diabetic Retinopathy Study [ETDRS] letters) and R² = 0.7 (MAE, 7.2 ETDRS letters) after injection at 3 and at 12 months after baseline (P < 0.001 for both), respectively. Best performing models included both baseline qOCT parameters and treatment response. Furthermore, we present proof-of-principle evidence that the incremental change in VA from an injection can be predicted: R² = 0.14 (MAE, 5.6 ETDRS letters) for injection 2 and R² = 0.11 (MAE, 5.0 ETDRS letters) for injection 3 (P < 0.001 for both).

CONCLUSIONS

Automatic segmentation enables rapid acquisition of quantitative and reproducible OCT biomarkers with potential to inform treatment decisions in the care of neovascular AMD. This furthers development of point-of-care decision-aid systems for personalized medicine.

Intelligent automated drug administration and therapy: future of healthcare

In the twenty-first century, the collaboration of control engineering and the healthcare sector has matured to some extent; however, the future will have promising opportunities, vast applications, and some challenges. Due to advancements in processing speed, the closed-loop administration of drugs has gained popularity for critically ill patients in intensive care units and routine life such as personalized drug delivery or implantable therapeutic devices. For developing a closed-loop drug delivery system, the control system works with a group of technologies like sensors, micromachining, wireless technologies, and pharmaceuticals. Recently, the integration of artificial intelligence techniques such as fuzzy logic, neural network, and reinforcement learning with the closed-loop drug delivery systems has brought their applications closer to fully intelligent automatic healthcare systems. This review's main objectives are to discuss the current developments, possibilities, and future visions in closed-loop drug delivery systems, for providing treatment to patients suffering from chronic diseases. It summarizes the present insight of closed-loop drug delivery/therapy for diabetes, gastrointestinal tract disease, cancer, anesthesia administration, cardiac ailments, and neurological disorders, from a perspective to show the research in the area of control theory.

Developing Insulin Delivery Devices with Glucose Responsiveness

Individuals with type 1 and advanced type 2 diabetes require daily insulin therapy to maintain blood glucose levels in normoglycemic ranges to prevent associated morbidity and mortality. Optimal insulin delivery should offer both precise dosing in response to real-time blood glucose levels as well as a feasible and low-burden administration route to promote long-term adherence. A series of glucose-responsive insulin delivery mechanisms and devices have been reported to increase patient compliance while mitigating the risk of hypoglycemia. This review discusses currently available insulin delivery devices, overviews recent developments towards the generation of glucose-responsive delivery systems, and provides commentary on the opportunities and barriers ahead regarding the integration and translation of current glucose-responsive insulin delivery designs.

In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus

In this paper, we test and evaluate policy gradient reinforcement learning for automated blood glucose control in patients with Type 1 Diabetes Mellitus. Recent research has shown that reinforcement learning is a promising approach to accommodate the need for individualized blood glucose level control algorithms. The motivation for using policy gradient algorithms comes from the fact that adaptively administering insulin is an inherently continuous task. Policy gradient algorithms are known to be superior in continuous high-dimensional control tasks. Previously, most of the approaches for automated blood glucose control using reinforcement learning has used a finite set of actions. We use the Trust-Region Policy Optimization algorithm in this work. It represents the state of the art for deep policy gradient algorithms. The experiments are carried out in-silico using the Hovorka model, and stochastic behavior is modeled through simulated carbohydrate counting errors to illustrate the full potential of the framework. Furthermore, we use a model-free approach where no prior information about the patient is given to the algorithm. Our experiments show that the reinforcement learning agent is able to compete with and sometimes outperform state-of-the-art model predictive control in blood glucose regulation.

An Insulin Bolus Advisor for Type 1 Diabetes Using Deep Reinforcement Learning

(1) Background: People living with type 1 diabetes (T1D) require self-management to maintain blood glucose (BG) levels in a therapeutic range through the delivery of exogenous insulin. However, due to the various variability, uncertainty and complex glucose dynamics, optimizing the doses of insulin delivery to minimize the risk of hyperglycemia and hypoglycemia is still an open problem. (2) Methods: In this work, we propose a novel insulin bolus advisor which uses deep reinforcement learning (DRL) and continuous glucose monitoring to optimize insulin dosing at mealtime. In particular, an actor-critic model based on deep deterministic policy gradient is designed to compute mealtime insulin doses. The proposed system architecture uses a two-step learning framework, in which a population model is first obtained and then personalized by subject-specific data. Prioritized memory replay is adopted to accelerate the training process in clinical practice. To validate the algorithm, we employ a customized version of the FDA-accepted UVA/Padova T1D simulator to perform in silico trials on 10 adult subjects and 10 adolescent subjects. (3) Results: Compared to a standard bolus calculator as the baseline, the DRL insulin bolus advisor significantly improved the average percentage time in target range (70-180 mg/dL) from 74.1%±8.4% to 80.9%±6.9% (p<0.01) and 54.9%±12.4% to 61.6%±14.1% (p<0.01) in the the adult and adolescent cohorts, respectively, while reducing hypoglycemia. (4) Conclusions: The proposed algorithm has the potential to improve mealtime bolus insulin delivery in people with T1D and is a feasible candidate for future clinical validation.

Reinforcement learning for intelligent healthcare applications: A survey

Discovering new treatments and personalizing existing ones is one of the major goals of modern clinical research. In the last decade, Artificial Intelligence (AI) has enabled the realization of advanced intelligent systems able to learn about clinical treatments and discover new medical knowledge from the huge amount of data collected. Reinforcement Learning (RL), which is a branch of Machine Learning (ML), has received significant attention in the medical community since it has the potentiality to support the development of personalized treatments in accordance with the more general precision medicine vision. This report presents a review of the role of RL in healthcare by investigating past work, and highlighting any limitations and possible future contributions.

Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

Background

Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in healthcare domains. Recent years have seen a great progress of applying RL in addressing decision-making problems in Intensive Care Units (ICUs). However, since the goal of traditional RL algorithms is to maximize a long-term reward function, exploration in the learning process may have a fatal impact on the patient. As such, a short-term goal should also be considered to keep the patient stable during the treating process.

Methods

We use a Supervised-Actor-Critic (SAC) RL algorithm to address this problem by combining the long-term goal-oriented characteristics of RL with the short-term goal of supervised learning. We evaluate the differences between SAC and traditional Actor-Critic (AC) algorithms in addressing the decision making problems of ventilation and sedative dosing in ICUs.

Results

Results show that SAC is much more efficient than the traditional AC algorithm in terms of convergence rate and data utilization.

Conclusions

The SAC algorithm not only aims to cure patients in the long term, but also reduces the degree of deviation from the strategy applied by clinical doctors and thus improves the therapeutic effect.

Artificial Intelligence: Can Information be Transformed into Intelligence in Surgical Education?

Artificial intelligence (AI) is being rapidly integrated into various medical applications. Although early application of AI has been achieved in image-based, as well as statistical computational models, translation into procedure-based specialties such as surgery may take longer to achieve. A potential application of AI in surgical education is as a teaching coach or mentor that interacts with the used via virtual and/or augmented reality. The question arises as to whether machines will achieve the wisdom and intelligence of human educators.

Reinforcement learning-based control of tumor growth under anti-angiogenic therapy

BACKGROUND AND OBJECTIVES

In recent decades, cancer has become one of the most fatal and destructive diseases which is threatening humans life. Accordingly, different types of cancer treatment are studied with the main aim to have the best treatment with minimum side effects. Anti-angiogenic is a molecular targeted therapy which can be coupled with chemotherapy and radiotherapy. Although this method does not eliminate the whole tumor, but it can keep the tumor size in a given state by preventing the formation of new blood vessels. In this paper, a novel model-free method based on reinforcement learning (RL) framework is used to design a closed-loop control of anti-angiogenic drug dosing administration.

METHODS

A Q-learning algorithm is developed for the drug dosing closed-loop control. This controller is designed using two different values of the maximum drug dosage to reduce the tumor volume up to a desired value. The mathematical model of tumor growth under anti-angiogenic inhibitor is used to simulate a real patient.

RESULTS

The effectiveness of the proposed method is shown through in silico simulation and its robustness to patient parameters variation is demonstrated. It is demonstrated that the tumor reaches its minimal volume in 84 days with maximum drug inlet of 30 mg/kg/day. Also, it is shown that the designed controller is robust with respect to  ± 20% of tumor growth parameters changes.

CONCLUSION

The proposed closed-loop reinforcement learning-based controller for cancer treatment using anti-angiogenic inhibitor provides an effective and novel result such that with a clinically valid and safe dosage of drug, the volume reduces up to 1mm³ in a reasonable short period compared to the literature.

Gastroenterology Meets Machine Learning: Status Quo and Quo Vadis

Machine learning has undergone a transition phase from being a pure statistical tool to being one of the main drivers of modern medicine. In gastroenterology, this technology is motivating a growing number of studies that rely on these innovative methods to deal with critical issues related to this practice. Hence, in the light of the burgeoning research on the use of machine learning in gastroenterology, a systematic review of the literature is timely. In this work, we present the results gleaned through a systematic review of prominent gastroenterology literature using machine learning techniques. Based on the analysis of 88 journal articles, we delimit the scope of application, we discuss current limitations including bias, lack of transparency, accountability, and data availability, and we put forward future avenues.

Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm

Background

Type-1 diabetes is a condition caused by the lack of insulin hormone, which leads to an excessive increase in blood glucose level. The glucose kinetics process is difficult to control due to its complex and nonlinear nature and with state variables that are difficult to measure.

Methods

This paper proposes a method for automatically calculating the basal and bolus insulin doses for patients with type-1 diabetes using reinforcement learning with feedforward controller. The algorithm is designed to keep the blood glucose stable and directly compensate for the external events such as food intake. Its performance was assessed using simulation on a blood glucose model. The usage of the Kalman filter with the controller was demonstrated to estimate unmeasurable state variables.

Results

Comparison simulations between the proposed controller with the optimal reinforcement learning and the proportional-integral-derivative controller show that the proposed methodology has the best performance in regulating the fluctuation of the blood glucose. The proposed controller also improved the blood glucose responses and prevented hypoglycemia condition. Simulation of the control system in different uncertain conditions provided insights on how the inaccuracies of carbohydrate counting and meal-time reporting affect the performance of the control system.

Conclusion

The proposed controller is an effective tool for reducing postmeal blood glucose rise and for countering the effects of external known events such as meal intake and maintaining blood glucose at a healthy level under uncertainties.

The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care

Sepsis is the third leading cause of death worldwide and the main cause of mortality in hospitals^1-3, but the best treatment strategy remains uncertain. In particular, evidence suggests that current practices in the administration of intravenous fluids and vasopressors are suboptimal and likely induce harm in a proportion of patients^1,4-6. To tackle this sequential decision-making problem, we developed a reinforcement learning agent, the Artificial Intelligence (AI) Clinician, which extracted implicit knowledge from an amount of patient data that exceeds by many-fold the life-time experience of human clinicians and learned optimal treatment by analyzing a myriad of (mostly suboptimal) treatment decisions. We demonstrate that the value of the AI Clinician's selected treatment is on average reliably higher than human clinicians. In a large validation cohort independent of the training data, mortality was lowest in patients for whom clinicians' actual doses matched the AI decisions. Our model provides individualized and clinically interpretable treatment decisions for sepsis that could improve patient outcomes.

Neural network-based model predictive control for type 1 diabetic rats on artificial pancreas system

Artificial pancreas system (APS) is a viable option to treat diabetic patients. Researchers, however, have not conclusively determined the best control method for APS. Due to intra-/inter-variability of insulin absorption and action, an individualized algorithm is required to control blood glucose level (BGL) for each patient. To this end, we developed model predictive control (MPC) based on artificial neural networks (ANNs), which combines ANN for BGL prediction based on inputs and MPC for BGL control based on the ANN (NN-MPC). First, we developed a mathematical model for diabetic rats, which was used to identify individual virtual subjects by fitting to empirical data collected through an APS, including BGL data, insulin injection, and food intake. Then, the virtual subjects were used to generate datasets for training ANNs. The NN-MPC determines control actions (insulin injection) based on BGL predicted by the ANN. To evaluate the NN-MPC, we conducted experiments using four virtual subjects under three different scenarios. Overall, the NN-MPC maintained BGL within the normal range about 90% of the time with a mean absolute deviation of 4.7 mg/dl from a desired BGL. Our findings suggest that the NN-MPC can provide subject-specific BGL control in conjunction with a closed-loop APS. Graphical abstract ᅟ.

Artificial Intelligence in Surgery: Promises and Perils

OBJECTIVE

The aim of this review was to summarize major topics in artificial intelligence (AI), including their applications and limitations in surgery. This paper reviews the key capabilities of AI to help surgeons understand and critically evaluate new AI applications and to contribute to new developments.

SUMMARY BACKGROUND DATA

AI is composed of various subfields that each provide potential solutions to clinical problems. Each of the core subfields of AI reviewed in this piece has also been used in other industries such as the autonomous car, social networks, and deep learning computers.

METHODS

A review of AI papers across computer science, statistics, and medical sources was conducted to identify key concepts and techniques within AI that are driving innovation across industries, including surgery. Limitations and challenges of working with AI were also reviewed.

RESULTS

Four main subfields of AI were defined: (1) machine learning, (2) artificial neural networks, (3) natural language processing, and (4) computer vision. Their current and future applications to surgical practice were introduced, including big data analytics and clinical decision support systems. The implications of AI for surgeons and the role of surgeons in advancing the technology to optimize clinical effectiveness were discussed.

CONCLUSIONS

Surgeons are well positioned to help integrate AI into modern practice. Surgeons should partner with data scientists to capture data across phases of care and to provide clinical context, for AI has the potential to revolutionize the way surgery is taught and practiced with the promise of a future optimized for the highest quality patient care.

Artificial Intelligence for Diabetes Management and Decision Support: Literature Review

Background

Artificial intelligence methods in combination with the latest technologies, including medical devices, mobile computing, and sensor technologies, have the potential to enable the creation and delivery of better management services to deal with chronic diseases. One of the most lethal and prevalent chronic diseases is diabetes mellitus, which is characterized by dysfunction of glucose homeostasis.

Objective

The objective of this paper is to review recent efforts to use artificial intelligence techniques to assist in the management of diabetes, along with the associated challenges.

Methods

A review of the literature was conducted using PubMed and related bibliographic resources. Analyses of the literature from 2010 to 2018 yielded 1849 pertinent articles, of which we selected 141 for detailed review.

Results

We propose a functional taxonomy for diabetes management and artificial intelligence. Additionally, a detailed analysis of each subject category was performed using related key outcomes. This approach revealed that the experiments and studies reviewed yielded encouraging results.

Conclusions

We obtained evidence of an acceleration of research activity aimed at developing artificial intelligence-powered tools for prediction and prevention of complications associated with diabetes. Our results indicate that artificial intelligence methods are being progressively established as suitable for use in clinical daily practice, as well as for the self-management of diabetes. Consequently, these methods provide powerful tools for improving patients’ quality of life.