An Actor-Critic based controller for glucose regulation in type 1 diabetes

End-to-end offline reinforcement learning for glycemia control

The development of closed-loop systems for glycemia control in type I diabetes relies heavily on simulated patients. Improving the performances and adaptability of these close-loops raises the risk of over-fitting the simulator. This may have dire consequences, especially in unusual cases which were not faithfully - if at all - captured by the simulator. To address this, we propose to use model-free offline RL agents, trained on real patient data, to perform the glycemia control. To further improve the performances, we propose an end-to-end personalization pipeline, which leverages offline-policy evaluation methods to remove altogether the need of a simulator, while still enabling an estimation of clinically relevant metrics for diabetes.

Advanced decision support system for individuals with diabetes on multiple daily injections therapy using reinforcement learning and nearest-neighbors: In-silico and clinical results

Many individuals with diabetes on multiple daily insulin injections therapy use carbohydrate ratios (CRs) and correction factors (CFs) to determine mealtime and correction insulin boluses. The CRs and CFs vary over time due to physiological changes in individuals' response to insulin. Errors in insulin dosing can lead to life-threatening abnormal glucose levels, increasing the risk of retinopathy, neuropathy, and nephropathy. Here, we present a novel learning algorithm that uses Q-learning to track optimal CRs and uses nearest-neighbors based Q-learning to track optimal CFs. The learning algorithm was compared with the run-to-run algorithm A and the run-to-run algorithm B, both proposed in the literature, over an 8-week period using a validated simulator with a realistic scenario created with suboptimal CRs and CFs values, carbohydrate counting errors, and random meals sizes at random ingestion times. From Week 1 to Week 8, the learning algorithm increased the percentage of time spent in target glucose range (4.0 to 10.0 mmol/L) from 51 % to 64 % compared to 61 % and 58 % with the run-to-run algorithm A and the run-to-run algorithm B, respectively. The learning algorithm decreased the percentage of time spent below 4.0 mmol/L from 9 % to 1.9 % compared to 3.4 % and 2.3 % with the run-to-run algorithm A and the run-to-run algorithm B, respectively. The algorithm was also assessed by comparing its recommendations with (i) the endocrinologist's recommendations on two type 1 diabetes individuals over a 16-week period and (ii) real-world individuals' therapy settings changes of 23 individuals (19 type 2 and 4 type 1) over an 8-week period using the commercial Bigfoot Unity Diabetes Management System. The full agreements (i) were 89 % and 76 % for CRs and CFs for the type 1 diabetes individuals and (ii) was 62 % for mealtime doses for the individuals on the commercial Bigfoot system. Therefore, the proposed algorithm has the potential to improve glucose control in individuals with type 1 and type 2 diabetes.

A Reinforcement Learning Based System for Blood Glucose Control without Carbohydrate Estimation in Type 1 Diabetes: In Silico Validation

Type 1 Diabetes (T1D) is a chronic autoimmune disease, which requires the use of exogenous insulin for glucose regulation. In current hybrid closed-loop systems, meal entry is manual which adds cognitive burden to the persons living with T1D. In this study, we proposed a control system based on Proximal Policy Optimisation (PPO) that controls both basal and bolus insulin infusion and only requires meal announcement, thus eliminating the need for carbohydrate estimation. We evaluated the system on a challenging meal scenario, using an open-source simulator based on the UVA/Padova 2008 model and achieved a mean Time in Range value of 65% for the adult subject cohort, while maintaining a moderate hypoglycemic and hyperglycemic risk profile. The approach shows promise and welcomes further research towards the translation to a real-life artificial pancreas. Clinical relevance- This was an in-silico analysis towards the development of an autonomous artificial pancreas system for glucose control. The proposed system show promise in eliminating the need for estimating the carbohydrate content in meals.

Long-term use of the hybrid artificial pancreas by adjusting carbohydrate ratios and programmed basal rate: A reinforcement learning approach

BACKGROUND AND OBJECTIVES

The hybrid artificial pancreas regulates glucose levels in people with type 1 diabetes. It delivers (i) insulin boluses at meal times based on the meals' carbohydrate content and the carbohydrate ratios (CRs) and (ii) insulin basal, between meals and at night, continuously modulated around individual-specific programmed basal rate. The CRs and programmed basal rate significantly vary between individuals and within the same individual with type 1 diabetes, and using suboptimal values in the hybrid artificial pancreas may degrade glucose control. We propose a reinforcement learning algorithm to adaptively optimize CRs and programmed basal rate to improve the performance of the hybrid artificial pancreas.

METHODS

The proposed reinforcement learning algorithm was designed using the Q-learning approach. The algorithm learns the optimal actions (CRs and programmed basal rate) by applying them to the individual's state (previous day's glucose levels and insulin delivery) based on an exploration and exploitation trade-off. First, outcomes from our simulator were compared to those of a clinical study in 23 individuals with type 1 diabetes and have yielded similar results. Second, the learning algorithm was tested using the simulator with two scenarios. Scenario 1 has fixed meal sizes and ingestion times and scenario 2 has a more realistic eating behavior with random meal sizes, ingestion times, and carbohydrate counting errors.

RESULTS

After about five weeks, the reinforcement learning algorithm improved the percentage of time spent in target range from 67% to 86.7% in scenario 1 and 65.5% to 86% in scenario 2. The percentage of time spent below 4.0 mmol/L decreased from 9% to 0.9% in scenario 1 and 9.5% to 1.1% in scenario 2.

CONCLUSIONS

Results indicate that the proposed algorithm has the potential to improve glucose control in people with type 1 diabetes using the hybrid artificial pancreas. The proposed algorithm is a key in making the hybrid artificial pancreas adaptive for the long-term real life outpatient studies.

Intelligent automated drug administration and therapy: future of healthcare

In the twenty-first century, the collaboration of control engineering and the healthcare sector has matured to some extent; however, the future will have promising opportunities, vast applications, and some challenges. Due to advancements in processing speed, the closed-loop administration of drugs has gained popularity for critically ill patients in intensive care units and routine life such as personalized drug delivery or implantable therapeutic devices. For developing a closed-loop drug delivery system, the control system works with a group of technologies like sensors, micromachining, wireless technologies, and pharmaceuticals. Recently, the integration of artificial intelligence techniques such as fuzzy logic, neural network, and reinforcement learning with the closed-loop drug delivery systems has brought their applications closer to fully intelligent automatic healthcare systems. This review's main objectives are to discuss the current developments, possibilities, and future visions in closed-loop drug delivery systems, for providing treatment to patients suffering from chronic diseases. It summarizes the present insight of closed-loop drug delivery/therapy for diabetes, gastrointestinal tract disease, cancer, anesthesia administration, cardiac ailments, and neurological disorders, from a perspective to show the research in the area of control theory.

In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus

In this paper, we test and evaluate policy gradient reinforcement learning for automated blood glucose control in patients with Type 1 Diabetes Mellitus. Recent research has shown that reinforcement learning is a promising approach to accommodate the need for individualized blood glucose level control algorithms. The motivation for using policy gradient algorithms comes from the fact that adaptively administering insulin is an inherently continuous task. Policy gradient algorithms are known to be superior in continuous high-dimensional control tasks. Previously, most of the approaches for automated blood glucose control using reinforcement learning has used a finite set of actions. We use the Trust-Region Policy Optimization algorithm in this work. It represents the state of the art for deep policy gradient algorithms. The experiments are carried out in-silico using the Hovorka model, and stochastic behavior is modeled through simulated carbohydrate counting errors to illustrate the full potential of the framework. Furthermore, we use a model-free approach where no prior information about the patient is given to the algorithm. Our experiments show that the reinforcement learning agent is able to compete with and sometimes outperform state-of-the-art model predictive control in blood glucose regulation.

Reinforcement learning for intelligent healthcare applications: A survey

Discovering new treatments and personalizing existing ones is one of the major goals of modern clinical research. In the last decade, Artificial Intelligence (AI) has enabled the realization of advanced intelligent systems able to learn about clinical treatments and discover new medical knowledge from the huge amount of data collected. Reinforcement Learning (RL), which is a branch of Machine Learning (ML), has received significant attention in the medical community since it has the potentiality to support the development of personalized treatments in accordance with the more general precision medicine vision. This report presents a review of the role of RL in healthcare by investigating past work, and highlighting any limitations and possible future contributions.

Preventing undesirable behavior of intelligent machines

Intelligent machines using machine learning algorithms are ubiquitous, ranging from simple data analysis and pattern recognition tools to complex systems that achieve superhuman performance on various tasks. Ensuring that they do not exhibit undesirable behavior-that they do not, for example, cause harm to humans-is therefore a pressing problem. We propose a general and flexible framework for designing machine learning algorithms. This framework simplifies the problem of specifying and regulating undesirable behavior. To show the viability of this framework, we used it to create machine learning algorithms that precluded the dangerous behavior caused by standard machine learning algorithms in our experiments. Our framework for designing machine learning algorithms simplifies the safe and responsible application of machine learning.

A Dual Mode Adaptive Basal-Bolus Advisor Based on Reinforcement Learning

Self-monitoring of blood glucose (SMBG) and continuous glucose monitoring (CGM) are commonly used by type 1 diabetes (T1D) patients to measure glucose concentrations. The proposed adaptive basal-bolus algorithm (ABBA) supports inputs from either SMBG or CGM devices to provide personalised suggestions for the daily basal rate and prandial insulin doses on the basis of the patients' glucose level on the previous day. The ABBA is based on reinforcement learning, a type of artificial intelligence, and was validated in silico with an FDA-accepted population of 100 adults under different realistic scenarios lasting three simulated months. The scenarios involve three main meals and one bedtime snack per day, along with different variabilities and uncertainties for insulin sensitivity, mealtime, carbohydrate amount, and glucose measurement time. The results indicate that the proposed approach achieves comparable performance with CGM or SMBG as input signals, without influencing the total daily insulin dose. The results are a promising indication that AI algorithmic approaches can provide personalised adaptive insulin optimization and achieve glucose control-independent of the type of glucose monitoring technology.

Artificial Intelligence for Diabetes Management and Decision Support: Literature Review

Background

Artificial intelligence methods in combination with the latest technologies, including medical devices, mobile computing, and sensor technologies, have the potential to enable the creation and delivery of better management services to deal with chronic diseases. One of the most lethal and prevalent chronic diseases is diabetes mellitus, which is characterized by dysfunction of glucose homeostasis.

Objective

The objective of this paper is to review recent efforts to use artificial intelligence techniques to assist in the management of diabetes, along with the associated challenges.

Methods

A review of the literature was conducted using PubMed and related bibliographic resources. Analyses of the literature from 2010 to 2018 yielded 1849 pertinent articles, of which we selected 141 for detailed review.

Results

We propose a functional taxonomy for diabetes management and artificial intelligence. Additionally, a detailed analysis of each subject category was performed using related key outcomes. This approach revealed that the experiments and studies reviewed yielded encouraging results.

Conclusions

We obtained evidence of an acceleration of research activity aimed at developing artificial intelligence-powered tools for prediction and prevention of complications associated with diabetes. Our results indicate that artificial intelligence methods are being progressively established as suitable for use in clinical daily practice, as well as for the self-management of diabetes. Consequently, these methods provide powerful tools for improving patients’ quality of life.

Model-Free Machine Learning in Biomedicine: Feasibility Study in Type 1 Diabetes

Although reinforcement learning (RL) is suitable for highly uncertain systems, the applicability of this class of algorithms to medical treatment may be limited by the patient variability which dictates individualised tuning for their usually multiple algorithmic parameters. This study explores the feasibility of RL in the framework of artificial pancreas development for type 1 diabetes (T1D). In this approach, an Actor-Critic (AC) learning algorithm is designed and developed for the optimisation of insulin infusion for personalised glucose regulation. AC optimises the daily basal insulin rate and insulin:carbohydrate ratio for each patient, on the basis of his/her measured glucose profile. Automatic, personalised tuning of AC is based on the estimation of information transfer (IT) from insulin to glucose signals. Insulin-to-glucose IT is linked to patient-specific characteristics related to total daily insulin needs and insulin sensitivity (SI). The AC algorithm is evaluated using an FDA-accepted T1D simulator on a large patient database under a complex meal protocol, meal uncertainty and diurnal SI variation. The results showed that 95.66% of time was spent in normoglycaemia in the presence of meal uncertainty and 93.02% when meal uncertainty and SI variation were simultaneously considered. The time spent in hypoglycaemia was 0.27% in both cases. The novel tuning method reduced the risk of severe hypoglycaemia, especially in patients with low SI.

Personalized tuning of a reinforcement learning control algorithm for glucose regulation

Artificial pancreas is in the forefront of research towards the automatic insulin infusion for patients with type 1 diabetes. Due to the high inter- and intra-variability of the diabetic population, the need for personalized approaches has been raised. This study presents an adaptive, patient-specific control strategy for glucose regulation based on reinforcement learning and more specifically on the Actor-Critic (AC) learning approach. The control algorithm provides daily updates of the basal rate and insulin-to-carbohydrate (IC) ratio in order to optimize glucose regulation. A method for the automatic and personalized initialization of the control algorithm is designed based on the estimation of the transfer entropy (TE) between insulin and glucose signals. The algorithm has been evaluated in silico in adults, adolescents and children for 10 days. Three scenarios of initialization to i) zero values, ii) random values and iii) TE-based values have been comparatively assessed. The results have shown that when the TE-based initialization is used, the algorithm achieves faster learning with 98%, 90% and 73% in the A+B zones of the Control Variability Grid Analysis for adults, adolescents and children respectively after five days compared to 95%, 78%, 41% for random initialization and 93%, 88%, 41% for zero initial values. Furthermore, in the case of children, the daily Low Blood Glucose Index reduces much faster when the TE-based tuning is applied. The results imply that automatic and personalized tuning based on TE reduces the learning period and improves the overall performance of the AC algorithm.

GoCARB in the Context of an Artificial Pancreas

BACKGROUND

In an artificial pancreas (AP), the meals are either manually announced or detected and their size estimated from the blood glucose level. Both methods have limitations, which result in suboptimal postprandial glucose control. The GoCARB system is designed to provide the carbohydrate content of meals and is presented within the AP framework.

METHOD

The combined use of GoCARB with a control algorithm is assessed in a series of 12 computer simulations. The simulations are defined according to the type of the control (open or closed loop), the use or not-use of GoCARB and the diabetics' skills in carbohydrate estimation.

RESULTS

For bad estimators without GoCARB, the percentage of the time spent in target range (70-180 mg/dl) during the postprandial period is 22.5% and 66.2% for open and closed loop, respectively. When the GoCARB is used, the corresponding percentages are 99.7% and 99.8%. In case of open loop, the time spent in severe hypoglycemic events (<50 mg/dl) is 33.6% without the GoCARB and is reduced to 0.0% when the GoCARB is used. In case of closed loop, the corresponding percentage is 1.4% without the GoCARB and is reduced to 0.0% with the GoCARB.

CONCLUSION

The use of GoCARB improves the control of postprandial response and glucose profiles especially in the case of open loop. However, the most efficient regulation is achieved by the combined use of the control algorithm and the GoCARB.

Applicability results of a nonlinear model-based robust blood glucose control algorithm

INTRODUCTION

Generating optimal control algorithms for an artificial pancreas is an intensively researched problem. The available models are all nonlinear and rather complex. Model predictive control or run-to-run-based methodologies have proven to be efficient solutions for individualized treatment of type 1 diabetes mellitus (T1DM). However, the controller has to ensure safety and stability under all circumstances. Robust control methods seek to provide this safety and guarantee to handle even the worst-case situations and, hence, to generalize and complement results obtained by individualized control algorithms.

METHODS

Modern robust (e.g., Hinf) control is a linear model-based methodology that we have combined with the nonlinear model-based linear parameter varying technique. The control algorithm was designed on the high-complexity modified nonlinear glucose-insulin model of Sorensen, and it was compared step-by-step with linear model-based Hinf control results published in the literature. The applicability of the developed algorithm was tested first on a control cohort of 10 healthy persons' oral glucose tolerance test results and then on a large meal absorption profile adapted from the literature. In the latter case, two preliminary virtual patients were generated based on 1-1 week real continuous glucose monitor measurements.

RESULTS

We have found that the algorithm avoids hypoglycemia (not caused by physical activity or stress) independently from the considered absorption profiles.

CONCLUSION

Use of hard constraints proved their efficiency in fitting blood glucose level within a defined interval. However, in the future, more data of different T1DM patients will be collected and tested, including dynamic absorption model and in silico tests on validated simulators.