Analysis of the Accuracy and Performance of a Continuous Glucose Monitoring Sensor Prototype: An In-Silico Study Using the UVA/PADOVA Type 1 Diabetes Simulator

Outcome-based analytical performance specifications: current status and future challenges

Analytical performance specifications (APS) based on outcomes refer to how 'good' the analytical performance of a test needs to be to do more good than harm to the patient. Analytical performance of a measurand affects its clinical performance. Without first setting clinical performance requirements, it is difficult to define how good analytically the test needs to be to meet medical needs. As testing is indirectly linked to health outcomes through clinical decisions on patient management, often simulation-based studies are used to assess the impact of analytical performance on the probability of clinical outcomes which is then translated to Model 1b APS according to the Milan consensus. This paper discusses the related key definitions, concepts and considerations that should assist in finding the most appropriate methods for deriving Model 1b APS. We review the advantages and limitations of published methods and discuss the criteria for transferability of Model 1b APS to different settings. We consider that the definition of the clinically acceptable misclassification rate is central to Model 1b APS. We provide some examples and guidance on a more systematic approach for first defining the clinical performance requirements for tests and we also highlight a few ideas to tackle the future challenges associated with providing outcome-based APS for laboratory testing.

Developing the UVA/Padova Type 1 Diabetes Simulator: Modeling, Validation, Refinements, and Utility

Arguably, diabetes mellitus is one of the best quantified human conditions. In the past 50 years, the metabolic monitoring technologies progressed from occasional assessment of average glycemia via HbA1c, through episodic blood glucose readings, to continuous glucose monitoring (CGM) producing data points every few minutes. The high-temporal resolution of CGM data enabled increasingly intensive treatments, from decision support assisting insulin injection or oral medication, to automated closed-loop control, known as the "artificial pancreas." Throughout this progress, mathematical models and computer simulation of the human metabolic system became indispensable for the technological progress of diabetes treatment, enabling every step, from assessment of insulin sensitivity via the now classic Minimal Model of Glucose Kinetics, to in silico trials replacing animal experiments, to automated insulin delivery algorithms. In this review, we follow these developments, beginning with the Minimal Model, which evolved through the years to become large and comprehensive and trigger a paradigm change in the design of diabetes optimization strategies: in 2007, we introduced a sophisticated model of glucose-insulin dynamics and a computer simulator equipped with a "population" of N = 300 in silico "subjects" with type 1 diabetes. In January 2008, in an unprecedented decision, the Food and Drug Administration (FDA) accepted this simulator as a substitute to animal trials for the pre-clinical testing of insulin treatment strategies. This opened the field for rapid and cost-effective development and pre-clinical testing of new treatment approaches, which continues today. Meanwhile, animal experiments for the purpose of designing new insulin treatment algorithms have been abandoned.

Generation of post-meal insulin correction boluses in type 1 diabetes simulation models for in-silico clinical trials: More realistic scenarios obtained using a decision tree approach

BACKGROUND AND OBJECTIVE

In type 1 diabetes (T1D) research, in-silico clinical trials (ISCTs) notably facilitate the design/testing of new therapies. Published simulation tools embed mathematical models of blood glucose (BG) and insulin dynamics, continuous glucose monitoring (CGM) sensors, and insulin treatments, but lack a realistic description of some aspects of patient lifestyle impacting on glucose control. Specifically, to effectively simulate insulin correction boluses, required to treat post-meal hyperglycemia (BG > 180 mg/dL), the timing of the bolus may be influenced by subjects' behavioral attitudes. In this work, we develop an easily interpretable model of the variability of correction bolus timing observed in real data, and embed it into a popular simulation tool for ISCTs.

METHODS

Using data collected in 196 adults with T1D monitored in free-living conditions, we trained a decision tree (DT) model to classify whether a correction bolus is injected in a future time window, based on predictors collected back in time, related to CGM data, previous insulin boluses and subject's characteristics. The performance was compared to that of a logistic regression classifier with LASSO regularization (LC), trained on the same dataset. After validation, the DT was embedded within a popular T1D simulation tool and an ISCT was performed to compare the simulated correction boluses against those observed in a subset of data not used for model training.

RESULTS

The DT provided better classification performance (accuracy: 0.792, sensitivity: 0.430, specificity: 0.878, precision: 0.455) than the LC and presented good interpretability. The most predictive features were related to CGM (and its temporal variations), time since the last insulin bolus, and time of the day. The correction boluses simulated by the DT, after implementation in the simulation tool, showed a good agreement with real-world data.

CONCLUSIONS

The DT developed in this work represents a simple set of rules to mimic the same timing of correction boluses observed on real data. The inclusion of the model in simulation tools allows investigators to perform ISCTs that more realistically represent the patient behavior in taking correction boluses and the post-prandial BG response. In the future, more complex models can be investigated.

Integrating an Islet-Based Biosensor in the Artificial Pancreas: In Silico Proof-of-Concept

OBJECTIVE

Current treatment of type 1 diabetes by closed-loop approaches depends on continuous glucose monitoring. However, glucose readings alone are insufficient for an artificial pancreas to truthfully restore glucose homeostasis where additional physiological regulators of insulin secretion play a considerable role. Previously, we have developed an electrophysiological biosensor of pancreatic islet activity, which integrates these additional regulators through electrical measurement. This work aims at investigating the performance of the biosensor in a blood glucose control loop, to establish an in silico proof-of-concept.

METHODS

Two islet algorithm models were identified on experimental data recorded with the biosensor. First, we validated electrical measurement as a means to exploit the inner regulation capabilities of islets for intravenous glucose measurement and insulin infusion. Then, an artificial pancreas integrating the islet-based biosensor was compared to standard treatment approaches using subcutaneous routes. The closed-loop simulations were performed in the UVA/Padova T1DM Simulator where a series of realistic meal scenarios were applied to virtual diabetic patients.

RESULTS

With intravenous routes, the endogenous islet algorithms successfully restored glucose homeostasis for all patient categories (mean time in range exceeds 90%) while mitigating the risk of adverse glycaemic events (mean BGI < 2). Using subcutaneous routes, the biosensor-based artificial pancreas was as performing as standard treatments, and outperformed them under challenging conditions.

CONCLUSION

This work validates the concept of using pancreatic islets algorithms in an artificial pancreas in silico.

SIGNIFICANCE

Pancreatic islet endogenous algorithms obtained via an electrophysiological biosensor successfully regulate blood glucose levels of virtual type 1 diabetic patients.

Mathematical Models of Meal Amount and Timing Variability With Implementation in the Type-1 Diabetes Patient Decision Simulator

BACKGROUND

In type 1 diabetes (T1D) research, in-silico clinical trials (ISCTs) have proven effective in accelerating the development of new therapies. However, published simulators lack a realistic description of some aspects of patient lifestyle which can remarkably affect glucose control. In this paper, we develop a mathematical description of meal carbohydrates (CHO) amount and timing, with the aim to improve the meal generation module in the T1D Patient Decision Simulator (T1D-PDS) published in Vettoretti et al.

METHODS

Data of 32 T1D subjects under free-living conditions for 4874 days were used. Univariate probability density function (PDF) parametric models with different candidate shapes were fitted, individually, against sample distributions of: CHO amounts of breakfast (CHOB), lunch (CHOL), dinner (CHOD), and snack (CHOS); breakfast timing (TB); and time between breakfast-lunch (TBL) and between lunch-dinner (TLD). Furthermore, a support vector machine (SVM) classifier was developed to predict the occurrence of a snack in future fixed-length time windows. Once embedded inside the T1D-PDS, an ISCT was performed.

RESULTS

Resulting PDF models were: gamma (CHOB, CHOS), lognormal (CHOL, TB), loglogistic (CHOD), and generalized-extreme-values (TBL, TLD). The SVM showed a classification accuracy of 0.8 over the test set. The distributions of simulated meal data were not statistically different from the distributions of the real data used to develop the models (α = 0.05).

CONCLUSIONS

The models of meal amount and timing variability developed are suitable for describing real data. Their inclusion in modules that describe patient behavior in the T1D-PDS can permit investigators to perform more realistic, reliable, and insightful ISCTs.

Toward a Framework for Outcome-Based Analytical Performance Specifications: A Methodology Review of Indirect Methods for Evaluating the Impact of Measurement Uncertainty on Clinical Outcomes

BACKGROUND

For medical tests that have a central role in clinical decision-making, current guidelines advocate outcome-based analytical performance specifications. Given that empirical (clinical trial-style) analyses are often impractical or unfeasible in this context, the ability to set such specifications is expected to rely on indirect studies to calculate the impact of test measurement uncertainty on downstream clinical, operational, and economic outcomes. Currently, however, a lack of awareness and guidance concerning available alternative indirect methods is limiting the production of outcome-based specifications. Therefore, our aim was to review available indirect methods and present an analytical framework to inform future outcome-based performance goals.

CONTENT

A methodology review consisting of database searches and extensive citation tracking was conducted to identify studies using indirect methods to incorporate or evaluate the impact of test measurement uncertainty on downstream outcomes (including clinical accuracy, clinical utility, and/or costs). Eighty-two studies were identified, most of which evaluated the impact of imprecision and/or bias on clinical accuracy. A common analytical framework underpinning the various methods was identified, consisting of 3 key steps: (a) calculation of "true" test values; (b) calculation of measured test values (incorporating uncertainty); and (c) calculation of the impact of discrepancies between (a) and (b) on specified outcomes. A summary of the methods adopted is provided, and key considerations are discussed.

CONCLUSIONS

Various approaches are available for conducting indirect assessments to inform outcome-based performance specifications. This study provides an overview of methods and key considerations to inform future studies and research in this area.