Empiric validation of simulation models for estimating glucose meter performance criteria for moderate levels of glycemic control

Error simulation modeling to assess the effects of bias and precision on bilirubin measurements used to screen for neonatal hyperbilirubinemia

OBJECTIVES

Error simulation models have been used to understand the relationship between analytical performance and clinical outcomes. We developed an error simulation model to understand the effects of method bias and precision on misclassification rate for neonatal hyperbilirubinemia using an age-adjusted risk assessment tool.

METHODS

For each of 176 measured total bilirubin (TSB_M) values, 10,000 simulated total bilirubin (TB_S) values were generated at each combination of bias and precision conditions for coefficient of variation (CV) between 1 and 15%, and for biases between -51.3 μmol/L and 51.3 μmol/L (-3 and 3 mg/dL) fixed bias. TB_S values were analyzed to determine if they were in the same risk zone as the TSB_M value. We then calculated sensitivity and specificity for prediction of ≥75th percentile for postnatal age values as a function of assay bias and precision, and determined the rate of critical errors (≥95th percentile for age TSB_M with <75th percentile TB_S).

RESULTS

A sensitivity >95% for predicting ≥75th percentile bilirubin values was observed when there is a positive fixed bias of greater than 17.1 μmol/L (1.0 mg/dL) and CV is maintained ≤10%. A specificity >70% for predicting <75th percentile bilirubin values was observed when positive systematic bias was 17.1 μmol/L (1 mg/dL) or less at CV ≤ 10%. Critical errors did not occur with a frequency >0.2% until negative bias was -17.1 μmol/L (-1 mg/dL) or lower.

CONCLUSIONS

A positive systematic bias of 17.1 μmol/L (1 mg/dL) may be optimal for balancing sensitivity and specificity for predicting ≥75th percentile TSB values. Negative systematic bias should be avoided to allow detection of high risk infants and avoid critical classification errors.

Toward a Framework for Outcome-Based Analytical Performance Specifications: A Methodology Review of Indirect Methods for Evaluating the Impact of Measurement Uncertainty on Clinical Outcomes

BACKGROUND

For medical tests that have a central role in clinical decision-making, current guidelines advocate outcome-based analytical performance specifications. Given that empirical (clinical trial-style) analyses are often impractical or unfeasible in this context, the ability to set such specifications is expected to rely on indirect studies to calculate the impact of test measurement uncertainty on downstream clinical, operational, and economic outcomes. Currently, however, a lack of awareness and guidance concerning available alternative indirect methods is limiting the production of outcome-based specifications. Therefore, our aim was to review available indirect methods and present an analytical framework to inform future outcome-based performance goals.

CONTENT

A methodology review consisting of database searches and extensive citation tracking was conducted to identify studies using indirect methods to incorporate or evaluate the impact of test measurement uncertainty on downstream outcomes (including clinical accuracy, clinical utility, and/or costs). Eighty-two studies were identified, most of which evaluated the impact of imprecision and/or bias on clinical accuracy. A common analytical framework underpinning the various methods was identified, consisting of 3 key steps: (a) calculation of "true" test values; (b) calculation of measured test values (incorporating uncertainty); and (c) calculation of the impact of discrepancies between (a) and (b) on specified outcomes. A summary of the methods adopted is provided, and key considerations are discussed.

CONCLUSIONS

Various approaches are available for conducting indirect assessments to inform outcome-based performance specifications. This study provides an overview of methods and key considerations to inform future studies and research in this area.

Traceability to a primary reference measurement procedure (ID-LCMS); A key step in validating the clinical accuracy and safety of hospital blood glucose monitoring systems

OBJECTIVE

A key step in the evaluation of the accuracy of blood glucose monitoring systems (BGMS) is using a comparator method aligned to a high order definitive reference method. We describe how we achieved traceability to an isotope dilution liquid chromatography mass spectrometry (ID-LCMS) method. We used ID-LCMS to evaluate the accuracy and specificity of two hospital BGMS used in China.

METHOD

ID-LCMS was used to verify the calibration alignment of the laboratory plasma hexokinase reference method using NIST standard reference material and clinical samples. The ID-LCMS aligned hexokinase method was used to evaluate the clinical accuracy of two BGMS in hospitalized patients. System accuracy was evaluated using Chinese consensus guidelines. BGMS accuracy was also assessed with interference factors known to be present in critically ill patients' blood.

RESULTS

The laboratory plasma hexokinase reference method was shown to calibrate closely with ID-LCMS. Two BGMS demonstrated good correlation with this reference method. Only one BGMS met the Chinese guidelines. The interference factors didn't influence this BGMS but adversely affected the clinical accuracy of the other.

CONCLUSIONS

We advocate that our IDMS calibration alignment approach for ensuring the accuracy of the glucose reference method should be adopted in evaluations assessing the accuracy of blood glucose monitoring systems.

Assessment of the performance of blood glucose monitoring systems for monitoring dysglycaemia in neonatal patients

OBJECTIVE

To validate a three-step protocol that assesses the clinical risk associated with using blood glucose monitoring systems (BGMS) in neonates for the management of dysglycaemia.

METHOD

The three-step validation approach included confirmation of the accuracy of the reference method using National Institute of Standards and Technology (NIST) glucose standards, assessment of analytical risk performed on whole blood collected from paediatric patients routinely tested for glucose and a clinical risk assessment performed using heel stick capillary samples collected from 147 new-born babies and neonates admitted to intensive care. BGMS glucose measurements were compared with the NIST aligned laboratory reference method.

RESULTS

The accuracy of the laboratory reference method was confirmed with the NIST standards. Specificity studies demonstrated that the accuracy of one of the BGMS was affected, particularly, in the hypoglycaemic range, by known interference factors including haematocrit, ascorbic acid, lactose, galactose, N-acetylcysteine and glutathione. The accuracy of the other BGMS was unaffected. The clinical performance of this BGMS in neonates met the system accuracy criteria of Clinical and Laboratory Standards Institute (CLSI) POCT 12-A3 standard for evaluating hospital BGMS with 95.1% of glucose measurements within±0.67 mmol/L for samples ≤5.55 mmol/L and 95.6% within±12.5% for samples>5.55 mmol/L.

CONCLUSIONS

This three-step validation protocol provides a challenging approach for determining the accuracy and reliability of BGMS for managing dysglycaemia in neonates. StatStrip BGMS achieved analytical and clinical performance criteria confirming its suitability for use in neonates. We advocate that this validation approach should be considered for performance evaluations of both BGMS and continuous glucose monitoring systems going forward.

Accuracy of Capillary and Arterial Whole Blood Glucose Measurements Using a Glucose Meter in Patients under General Anesthesia in the Operating Room

BACKGROUND

The aim of this study was to evaluate the use of a glucose meter with surgical patients under general anesthesia in the operating room.

METHODS

Glucose measurements were performed intraoperatively on 368 paired capillary and arterial whole blood samples using a Nova StatStrip (Nova Biomedical, USA) glucose meter and compared with 368 reference arterial whole blood glucose measurements by blood gas analyzer in 196 patients. Primary outcomes were median bias (meter minus reference), percentage of glucose meter samples meeting accuracy criteria for subcutaneous insulin dosing as defined by Parkes error grid analysis for type 1 diabetes mellitus, and accuracy criteria for intravenous insulin infusion as defined by Clinical and Laboratory Standards Institute guidelines. Time under anesthesia, patient position, diabetes status, and other variables were studied to determine whether any affected glucose meter bias.

RESULTS

Median bias (interquartile range) was -4 mg/dl (-9 to 0 mg/dl), which did not differ from median arterial meter bias of -5 mg/dl (-9 to -1 mg/dl; P = 0.32). All of the capillary and arterial glucose meter values met acceptability criteria for subcutaneous insulin dosing, whereas only 89% (327 of 368) of capillary and 93% (344 of 368) arterial glucose meter values met accuracy criteria for intravenous insulin infusion. Time, patient position, and diabetes status were not associated with meter bias.

CONCLUSIONS

Capillary and arterial blood glucose measured using the glucose meter are acceptable for intraoperative subcutaneous insulin dosing. Whole blood glucose on the meter did not meet accuracy guidelines established specifically for more intensive (e.g., intravenous insulin) glycemic control in the acute care environment.

A Model of Self-Monitoring Blood Glucose Measurement Error

BACKGROUND

A reliable model of the probability density function (PDF) of self-monitoring of blood glucose (SMBG) measurement error would be important for several applications in diabetes, like testing in silico insulin therapies. In the literature, the PDF of SMBG error is usually described by a Gaussian function, whose symmetry and simplicity are unable to properly describe the variability of experimental data. Here, we propose a new methodology to derive more realistic models of SMBG error PDF.

METHODS

The blood glucose range is divided into zones where error (absolute or relative) presents a constant standard deviation (SD). In each zone, a suitable PDF model is fitted by maximum-likelihood to experimental data. Model validation is performed by goodness-of-fit tests. The method is tested on two databases collected by the One Touch Ultra 2 (OTU2; Lifescan Inc, Milpitas, CA) and the Bayer Contour Next USB (BCN; Bayer HealthCare LLC, Diabetes Care, Whippany, NJ). In both cases, skew-normal and exponential models are used to describe the distribution of errors and outliers, respectively.

RESULTS

Two zones were identified: zone 1 with constant SD absolute error; zone 2 with constant SD relative error. Goodness-of-fit tests confirmed that identified PDF models are valid and superior to Gaussian models used so far in the literature.

CONCLUSIONS

The proposed methodology allows to derive realistic models of SMBG error PDF. These models can be used in several investigations of present interest in the scientific community, for example, to perform in silico clinical trials to compare SMBG-based with nonadjunctive CGM-based insulin treatments.

Bedside Glucose Monitoring-Is it Safe? A New, Regulatory-Compliant Risk Assessment Evaluation Protocol in Critically Ill Patient Care Settings

Supplemental Digital Content is available in the text.

Objectives:

New data have emerged from ambulatory and acute care settings about adverse patient events, including death, attributable to erroneous blood glucose meter measurements and leading to questions over their use in critically ill patients. The U.S. Food and Drug Administration published new, more stringent guidelines for glucose meter manufacturers to evaluate the performance of blood glucose meters in critically ill patient settings. The primary objective of this international, multicenter, multidisciplinary clinical study was to develop and apply a rigorous clinical accuracy assessment algorithm, using four distinct statistical tools, to evaluate the clinical accuracy of a blood glucose monitoring system in critically ill patients.

Design:

Observational study.

Setting:

Five international medical and surgical ICUs.

Patients:

All patients admitted to critical care settings in the centers.

Interventions:

None.

Measurements and Main Results:

Glucose measurements were performed on 1,698 critically ill patients with 257 different clinical conditions and complex treatment regimens. The clinical accuracy assessment algorithm comprised four statistical tools to assess the performance of the study blood glucose monitoring system compared with laboratory reference methods traceable to a definitive standard. Based on POCT12-A3, the Clinical Laboratory Standards Institute standard for hospitals about hospital glucose meter procedures and performance, and Parkes error grid clinical accuracy performance criteria, no clinically significant differences were observed due to patient condition or therapy, with 96.1% and 99.3% glucose results meeting the respective criteria. Stratified sensitivity and specificity analysis (10 mg/dL glucose intervals, 50–150 mg/dL) demonstrated high sensitivity (mean = 95.2%, sd = ± 0.02) and specificity (mean = 95. 8%, sd = ± 0.03). Monte Carlo simulation modeling of the study blood glucose monitoring system showed low probability of category 2 and category 3 insulin dosing error, category 2 = 2.3% (41/1,815) and category 3 = 1.8% (32/1,815), respectively. Patient trend analysis demonstrated 99.1% (223/225) concordance in characterizing hypoglycemic patients.

Conclusions:

The multicomponent, clinical accuracy assessment algorithm demonstrated that the blood glucose monitoring system was acceptable for use in critically ill patient settings when compared to the central laboratory reference method. This clinical accuracy assessment algorithm is an effective tool for comprehensively assessing the validity of whole blood glucose measurement in critically ill patient care settings.

AMERICAN ASSOCIATION OF CLINICAL ENDOCRINOLOGISTS AND AMERICAN COLLEGE OF ENDOCRINOLOGY 2016 OUTPATIENT GLUCOSE MONITORING CONSENSUS STATEMENT

This document represents the official position of the American Association of Clinical Endocrinologists and American College of Endocrinology. Where there were no randomized controlled trials or specific U.S. FDA labeling for issues in clinical practice, the participating clinical experts utilized their judgment and experience. Every effort was made to achieve consensus among the committee members. Position statements are meant to provide guidance, but they are not to be considered prescriptive for any individual patient and cannot replace the judgment of a clinician.

Impact of Glucose Meter Error on Glycemic Variability and Time in Target Range During Glycemic Control After Cardiovascular Surgery

BACKGROUND

We retrospectively studied the impact of glucose meter error on the efficacy of glycemic control after cardiovascular surgery.

METHOD

Adult patients undergoing intravenous insulin glycemic control therapy after cardiovascular surgery, with 12-24 consecutive glucose meter measurements used to make insulin dosing decisions, had glucose values analyzed to determine glycemic variability by both standard deviation (SD) and continuous overall net glycemic action (CONGA), and percentage glucose values in target glucose range (110-150 mg/dL). Information was recorded for 70 patients during each of 2 periods, with different glucose meters used to measure glucose and dose insulin during each period but no other changes to the glycemic control protocol. Accuracy and precision of each meter were also compared using whole blood specimens from ICU patients.

RESULTS

Glucose meter 1 (GM1) had median bias of 11 mg/dL compared to a laboratory reference method, while glucose meter 2 (GM2) had a median bias of 1 mg/dL. GM1 and GM2 differed little in precision (CV = 2.0% and 2.7%, respectively). Compared to the period when GM1 was used to make insulin dosing decisions, patients whose insulin dose was managed by GM2 demonstrated reduced glycemic variability as measured by both SD (13.7 vs 21.6 mg/dL, P < .0001) and CONGA (13.5 vs 19.4 mg/dL, P < .0001) and increased percentage glucose values in target range (74.5 vs 66.7%, P = .002).

CONCLUSIONS

Decreasing glucose meter error (bias) was associated with decreased glycemic variability and increased percentage of values in target glucose range for patients placed on intravenous insulin therapy following cardiovascular surgery.

Performance of Cleared Blood Glucose Monitors

Cleared blood glucose monitor (BGM) systems do not always perform as accurately for users as they did to become cleared. We performed a literature review of recent publications between 2010 and 2014 that present data about the frequency of inaccurate performance using ISO 15197 2003 and ISO 15197 2013 as target standards. We performed an additional literature review of publications that present data about the clinical and economic risks of inaccurate BGMs for making treatment decisions or calibrating continuous glucose monitors (CGMs). We found 11 publications describing performance of 98 unique BGM systems. 53 of these 98 (54%) systems met ISO 15197 2003 and 31 of the 98 (32%) tested systems met ISO 15197 2013 analytical accuracy standards in all studies in which they were evaluated. Of the tested systems, 33 were identified by us as FDA-cleared. Among these FDA-cleared BGM systems, 24 out of 32 (75%) met ISO 15197 2003 and 15 out of 31 (48.3%) met ISO 15197 2013 in all studies in which they were evaluated. Among the non-FDA-cleared BGM systems, 29 of 65 (45%) met ISO 15197 2003 and 15 out of 65 (23%) met ISO 15197 2013 in all studies in which they were evaluated. It is more likely that an FDA-cleared BGM system, compared to a non-FDA-cleared BGM system, will perform according to ISO 15197 2003 (χ(2) = 6.2, df = 3, P = 0.04) and ISO 15197 2013 (χ(2) = 11.4, df = 3, P = 0.003). We identified 7 articles about clinical risks and 3 articles about economic risks of inaccurate BGMs. We conclude that a significant proportion of cleared BGMs do not perform at the level for which they were cleared or according to international standards of accuracy. Such poor performance leads to adverse clinical and economic consequences.

Retrospective evaluation of the accuracy of Roche AccuChek Inform and Nova StatStrip glucose meters when used on critically ill patients

BACKGROUND

More stringent accuracy guidelines for hospital-use glucose meters have recently been published, but it remains unclear whether glucose meters can meet these accuracy guidelines when measurement is performed on critically ill patients with fresh whole blood samples.

MATERIALS AND METHODS

We performed a retrospective evaluation of a conventional (Roche Diagnostics [Indianapolis, IN] AccuChek® Inform) and a newer-generation (Nova Biomedical [Waltham, MA] StatStrip®) glucose system by comparing paired (drawn within 5 min of each other) whole blood glucose meter and laboratory serum glucose values obtained from intensive care unit (ICU) patients. We also performed a prospective evaluation of the accuracy of the Nova StatStrip.

RESULTS

The median (interquartile range) bias between Roche AccuChek Inform and serum laboratory glucose measurements was 11 (6-18) mg/dL, compared with a median bias between the Nova StatStrip and serum glucose measurements of 1 (-5 to 5) mg/dL. StatStrip met International Organization for Standardization 15197 and Clinical and Laboratory Standards Institute (CLSI) POCT12-A3 accuracy guidelines using both retrospective and prospective datasets.

CONCLUSIONS

The newer-generation (StatStrip) glucose meter met more stringent CLSI POCT12-A3 accuracy criteria because of reduced bias compared with the previous-generation device. Reduced glucose meter bias led to fewer insulin dosing discrepancies when the insulin dose determined from serum glucose was compared with that determined from the glucose meter value using the institutional glycemic control protocol.