Development of the Diabetes Technology Society Blood Glucose Monitor System Surveillance Protocol

Post-Market Surveillance Assessment of the Clinical Accuracy of a Blood Glucose Monitoring System with an Improved Algorithm for Enhanced Product Performance

BACKGROUND

On-going manufacturer-led post-market surveillance (PMS), assessing the clinical accuracy of blood glucose monitoring (BGM) systems, is critical to substantiate the performance of such products for people with diabetes.

MATERIALS AND METHODS

Batches of Verio test-strip product were randomly and routinely selected over the period from launch of an improved-algorithm product to reporting date and sent to 3 clinic sites for clinician-led accuracy assessment. Accuracy is reported as per recently adopted FDA guidance for BGM systems, EN ISO 15197:2015 and MARD/MAD (Mean absolute relative difference/Mean absolute difference).

RESULTS

Thirty-three individual test-strip batches were evaluated corresponding to 506 unique donors. Accuracy performance - FDA: 98.9% of values within ±15% of comparator; ISO: 99.0% within ±15 mg/dL or ±15% at <100 mg/dL (<5.55 mmol/L) or ≥100 mg/dL (≥5.55 mmol/L) glucose, respectively. Overall MARD was 4.19% with a MARD range of 3.54%-5.73% across all test strip batches.

CONCLUSIONS

This post-market surveillance program demonstrates the new BGM system consistently meets measures of clinical accuracy specified by regulators. This program supports a growing demand by regulators for real-world evidence demonstrating consistent in-market product efficacy as opposed to the current largely passive approach that relies on assessment of reports filed by device users.

Five-Year Two-Center Retrospective Comparison of Central Laboratory Glucose to GEM 4000 and ABL 800 Blood Glucose: Demonstrating the (In)adequacy of Blood Gas Glucose

PURPOSE

To evaluate the glucose assays of two blood gas analyzers (BGAs) in intensive care unit (ICU) patients by comparing ICU BGA glucoses to central laboratory (CL) glucoses of almost simultaneously drawn specimens.

METHODS

Data repositories provided five years of ICU BGA glucoses and contemporaneously drawn CL glucoses from a Calgary, Alberta ICU equipped with IL GEM 4000 and CL Roche Cobas 8000-C702, and an Edmonton, Alberta ICU equipped with Radiometer ABL 800 and CL Beckman-Coulter DxC. Blood glucose analyzer and CL glucose differences were evaluated if they were both drawn either within ±15 or ±5 minutes. Glucose differences were assessed graphically and quantitatively with simple run charts and the surveillance error grid (SEG) and quantitatively with the 2016 Food and Drug Administration guidance document, with ISO 15197 and SEG statistical summaries. As the GEM glucose exhibits diurnal variation, CL-arterial blood gas (ABG) differences were evaluated according to time of day.

RESULTS

Compared to the GEM glucoses measured between 0200 and 0800, the run charts of (GEM-CL) glucose demonstrate significant outliers between 0800 and 0200 which are identified as moderate to severe clinical outliers by SEG analysis (P < .002 and P < .0005 for 5- and 15-minute intervals). Over the entire 24-hour period, the rates of moderate to severe glucose clinical outliers are 3.5/1000 (GEM) and 0.6/1000 glucoses (ABL), respectively, using the 15-minute interval (P < .0001).

DISCUSSION

The GEM ABG glucose is associated with a higher frequency of moderate to severe glucose clinical outliers, especially between 0800 and 0200, increased CL testing and higher average patient glucoses.

System Accuracy Assessment of a Combined Invasive and Noninvasive Glucometer

BACKGROUND

The pain associated with pricking the fingertip for blood glucose self-testing is considered to be a major burden in diabetes treatment. This study was performed to evaluate the system accuracy of the invasive TensorTip Combo Glucometer (CoG) device component in accordance with ISO15197:2015 requirements and to explore the accuracy of the noninvasive tissue glucose prediction component.

METHODS

One hundred samples were obtained from people with type 1 and type 2 diabetes and healthy volunteers (43 females, 57 males; age: 53 ± 16 years), with glucose distribution as requested by the ISO standard. Three strip lots were tested twice by healthcare professionals in comparison to YSI 2300 Stat Plus reference method followed by a noninvasive tissue glucose reading (NI-CoG). Mean Absolute (Relative) Difference (MARD) was calculated and a consensus error grid (CEG) analysis was performed.

RESULTS

The ISO system accuracy criteria were met with the invasive strip technology by 586/600 of the data points (97.1%) and for each strip lot separately. All invasive results (100%) were within CEG-zone A and total MARD was calculated to be 7.1%. With the noninvasive reading, 99% of raw data points were in A + B (91.1% and 7.8%), and the total MARD was calculated to be 18.1%.

DISCUSSION

The invasive component of the CoG device was shown to be in full compliance with the current ISO15197 criteria. Good results were also obtained with the NI-CoG tissue glucose prediction. This noninvasive technology would potentially be suitable for frequent pain-free glucose monitoring in many people with diabetes.

System accuracy assessments with a blood glucose meter with combined glucose and ß-hydroxybutyrate measurement capabilities

Background: We evaluated the Wellion Galileo GLU/KET blood and ketone (ß-Hydroxybutyrate, ß-OHB) meter to demonstrate that it meets ISO15107:2015 regulatory approval criteria. Research Design and Methods: A total of 100 subjects (52 female, age: 30 to 84 years, diabetes: 10 type 1/90 type 2) with blood glucose levels distributed over the entire measurement range as required by the ISO15197 protocol were tested (double determinations with 3 strip lots and two devices). A similar test protocol was followed to test ß-OHB strip performance (reference devices: YSI 2300plus for glucose and STANBIO ß-HOB LiquiColor TestKit for ß-OHB). Precision was tested for glucose with 3 blood glucose concentrations (ß-OHB: 2 control solutions). Results: All glucose test-strip lots met the strict ISO acceptance criteria. Mean absolute relative difference (MARD) was 4.9% and all data pairs were in zone A of the consensus error grid. The ß-OHB test-strips also met the pre-defined acceptance criteria. Within-run and between-run precision was calculated to be 2.3% and 0.7% for the glucose strips (3.7%/0.8% for the ketone strips). Conclusions: When tested according to the ISO15197:2015 guideline, the device showed very accurate measurement performance for glucose and ß-OHB testing and fully met regulatory accuracy approval criteria.

Analysis of "Capillary and Venous Blood Glucose Accuracy in Blood Glucose Meters Versus Reference Standards: The Impact of Study Design on Accuracy Evaluations"

The article by Macleod and coauthors outlines an accuracy study of two blood glucose monitoring systems (BGMSs) against glucose-oxidase- and hexokinase-based assays showing how the reference/comparison method and inappropriate sample comparisons can affect accuracy conclusions. The dangers of independent institutions producing inappropriate conclusions when the methodology used for product regulatory or registration accuracy requirements is not according to best practice are stressed. The authors highlight several important aspects of a multistep accuracy evaluation protocol. However, it is essential that anyone undertaking or reporting BGMS accuracy studies should have a clear understanding of each and every protocol point, best practice, and how each can influence accuracy conclusions. Claims against regulatory accuracy criteria should be made only if the detailed specified protocol design and analysis is followed.

Efficacy and Safety of Degludec Compared to Glargine 300 Units/mL in Insulin-Experienced Patients With Type 2 Diabetes: Trial Protocol Amendment (NCT03078478)

BACKGROUND

A head-to-head trial (NCT03078478) between insulin degludec and insulin glargine U300 with the primary objective of comparing the risk of hypoglycemia is being conducted. During trial conduct, safety concerns related to the glycemic data collection system led to a postinitiation protocol amendment, described here.

METHODS

This randomized (1:1), open-label, treat-to-target, multinational trial was initiated in March 2017 with a planned treatment period of 52 weeks (16 weeks titration + 36 weeks maintenance). Overall, ~1600 insulin-experienced patients at risk of developing hypoglycemia based on predefined risk factors were included. The protocol amendment implemented in February 2018 resulted in assuring patient safety and an extension of the total treatment period up to 88 weeks (16 weeks titration + variable maintenance 1 + 36 weeks maintenance 2). The original glycemic data collection system (MyGlucoHealth blood glucose meter + electronic diary) was discontinued because of safety concerns and replaced with an Abbott blood glucose meter and paper diary to collect self-measured blood glucose and hypoglycemic episodes. The primary endpoint of number of severe or blood-glucose confirmed symptomatic hypoglycemic episodes will be evaluated with the same analysis duration and statistical methods as the original protocol. Only relevant changes were implemented to maintain patient safety while permitting evaluation of the scientific objectives of the trial.

CONCLUSIONS

These observations highlight the importance of safety surveillance during trial conduct despite the use of currently marketed glucose monitoring devices. The prompt protocol amendment and ensuing actions ensured that the scientific integrity of the trial was not compromised.

System Accuracy Assessment of a Blood Glucose Meter With Wireless Internet Access Associated With Unusual Hypoglycemia Patterns in Clinical Trials

BACKGROUND

In recent randomized clinical trials, an unusual reporting pattern of glycemic data and hypoglycemic events potentially related to an internet enabled blood glucose meter (MyGlucoHealth, BGM) was observed. Therefore, this clinical study was conducted to evaluate the system accuracy of the BGM in accordance with the ISO15197:2015 guidelines with additional data collection.

METHODS

To investigate system accuracy, 10 of 3088 devices and 6 of 23 strip lots, used in the trials, were selected by a randomization procedure and a standard repeatability assessment. YSI 2300 STAT Plus was used as the standard reference method. The samples were distributed as per the ISO15197:2015 recommendations with 20 additional samples in the hypoglycemic range. Each sample was tested with 6 devices and 6 strip lots with double determinations.

RESULTS

Overall, 121 subjects with blood glucose values 26-423 mg/dL were analyzed, resulting in 1452 data points. In all, 186/1452 readings (12.8%) did not meet the ISO acceptance criteria. Data evaluated according to the FDA guidelines showed that 336/1452 (23.1%) readings did not meet the acceptance criteria. A clear bias toward elevated values was observed for BG <100 mg/dL (MARD: 11.0%).

CONCLUSIONS

The results show that the BGM, although approved according to standard regulatory guidelines, did not meet the level of analytical accuracy required for clinical treatment decisions according to ISO 15197:2015 and FDA requirements. In general, caution should be exercised before selection of BGMs for patients and in clinical trials.

Investigation of the Accuracy of 18 Marketed Blood Glucose Monitors

OBJECTIVE

Cleared blood glucose monitors (BGMs) for personal use may not always deliver levels of accuracy currently specified by international and U.S. regulatory bodies. This study's objective was to assess the accuracy of 18 such systems cleared by the U.S. Food and Drug Administration representing approximately 90% of commercially available systems used from 2013 to 2015.

RESEARCH DESIGN AND METHODS

A total of 1,035 subjects were recruited to have a capillary blood glucose (BG) level measured on six different systems and a reference capillary sample prepared for plasma testing at a reference laboratory. Products were obtained from consumer outlets and tested in three triple-blinded studies. Each of the three participating clinical sites tested a different set of six systems for each of the three studies in a round-robin. In each study, on average, a BGM was tested on 115 subjects. A compliant BG result was defined as within 15% of a reference plasma value (for BG ≥100 mg/dL [5.55 mmol/L]) or within 15 mg/dL (0.83 mmol/L) (for BG <100 mg/dL [5.55 mmol/L]). The proportion of compliant readings in each study was compared against a predetermined accuracy standard similar to, but more lenient than, current regulatory standards. Other metrics of accuracy included the overall compliance proportion; the proportion of extreme outlier readings differing from the reference value by >20%; modified Bland-Altman analysis including average bias, coefficient of variation, and 95% limits of agreement; and proportion of readings with no clinical risk as determined by the Surveillance Error Grid.

RESULTS

The different accuracy metrics produced almost identical BGM rankings. Six of the 18 systems met the predetermined accuracy standard in all three studies, 5 systems met it in two studies, and 3 met it in one study. Four BGMs did not meet the accuracy standard in any of the three studies.

CONCLUSIONS

Cleared BGMs do not always meet the level of analytical accuracy currently required for regulatory clearance. This information could assist patients, professionals, and payers in choosing products and regulators in evaluating postclearance performance.

Emerging Technologies for Diabetes Care

New therapies, monitoring, and revolutionary enabling technologies applied to healthcare represent an historic opportunity to improve the lives of people with diabetes. These advances enable more meaningful monitoring of blood glucose values with the facilitation of more optimal insulin dosing and delivery. Newer insulins and delivery systems are in development that seek to mitigate both hyperglycemia and hypoglycemia and increase time in range. Information systems now exist that may be leveraged to merge data from previously discrete systems into new models of connected care. This review highlights important developments that serve to increase effectiveness while reducing the burden of diabetes care in the near future.

Seven-Year Surveillance of the Clinical Performance of a Blood Glucose Test Strip Product

BACKGROUND

A key approach in enabling people with diabetes to better manage their condition is through self-monitoring of blood glucose (SMBG). Any functional SMBG system should demonstrate clinical accuracy across a broad glucose range and be insensitive to hematocrit. Furthermore, it should be incumbent on the manufacturer to demonstrate that their product continues to meet clinical accuracy claims during product lifetime.

METHODS

Test strips from a globally distributed SMBG product were sampled from randomly selected production batches as part of the manufacturer's routine product evaluation process. Clinical accuracy was assessed within diabetes patients at 3 clinic sites against a standard reference method and evaluated against system accuracy in accordance with the ISO 15197:2015 standard (unchanged from ISO 15197:2013 in terms of performance specifications). Data were collected over 7 years (2010-2016) and comprised 73,600 individual glucose results. Overall clinic performance was assessed, as was accuracy at low and high glucose levels and extremes of hematocrit.

RESULTS

Across the 7-year surveillance period, overall test strip clinical accuracy was 97.8% versus the 95% ISO-defined minimum criterion with by-year values of 97.0-98.6%. Accuracy at the lowest (≤50 mg/dL) and highest (>400 mg/dL) ranges of glucose was 97.0% and 98.3% respectively. Within these low/high blood glucose subpopulations, accuracy at the lower and upper first percentile hematocrit ranges, was 98.9%, and 97.1% respectively.

CONCLUSIONS

This 7-year surveillance program showed the test strips to have excellent clinical accuracy at the outer ranges of subject blood glucose and hematocrit, based on assessment against the ISO 15197:2015 clinical accuracy criterion.

Why the Diabetes Technology Society Surveillance Protocol for Glucose Meters Needs to Be Revised

The Diabetes Technology Society surveillance protocol provides a seal of approval for a glucose meter if a sufficient number of a candidate glucose meter's results meet ISO 15197:2013 limits. The protocol provides clear details about how to conduct this study and analyze the data but has two flaws. There is no specification about the size of glucose meter errors that are outside of ISO limits. A meter that has a result in the E zone of a glucose meter error grid could receive the DTS seal of approval. In addition, the protocol uses the ISO standard, which could be considered a "state of the art" standard instead of an error grid, which is a clinical standard. Remedies for these problems are to replace the ISO standard with an error grid and to include requirements for errors found in C or higher zones of an error grid.

Do the New FDA Guidance Documents Help Improving Performance of Blood Glucose Monitoring Systems Compared With ISO 15197?

Recently, the Food and Drug Administration (FDA) published guidance documents for point-of-care testing (POCT) and over-the-counter (OTC) blood glucose monitoring systems (BGMS). These are expected to improve weaknesses of the already established ISO 15197:2013 standard accuracy evaluation for OTC BGMS; however, the proposed criteria and procedures rather raise new questions. As an example, the OTC guidance stipulates tighter accuracy criteria in the low glycemic range than the POCT guidance for no obvious reason. Furthermore, a clear definition of requirements for the reference method is missing under consideration of the strict accuracy criteria. External surveillance of BGMS after introduction to the market is an issue that is addressed neither in the FDA documents, nor in ISO 15197, but in the Clinical and Laboratory Standards Institute (CLSI) guideline POCT12-A3.

Analysis of "Accuracy and User Performance Evaluation of a New, Wireless-Enabled Blood Glucose Monitoring System That Links to a Smart Mobile Device"

In the study published in Journal of Diabetes, Science and Technology, Timothy S. Bailey and coauthors evaluated system accuracy and user performance of a new, wireless-enabled blood glucose monitoring system, the Contour Plus ONE blood glucose monitoring system. The authors declare that the study protocol was performed according to ISO 15197:2013 guidelines, in particular to clauses 6.3 and 8. Results show a high level of accuracy of the product in a laboratory setting as well as in the hands of lay users. However, differences between both study parts emerge that stimulate discussion about the used comparison method.

A Comprehensive Performance Evaluation of Five Blood Glucose Systems in the Hypo-, Eu-, and Hyperglycemic Range

BACKGROUND

The objective was to evaluate the performance (in terms of accuracy, precision, and trueness) of 5 CE-certified and commercially available blood glucose (BG) systems (meters plus test strips) using an innovative clinical-experimental study design with a 3-step glucose clamp approach and frequent capillary BG measurements.

METHODS

Sixteen subjects with type 1 diabetes participated in this open label, single center trial. BG was clamped at 3 levels for 60 minutes each: 60-100-200 mg/dL. Medical staff performed regular finger pricks (up to 10 per BG level) to obtain capillary blood samples for paired BG measurements with the 5 BG systems and a laboratory method as comparison.

RESULTS

Three BG systems displayed significantly lower mean absolute relative deviations (MARD) (ACCU-Chek® Aviva Nano [5.4%], BGStar® [5.1%], iBGStar® [5.3%]) than 2 others (FreeStyle InsuLinx® [7.7%], OneTouch Verio®IQ [10.3%]). The measurement precision of all BG systems was comparable, but relative bias was also lower for the 3 systems with lower MARD (ACCU-Chek [1.3%], BGStar [-0.9%], iBGStar [1.0%]) compared with the 2 others (FreeStyle [-7.2%], OneTouch [8.9%]).

CONCLUSIONS

This 3 range glucose clamp approach enables a systematic performance evaluation of BG systems under controlled and reproducible conditions. The random error of the tested BG systems was comparable, but some showed a lower systematic error than others. These BG systems allow an accurate glucose measurement at low, normal and high BG levels.