Application of a Simulation Model to Estimate Treatment Error and Clinical Risk Derived from Point-of-Care International Normalized Ratio Device Analytic Performance

Evaluation of Precision and Bias Specifications Required to Achieve the 2018 FDA Guidance Criteria for Glucose Meter Performance Using Simulation Models

BACKGROUND

The objective of this study was to estimate the combinations of total bias and total imprecision required for devices to meet the Food and Drug Administration (FDA) specifications using Monte Carlo simulation rather than collection and analysis of experimental data.

METHODS

A model Gaussian distribution of true-glucose values was altered by adding bias and imprecision to create measured-glucose values affected by analytic error. The fraction of measured-glucose values that met the 2018 FDA criteria for blood glucose monitoring system (BGMS) or self-monitoring blood glucose (SMBG) devices was determined as a function of bias and imprecision.

RESULTS

The BGMS model determined that a maximum total imprecision of 6% was required with no bias, and with a total bias of +10 mg/dL the total imprecision allowed was reduced to 5% to achieve the 95% FDA performance expectation: 95% of results ≥75 mg/dL within ±12% and 95% of results <75 mg/dL within ±12 mg/dL. The SMBG model determined that a maximum total imprecision of 6% was required at no bias, and with a total bias of +10 mg/dL the total imprecision allowed was reduced to 4% to achieve the 98% FDA expectation: 98% of results ±75 mg/dL within ±15% and 98% of results <75 mg/dL within ±15 mg/dL.

CONCLUSIONS

The 2018 FDA guidance criteria require strict conditions for glucose meter clinical trials to achieve <10 mg/dL total bias and total imprecision of <5%. Total imprecision and bias values assessed in models in this study represent the cumulative imprecision and bias errors for the glucose meters, the reference method, and preanalytic processes.

Toward a Framework for Outcome-Based Analytical Performance Specifications: A Methodology Review of Indirect Methods for Evaluating the Impact of Measurement Uncertainty on Clinical Outcomes

BACKGROUND

For medical tests that have a central role in clinical decision-making, current guidelines advocate outcome-based analytical performance specifications. Given that empirical (clinical trial-style) analyses are often impractical or unfeasible in this context, the ability to set such specifications is expected to rely on indirect studies to calculate the impact of test measurement uncertainty on downstream clinical, operational, and economic outcomes. Currently, however, a lack of awareness and guidance concerning available alternative indirect methods is limiting the production of outcome-based specifications. Therefore, our aim was to review available indirect methods and present an analytical framework to inform future outcome-based performance goals.

CONTENT

A methodology review consisting of database searches and extensive citation tracking was conducted to identify studies using indirect methods to incorporate or evaluate the impact of test measurement uncertainty on downstream outcomes (including clinical accuracy, clinical utility, and/or costs). Eighty-two studies were identified, most of which evaluated the impact of imprecision and/or bias on clinical accuracy. A common analytical framework underpinning the various methods was identified, consisting of 3 key steps: (a) calculation of "true" test values; (b) calculation of measured test values (incorporating uncertainty); and (c) calculation of the impact of discrepancies between (a) and (b) on specified outcomes. A summary of the methods adopted is provided, and key considerations are discussed.

CONCLUSIONS

Various approaches are available for conducting indirect assessments to inform outcome-based performance specifications. This study provides an overview of methods and key considerations to inform future studies and research in this area.