An approach for determining allowable between reagent lot variation

When bias becomes part of imprecision: how to use analytical performance specifications to determine acceptability of lot-lot variation and other sources of possibly unacceptable bias

ISO 15189 requires laboratories to estimate the uncertainty of their quantitative measurements and to maintain them within relevant performance specifications. Furthermore, it refers to ISO TS 20914 for instructions on how to estimate the uncertainty and what to take into consideration when communicating uncertainty of measurement with requesting clinicians. These instructions include the responsibility of laboratories to verify that bias is not larger than medically significant. If estimated to be larger than acceptable, such bias first needs to be eliminated or (temporarily) corrected for. In the latter case, the uncertainty of such correction becomes part of the estimation of the total measurement uncertainty. If small enough to be acceptable, bias becomes part of the long term within laboratory random variation. Sources of possible bias are (not limited to) changes in reagent or calibrator lot variation or calibration itself. In this paper we clarify how the rationale and mathematics from an EFLM WG ISO/A position paper on allowable between reagent lot variation can be applied to calculate whether bias can be accepted to become part of long-term imprecision. The central point of this rationale is to prevent the risk that requesting clinicians confuse changes in bias with changes in the steady state of their patients.

Modelling the impact of bias in fecal immunochemical testing on long-term outcomes of colorectal cancer screening

BACKGROUND

As the impact of unmanaged bias (i.e. systematic source of inaccuracy) in fecal immunochemical test (FIT) analytical performance on long-term colorectal cancer (CRC) outcomes is unknown, we assessed the impact bias in FIT performance in an ongoing FIT-based CRC screening program.

METHODS

This study consisted of two parts: cross-sectional observational data analysis to estimate change in short-term outcomes and microsimulation modelling to estimate change in long-term outcomes assuming different levels of bias by assuming 15 % lower up to 15 % higher Hemoglobin detected in the stool compared to observed. Two scenarios were considered: bias occurring 1) one-time only, due to the occasional bias associated with the FIT kits used in 2020 and 2) consistently due to a constant bias associated with the FIT kits used from 2020 onwards.

RESULTS

With a hypothetical bias of -15 % to +15 %, we observed a positivity rate ranging from 6.7 % to 7.8 %, and a detection rate for CRC between 0.65 % and 0.68 %. Single biases in FIT performance resulted in less than 0.1 % change in long-term CRC screening outcomes, while consistent biases resulted in a much larger change (up to 1.4 % in CRC cases and CRC-related deaths and up to 2.07 % in total costs). Detecting lower Hemoglobin concentrations resulted in a relatively larger change on long-term CRC outcomes in comparison to positive bias.

CONCLUSIONS

Because of the substantial impact of consistent FIT bias, it is important to set evidence-based acceptance criteria of bias on long-term CRC screening outcomes and in particular, the introduction of an asymmetrical or upward shifted tolerance interval for FIT bias.

To rule-in, or not to falsely rule-out, that is the question: evaluation of hs-cTnT EQA performance in light of the ESC-2020 guideline

OBJECTIVES

To accurately evaluate non-ST-elevated acute cardiac syndrome (NSTE-ACS), the quality of high-sensitive cardiac troponin (hs-cTn) assays is of vital importance. The 2020 revision of the NSTE-ACS guideline includes clinical decision-limits (CDL's) to both rule-in and rule-out NSTE-ACS for most commercially available platforms, providing both 0/1 h and 0/2 h delta limits. Our study evaluated whether laboratories are able to meet the analytical performance specifications for imprecision (APS) for hs-cTnT.

METHODS

Results from external quality assurance (EQA) in commutable samples were used to evaluate the current and historic performance of analyzers. The performance of analyzers that either passed or failed to comply with 0/1 h-APS were used on a real-world dataset of first hs-cTnT-values to simulate 10.000 samples of t=0, t=1 and t=2 h values with multiple delta's for all relevant CDL's. We compared the simulated values to the input values to obtain the percentage of aberrant results simulated.

RESULTS

The majority of analyzers complies with APS for rule-in in 2022 (0/1 h: 90.4 % and 0/2 h: 100 %), compliance for the 0/1 h rule-out is still far from optimal (0/1 h: 30.7 %, 0/2 h: 75.4 %), with improving compliance over the past years (rule-in p=<0.0001, rule-out p=0.011, χ2). Whilst 0/1 h-APS-passing analyzers have a minute risk to falsely rule-out patients whom should be ruled-in (0.0001 %), failing performance increases this risk to 2.1 % upon using 0/1 h CDL's. Here, adopting 0/2 h CDL's is favorable (0.01 %).

CONCLUSIONS

Laboratories that fail to meet hs-cTnT 0/1 h-APS should improve their performance to the required and achievable level. Until performance is reached clinics should adopt the 0/2 h CDL's.

Overcoming Lot-to-Lot Variability in Protein Activity Using Epitope-Specific Calibration-Free Concentration Analysis

Concentration determination is a fundamental hallmark of protein reagent characterization, providing a means to ensure reproducibility and unify measurements from various assays. However, lot-to-lot differences in protein activity often still occur, leading to uncertainty in the accuracy of downstream measurements. Here, we postulate that those differences are caused by a misrepresentation of the protein concentration as measured by traditional total protein techniques, which can include multiple types of inactive protein species. To overcome this, we developed a standardized method to quantify a protein's active concentration via calibration-free concentration analysis (CFCA). As a pilot study, we compare the biophysical and immunoassay responses from three batches of recombinant soluble lymphocyte-activation gene 3 (sLAG3), as defined by either their total or active concentrations. Defining the sLAG3 reagents by their assay-specific concentration improved consistency in reported kinetic binding parameters and decreased immunoassay lot-to-lot coefficients of variation (CVs) by over 600% compared to the total protein concentration. These findings suggest that the total concentration of a protein reagent may not be the ideal metric to correlate in-assay signals between lots, and by instead quantifying the concentrations of a reagent's assay-specific epitopes, CFCA may prove a useful tool in overcoming lot-to-lot variability.

APS calculator: a data-driven tool for setting outcome-based analytical performance specifications for measurement uncertainty using specific clinical requirements and population data

OBJECTIVES

According to ISO 15189:2022, analytical performance specifications (APS) should relate to intended clinical use and impact on patient care. Therefore, we aimed to develop a web application for laboratory professionals to calculate APS based on a simulation of the impact of measurement uncertainty (MU) on the outcome using the chosen decision limits, agreement thresholds, and data of the population of interest.

METHODS

We developed the "APS Calculator" allowing users to upload and select data of concern, specify decision limits and agreement thresholds, and conduct simulations to determine APS for MU. The simulation involved categorizing original measurand concentrations, generating measured (simulated) results by introducing different degrees of MU, and recategorizing measured concentrations based on clinical decision limits and acceptable clinical misclassification rates. The agreements between original and simulated result categories were assessed, and values that met or exceeded user-specified agreement thresholds that set goals for the between-category agreement were considered acceptable. The application generates contour plots of agreement rates and corresponding MU values. We tested the application using National Health and Nutrition Examination Survey data, with decision limits from relevant guidelines.

RESULTS

We determined APS for MU of six measurands (blood total hemoglobin, plasma fasting glucose, serum total and high-density lipoprotein cholesterol, triglycerides, and total folate) to demonstrate the potential of the application to generate APS.

CONCLUSIONS

The developed data-driven web application offers a flexible tool for laboratory professionals to calculate APS for MU using their chosen decision limits and agreement thresholds, and the data of the population of interest.

Albumin determined by bromocresol green leads to erroneous results in routine evaluation of patients with chronic kidney disease

OBJECTIVES

Measurement of plasma albumin is pivotal for clinical decision-making in patients with chronic kidney disease (CKD). Routinely used methods as bromocresol green (BCG) and bromocresol purple (BCP) can suffer from aselectivity, but the impact of aselectivity on the accuracy of plasma albumin results of CKD-patients is still unknown. Therefore, we evaluated the performance of BCG-, BCP- and JCTLM-endorsed immunological methods in patients with various stages of CKD.

METHODS

We evaluated the performance of commonly used albumin methods in patients with CKD stages G1 through G5, the latter divided in two groups based on whether they received hemodialysis treatment. In total, 163 patient plasma samples were measured at 14 laboratories, on six different BCG and BCP-platforms, and four different immunological platforms. The results were compared with an ERM-DA-470k-corrected nephelometric assay. The implications on outcome is evaluated by the proportion of patient results <38 g/L for the diagnosis of protein energy wasting.

RESULTS

Albumin results determined with BCP- and immunological methods showed the best agreement with the target value (92.7 and 86.2 %, respectively vs. 66.7 % for BCG, namely due to overestimation). The relative agreement of each method with the target value was platform-dependent, with larger variability in agreement between platforms noted for BCG and immunological methods (3.2-4.6 and 2.6-5.3 %) as opposed to BCP (0.7-1.5 %). The stage of CKD had similar effects on the variability in agreement for the three method-groups (0.6-1.8 % vs. 0.7-1.5 % vs. 0.4-1.6 %). The differences between methods cause discrepancies in clinical decision-making, as structurally fewer patients were diagnosed with protein energy wasting upon using BCG-based albumin results.

CONCLUSIONS

Our study shows that BCP is fit for the intended use to measure plasma albumin levels in CKD patients from all stages, including patients on hemodialysis. In contrast, most BCG-based platforms falsely overestimate the plasma albumin concentration.

Difference- and regression-based approaches for detection of bias

OBJECTIVE

This simulation study was undertaken to assess the statistical performance of six commonly used rejection criteria for bias detection.

METHODS

The false rejection rate (i.e. rejection in the absence of simulated bias) and the probability of bias detection were assessed for the following: difference in measurements for individual sample pair, the mean of the paired differences, t-statistics (paired t-test), slope < 0.9 or > 1.1, intercept > 50% of the lower limit of measurement range, and coefficient of determination (R²) > 0.95. The linear regressions evaluated were ordinary least squares, weighted least squares and Passing-Bablok regressions. A bias detection rate of < 50% and false rejection rates of >10% are considered unacceptable for the purpose of this study.

RESULTS

Rejection criteria based on regression slope, intercept and paired difference (10%) for individual samples have high false rejection rates and/ or low probability of bias detection. T-statistics (α = 0.05) performed best in low range ratio (lowest-to-highest concentration in measurement range) and low imprecision scenarios. Mean difference (10%) performed better in all other range ratio and imprecision scenarios. Combining mean difference and paired-t test improves the power of bias detection but carries higher false rejection rates.

CONCLUSIONS

This study provided objective evidence on commonly used rejection criteria to guide laboratory on the experimental design and statistical assessment for bias detection during method evaluation or reagent lot verification.

Factors affecting the evaluation and use of a hemoglobin A2 method - Lot-to-lot variation, long-term deviation and carry-over

BACKGROUND

We compared two Hb A2 methods, high pressure liquid chromatography and capillary zone electrophoresis, at two occasions. In 2014 the methods showed good agreement, while in 2020 HPLC results were clearly higher than CE results. This finding prompted us to investigate the external quality assessment (EQA) outcome and our total patient results obtained by high pressure liquid chromatography over several years.

METHODS

The methods compared were Bio-Rad Variant II Beta-Thal Short Program (HPLC) and Sebia Capilllarys Flex Piercing (CE).

RESULTS

Our annual patient results obtained by HPLC increased significantly from 2014 to 2020. A similar trend was also seen in our EQA results. When patient results were grouped according to different reagent lots it became evident that method comparisons might be severely affected by lot-to-lot variation. We also noticed that samples analyzed with the HPLC method following a sample containing a high proportion of Hb E where prone to give falsely increased Hb A2 results.

CONCLUSION

Standardization of the measurement of Hb A2 is urgently needed. Furthermore, the lot-to-lot variation must be minimized. While waiting for these improvements each laboratory ought to repeatedly evaluate their distribution of patient Hb A2 results.

ISO 15189 is a sufficient instrument to guarantee high-quality manufacture of laboratory developed tests for in-house-use conform requirements of the European In-Vitro-Diagnostics Regulation

The EU In-Vitro Diagnostic Device Regulation (IVDR) aims for transparent risk-and purpose-based validation of diagnostic devices, traceability of results to uniquely identified devices, and post-market surveillance. The IVDR regulates design, manufacture and putting into use of devices, but not medical services using these devices. In the absence of suitable commercial devices, the laboratory can resort to laboratory-developed tests (LDT) for in-house use. Documentary obligations (IVDR Art 5.5), the performance and safety specifications of ANNEX I, and development and manufacture under an ISO 15189-equivalent quality system apply. LDTs serve specific clinical needs, often for low volume niche applications, or correspond to the translational phase of new tests and treatments, often extremely relevant for patient care. As some commercial tests may disappear with the IVDR roll-out, many will require urgent LDT replacement. The workload will also depend on which modifications to commercial tests turns them into an LDT, and on how national legislators and competent authorities (CA) will handle new competences and responsibilities. We discuss appropriate interpretation of ISO 15189 to cover IVDR requirements. Selected cases illustrate LDT implementation covering medical needs with commensurate management of risk emanating from intended use and/or design of devices. Unintended collateral damage of the IVDR comprises loss of non-profitable niche applications, increases of costs and wasted resources, and migration of innovative research to more cost-efficient environments. Taking into account local specifics, the legislative framework should reduce the burden on and associated opportunity costs for the health care system, by making diligent use of existing frameworks.

Time to address quality control processes applied to antibody testing for infectious diseases

As testing for infectious diseases moves from manual, biological testing such as complement fixation to high throughput automated autoanalyzer, the methods for controlling these assays have also changed to reflect those used in clinical chemistry. However, there are many differences between infectious disease serology and clinical chemistry testing, and these differences have not been considered when applying traditional quality control methods to serology. Infectious disease serology, which is highly regulated, detects antibodies of varying classes and to multiple and different antigens that change according to the organisms' genotype/serotype and stage of disease. Although the tests report a numerical value (usually signal to cut-off), they are not measuring an amount of antibodies, but the intensity of binding within the test system. All serology assays experience lot-to-lot variation, making the use of quality control methods used in clinical chemistry inappropriate. In many jurisdictions, the use of the manufacturer-provided kit controls is mandatory to validate the test run. Use of third-party controls, which are highly recommended by ISO 15189 and the World Health Organization, must be manufactured in a manner whereby they have minimal lot-to-lot variation and at a level where they detect exceptional variation. This paper outlines the differences between clinical chemistry and infectious disease serology and offers a range of recommendations when addressing the quality control of infectious disease serology.

Lot-to-lot variation and verification

Lot-to-lot verification is an integral component for monitoring the long-term stability of a measurement procedure. The practice is challenged by the resource requirements as well as uncertainty surrounding experimental design and statistical analysis that is optimal for individual laboratories, although guidance is becoming increasingly available. Collaborative verification efforts as well as application of patient-based monitoring are likely to further improve identification of any differences in performance in a relatively timely manner. Appropriate follow up actions of failed lot-to-lot verification is required and must balance potential disruptions to clinical services provided by the laboratory. Manufacturers need to increase transparency surrounding release criteria and work closer with laboratory professionals to ensure acceptable reagent lots are released to end users. A tripartite collaboration between regulatory bodies, manufacturers, and laboratory medicine professional bodies is key to developing a balanced system where regulatory, manufacturing, and clinical requirements of laboratory testing are met, to minimize differences between reagent lots and ensure patient safety. Clinical Chemistry and Laboratory Medicine has served as a fertile platform for advancing the discussion and practice of lot-to-lot verification in the past 60 years and will continue to be an advocate of this important topic for many more years to come.

Quality in laboratory medicine and the Journal: walking together

Quality in laboratory medicine is defined as "an unfinished journey", as the more essential the laboratory information provided, the more assured its quality should be. In the past decades, the Journal Clinical Chemistry and Laboratory Medicine has provided a valuable forum for garnering new insights into the analytical and extra-analytical phases of the testing cycle, and for debating crucial aspects of quality in clinical laboratories. The impressive number of papers published in the Journal is testimony to the efforts made by laboratory professionals, national and international scientific societies and federations in the quest to continuously improve upon the pre-, intra- and post-analytical steps of the testing cycle, thus enhancing the quality of laboratory information. The paper appearing in this special issue summarizes the most important and interesting contributions published in the Journal, thus updating our knowledge on quality in laboratory medicine and offering further stimuli to identify the most valuable measures of quality in clinical laboratories.

Lot-to-lot reagent verification: Effect of sample size and replicate measurement on linear regression approaches

BACKGROUND

We investigate the simulated impact of varying sample size and replicate number using ordinary least squares (OLS) and Deming regression (DR) in both weighted and unweighted forms, when applied to paired measurements in lot-to-lot verification.

METHODS

Simulation parameter investigated in this study were: range ratio, analytical coefficient of variation, sample size, replicates, alpha (level of significance) and constant and proportional biases. For each simulation scenario, 10,000 iterations were performed, and the average probability of bias detection was determined.

RESULTS

Generally, the weighted forms of regression significantly outperformed the unweighted forms for bias detection. At the low range ratio (1:10), for both weighted OLS and DR, improved bias detection was observed with greater number of replicates, than increasing the number of comparison samples. At the high range ratio (1:1000), for both weighted OLS and DR, increasing the number of replicates above two is only slightly more advantageous in the scenarios examined. Increasing the numbers of comparison samples resulted in better detection of smaller biases between reagent lots.

CONCLUSIONS

The results of this study allow laboratories to determine a tailored approach to lot-to-lot verification studies, balancing the number of replicates and comparison samples with the analytical performance of measurement procedures involved.

Comparison of six regression-based lot-to-lot verification approaches

OBJECTIVES

Detection of between-lot reagent bias is clinically important and can be assessed by application of regression-based statistics on several paired measurements obtained from the existing and new candidate lot. Here, the bias detection capability of six regression-based lot-to-lot reagent verification assessments, including an extension of the Bland-Altman with regression approach are compared.

METHODS

Least squares and Deming regression (in both weighted and unweighted forms), confidence ellipses and Bland-Altman with regression (BA-R) approaches were investigated. The numerical simulation included permutations of the following parameters: differing result range ratios (upper:lower measurement limits), levels of significance (alpha), constant and proportional biases, analytical coefficients of variation (CV), and numbers of replicates and sample sizes. The sample concentrations simulated were drawn from a uniformly distributed concentration range.

RESULTS

At a low range ratio (1:10, CV 3%), the BA-R performed the best, albeit with a higher false rejection rate and closely followed by weighted regression approaches. At larger range ratios (1:1,000, CV 3%), the BA-R performed poorly and weighted regression approaches performed the best. At higher assay imprecision (CV 10%), all six approaches performed poorly with bias detection rates <50%. A lower alpha reduced the false rejection rate, while greater sample numbers and replicates improved bias detection.

CONCLUSIONS

When performing reagent lot verification, laboratories need to finely balance the false rejection rate (selecting an appropriate alpha) with the power of bias detection (appropriate statistical approach to match assay performance characteristics) and operational considerations (number of clinical samples and replicates, not having alternate reagent lot).

Lot-to-lot reagent verification: challenges and possible solutions

Lot-to-lot verification is an important laboratory activity that is performed to monitor the consistency of analytical performance over time. In this opinion paper, the concept, clinical impact, challenges and potential solutions for lot-to-lot verification are exained.