Application of Sigma metrics in assessing the clinical performance of verified versus non-verified reagents for routine biochemical analytes

Bias in Laboratory Medicine: The Dark Side of the Moon

Physicians increasingly use laboratory-produced information for disease diagnosis, patient monitoring, treatment planning, and evaluations of treatment effectiveness. Bias is the systematic deviation of laboratory test results from the actual value, which can cause misdiagnosis or misestimation of disease prognosis and increase healthcare costs. Properly estimating and treating bias can help to reduce laboratory errors, improve patient safety, and considerably reduce healthcare costs. A bias that is statistically and medically significant should be eliminated or corrected. In this review, the theoretical aspects of bias based on metrological, statistical, laboratory, and biological variation principles are discussed. These principles are then applied to laboratory and diagnostic medicine for practical use from clinical perspectives.

Verification of examination procedures for 72 biochemical parameters on the atellica® clinical chemistry and immunoassay analyzers

The verification of examination procedures is a responsibility for clinical laboratories in order to guarantee that their performance characteristics comply with the specifications obtained during the validation process and are congruent with the intended scope of the assay. The aim was to perform an evaluation of precision, bias, linearity, linear drift, sample carry-over, and comparability of 73 assays from Siemens Healthineers, by following the CLSI EP10-A3 guidelines. The verification was performed by measuring 72 biochemical parameters in quality control (QC) materials from Bio-Rad (except for IL6) with 73 assays installed on eight measuring systems (five Atellica^® CH 930 and three IM 1600 analyzers from Siemens Healthcare Diagnostics). The following information was collected: validation data from manufacturer, biological variation data from the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) database, and specifications for fβhCG and PAPP-A assays to meet the Fetal Medicine Foundation standards. A total of 17550 results were obtained during EP10 verification process. Out of the 73 methods, only Cl-S, Mg-S, and Na-S failed the criteria for adequate precision, trueness, and comparability. The assays did not show significant loss of linearity, linear drift, or sample carry-over. This study allowed the initial training and familiarization with the instruments and the identification of operational issues. It also represented an opportunity to evaluate the QCs and to obtain analytical performance information for application of sigma six metrics for quality assurance. Professionals are advised to adequately standardize and protocolize their verification processes to ensure laboratory competence and patient safety.

The application of sigma metrics in the laboratory to assess quality control processes in South Africa

Background

Laboratories use quality control processes to monitor and evaluate analytical performance in terms of precision and bias. Sigma metrics provide an objective assessment of laboratory quality using the total allowable error as an additional parameter.

Objective

This study aimed to determine the sigma metrics of analytes when using different total allowable error guidelines.

Methods

A retrospective analysis was performed on 19 general chemistry analytes at Charlotte Maxeke Johannesburg Academic Hospital in South Africa between January 2017 and December 2017. Sigma metrics were calculated on two identical analysers, using internal quality control data and total allowable error guidelines from the Ricos biological variation database and three alternative sources (the Royal College of Pathologists of Australasia, the Clinical Laboratory Improvements Amendment, and the European Federation of Clinical Chemistry and Laboratory Medicine).

Results

The sigma performance was similar on both analysers but varied based on the guideline used, with the Clinical Laboratory Improvements Amendment guidelines resulting in the best sigma metrics (53% of analytes on one analyser and 46% on the other had acceptable sigma metrics) and the Royal College of Pathologists of Australia guidelines being the most stringent (21% and 23%). Sodium and chloride performed poorly across all guidelines (sigma < 3). There were also month-to-month variations that may result in acceptable sigma despite poor performance during certain months.

Conclusion

The sigma varies greatly depending on the total allowable error, but could be a valuable tool to save time and decrease costs in high-volume laboratories. Sigma metrics calculations need to be standardised.

Sigma Metrics in Laboratory Medicine: A Call for Harmonization

BACKGROUND AND AIM

Sigma metrics are applied in clinical laboratories to assess the quality of analytical processes. A parameter associated to a Sigma >6 is considered "world class" whereas a Sigma <3 is "poor" or "unacceptable". The aim of this retrospective study was to quantify the impact of different approaches for Sigma metrics calculation.

MATERIAL AND METHODS

Two IQC levels of 20 different parameters were evaluated for a 12-month period. Sigma metrics were calculated using the formula: (allowable total error (TEa) (%) - bias (%))/(coefficient of variation (CV) (%)). Method precision was calculated monthly or annually. The bias was obtained from peer comparison program (PCP) or external quality assessment program (EQAP), and 9 different TEa sources were included.

RESULTS

There was a substantial monthly variation of Sigma metrics for all combinations, with a median variation of 32% (IQR, 25.6-41.3%). Variation across multiple analyzers and IQC levels were also observed. Furthermore, TEa source had the highest impact on Sigma calculation with proportions of Sigma >6 ranging from 17.5% to 84.4%. The nature of bias was less decisive.

CONCLUSION

In absence of a clear consensus, we recommend that laboratories calculate Sigma metrics on a sufficiently long period of time (>6 months) and carefully evaluate the choice of TEa source.

Application of Sigma metrics in the quality control strategies of immunology and protein analytes

Background

Six Sigma (6σ) is an efficient laboratory management method. We aimed to analyze the performance of immunology and protein analytes in terms of Six Sigma.

Methods

Assays were evaluated for these 10 immunology and protein analytes: Immunoglobulin G (IgG), Immunoglobulin A (IgA), Immunoglobulin M (IgM), Complement 3 (C3), Complement 4 (C4), Prealbumin (PA), Rheumatoid factor (RF), Anti streptolysin O (ASO), C‐reactive protein (CRP), and Cystatin C (Cys C). The Sigma values were evaluated based on bias, four different allowable total error (TEa) and coefficient of variation (CV) at QC materials levels 1 and 2 in 2020. Sigma Method Decision Charts were established. Improvement measures of analytes with poor performance were recommended according to the quality goal index (QGI), and appropriate quality control rules were given according to the Sigma values.

Results

While using the TEa_NCCL, 90% analytes had a world‐class performance with σ>6, Cys C showed marginal performance with σ<4. While using minimum, desirable, and optimal biological variation of TEa, only three (IgG, IgM, and CRP), one (CRP), and one (CRP) analytes reached 6σ level, respectively. Based on σ_NCCL that is calculated from TEa_NCCL, Sigma Method Decision Charts were constructed. For Cys C, five multi‐rules (1_3s/2_2s/R_4s/4_1s/6_X, N = 6, R = 1, Batch length: 45) were adopted for QC management. The remaining analytes required only one QC rule (1_3s, N = 2, R = 1, Batch length: 1000). Cys C need to improve precision (QGI = 0.12).

Conclusions

The laboratories should choose appropriate TEa goals and make judicious use of Sigma metrics as a quality improvement tool.

Internal Quality Control Data of Urine Reagent Strip Tests and Derivation of Control Rules Based on Sigma Metrics

Background

Urine reagent strip test (URST) results are semi-quantitative; therefore, the precision of URSTs is evaluated as the proportion of categorical results from repeated measurements of a sample that are concordant with an expected result. However, URSTs have quantitative readout values before ordinal results challenging statistical monitoring for internal quality control (IQC) with control rules. This study aimed to determine the sigma metric of URSTs and derive appropriate control rules for IQC.

Methods

The URiSCAN Super Plus fully automated urine analyzer (YD Diagnostics, Yongin, Korea) was used for URSTs. Change in reflectance rate (change %R) data from IQC for URSTs performed between November 2018 and May 2020 were analyzed. Red blood cells, bilirubin, urobilinogen, ketones, protein, glucose, leukocytes, and pH were measured from 2-3 levels of control materials. The total allowable error (TE_a) for a grade was the difference in midpoints of a predefined change %R range between two adjacent grades. The sigma metric was calculated as TE_a/SD. Sigma metric-based control rules were determined with Westgard EZ Rules 3 software (Westgard QC, Madison, WI, USA).

Results

Seven out of the eight analytes had a sigma metric >4 in the control materials with a negative grade (-), which were closer to the cut-offs. Corresponding control rules ranged from 1_2.5s to 1_3.5s.

Conclusions

Although the URST is a semi-quantitative test, statistical IQC can be performed using the readout values. According to the sigma metric, control rules recommended for URST IQC in routine clinical practice are 1_2.5s to 1_3.5s.

Sigma metrics application for validated and non-validated detecting systems performance assessment

Background

Sigma metrics provide an objective and quantitative methodology for analytical quality evaluation of clinical laboratory. This study investigated the testing performance of validated systems and non‐validated systems based on sigma metrics, and explored the major parameters affecting the system performance.

Methods

Sigma metrics were evaluated by six biochemistry assays based on Beckman and Mindray validated and non‐validated systems through crossing the reagents and analyzers. Imprecision and bias were assessed for all assays based on trueness programs organized by National Centre for Clinical Laboratory. Total error allowance obtained from the Chinese Ministry of Health Clinical Laboratory Centre Industry Standard (WS/T403‐2012).

Results

The imprecision for all systems meets the quality specifications except TP assay (2.19%) detected by Mindray non‐validated system, and the bias for four assays measured by non‐validated systems cannot fulfill the criterion, including lactate dehydrogenase (LDH), total protein (TP), triglycerides (TG), and glucose (GLU). Higher biases were detected in six assays at different levels among non‐validated and validated systems. Systems performed poorly or unacceptably for TP assay with sigma metrics lower than 3 except Mindray non‐validated system. The sigma metrics for other assays with four systems were greater than 3 except the LDH evaluated on Mindray non‐validated systems.

Conclusion

Non‐validated systems may introduce performance uncertainty compared with validated systems based on sigma metrics evaluation, and lower bias was provided by validated systems. The performance of non‐validated systems should be evaluated thoroughly in the clinical laboratory before they were adopted for routine use.

Evaluation of the analytical performance of endocrine analytes using sigma metrics

Background

(a) To evaluate the clinical performance of endocrine analytes using the sigma metrics (σ) model. (b) To redesign quality control strategies for performance improvement.

Methods

The sigma values of the analytes were initially evaluated based on the allowable total error (TEa), bias, and coefficient of variation (CV) at QC materials level 1 and 2 in March 2018. And then, the normalized QC performance decision charts, personalized QC rules, quality goal index (QGI) analysis, and root causes analysis (RCA) were performed based on the sigma values of the analytes. Finally, the sigma values were re‐evaluated in September 2018 after a series of targeted corrective actions.

Results

Based on the initial sigma values, two analytes (FT3 and TSH) with σ > 6, only needed one QC rule (1_3S) with N2 and R500 for QC management. On the other hand, seven analytes (FT4, TT4, CROT, E2, PRL, TESTO, and INS) with σ < 4 at one QC material level or both needed multiple rules (1_3S/2_2S/R_4S/4_1S/10_X) with N6 and R10‐500 depending on different sigma values for QC management. Subsequently, detailed and comprehensive RCA and timely corrective actions were performed on all the analytes base on the QGI analysis. Compared with the initial sigma values, the re‐evaluated sigma metrics of all the analytes increased significantly.

Conclusions

It was demonstrated that the combination of sigma metrics, QGI analysis, and RCA provided a useful evaluation system for the analytical performance of endocrine analytes.

Performance characteristics of automated clinical chemistry analyzers using commercial assay reagents contributing to quality assurance and clinical decision in a hospital laboratory

Background: Clinical laboratories provide essential diagnostic services that are essential in clinical decision making, contributing to the quality of healthcare. The performance of two Siemens ADVIA 1800 analyzers was characterized in a hospital Biochemistry laboratory in order to evaluate the analytical characteristics of such automated analyzer systems using nonoriginal assay reagents attempting to support laboratory quality service and crucial clinical decision making. Methods: We independently completed performance validation studies including trueness, precision, sensitivity as well as measurement of uncertainty and sigma metrics calculation for 25 biochemical parameters. Results: Trueness expressed as bias was less than 20% for both ADVIA 1800 analyzers. Within run and total precisions expressed as CV% were ≤9.85% on both analyzers for most parameters studied with few exceptions (Mg, TB, DB, Cl, HDL and UA) observed either in low or in high level samples and between the two analyzers. LoB, LoD and LoQ values produced by the two analyzers were comparable except Cl. Uncertainty values produced by the two analyzers were comparable with no significant differences. Quality performance of reagent assays was studied using the sigma metrics system. The sigma values were plotted on normalized method decision charts for graphical representation of assay performances for each analyzer. Conclusions: The two ADVIA systems, independently evaluated, showed consistent performance characteristics with certain discrepancies by several reagents. Sigma analysis was helpful for revealing the quality performance of non-original reagents supporting the need for strict assessment of quality assurance and in some instances optimization/improvement of assay methods.

Assessment of quality control system by sigma metrics and quality goal index ratio: A roadmap towards preparation for NABL

AIM

To study sigma metrics and quality goal index ratio (QGI).

METHODS

The retrospective study was conducted at the Clinical Biochemistry Laboratory, PGIMS, Rohtak, which recently became a National Accreditation Board for Testing and Calibration of Laboratories accredited lab as per the International Organization for Standardization 15189:2012 and provides service to a > 1700-bed tertiary care hospital. Data of 16 analytes was extracted over a period of one year from January 2017 to December 2017 for calculation of precision, accuracy, sigma metrics, total error, and QGI.

RESULTS

The average coefficient of variation ranged from 2.12% (albumin) to 5.42% (creatinine) for level 2 internal quality control and 2% (albumin) to 3.62% (high density lipoprotein-cholesterol) for level 3 internal quality control. Average coefficient of variation of all the parameters was below 5%, reflecting very good precision. The sigma metrics for level 2 indicated that 11 (68.5%) of the 16 parameters fall short of meeting Six Sigma quality performance. Of these, five failed to meet minimum sigma quality performance with metrics less than 3, and another six just met minimal acceptable performance with sigma metrics between 3 and 6. For level 3, the data collected indicated eight (50%) of the parameters did not achieve Six Sigma quality performance, out of which three had metrics less than 3, and five had metrics between 3 and 6. QGI ratio indicated that the main problem was inaccuracy in the case of total cholesterol, aspartate transaminase, and alanine transaminase (QGI > 1.2), imprecision in the case of urea (QGI < 0.8), and both imprecision and inaccuracy for glucose.

CONCLUSION

On the basis of sigma metrics and QGI, it may be concluded that the Clinical Biochemistry Laboratory, PGIMS, Rohtak was able to achieve satisfactory results with world class performance for many analytes one year preceding the accreditation by the National Accreditation Board for Testing and Calibration of Laboratories. Aspartate transaminase and alanine transaminase required strict external quality assurance scheme monitoring and modification in quality control procedure as their QGI ratio showed inaccuracy.