The Evaluation of Analytical Performance of Immunoassay Tests by using Six-sigma Method

Comparison of Sigma metrics computed by three bias estimation approaches for 33 chemistry and 26 immunoassay analytes

Objectives

Sigma metric can be calculated using a simple equation. However, there are multiple sources for the elements in the equation that may produce different Sigma values. This study aimed to investigate the importance of different bias estimation approaches for Sigma metric calculation.

Methods

Sigma metrics were computed for 33 chemistry and 26 immunoassay analytes on the Roche Cobas 6000 analyzer. Bias was estimated by three approaches: (1) averaging the monthly bias values obtained from the external quality assurance (EQA) studies; (2) calculating the bias values from the regression equation derived from the EQA data; and (3) averaging the monthly bias values from the internal quality control (IQC) events. Sigma metrics were separately calculated for the two levels of the IQC samples using three bias estimation approaches. The resulting Sigma values were classified into five categories considering Westgard Sigma Rules as ≥6, <6 and ≥5, <5 and ≥4, <4 and ≥3, and <3.

Results

When classifying Sigma metrics estimated by three bias estimation approaches for each assay, 16 chemistry assays at the IQC level 1 and 2 were observed to fall into different Sigma categories under at least one bias estimation approach. Similarly, for 12 immunoassays at the IQC level 1 and 2, Sigma category was different depending on bias estimation approach.

Conclusions

Sigma metrics may differ depending on bias estimation approaches. This should be considered when using Six Sigma for assessing analytical performance or scheduling the IQC events.

Comparación de la métrica Sigma calculada con tres métodos de estimación del sesgo en 33 magnitudes químicas y 26 de inmunoensayo

Objetivos

Aunque la métrica Sigma se puede calcular mediante una sencilla ecuación, la diversidad de fuentes de las que se extraen los elementos de la ecuación pueden arrojar diferentes valores Sigma. El objetivo de este estudio era investigar la importancia de las distintas estrategias de estimación del sesgo para el cálculo de la métrica Sigma.

Métodos

Se calculó la métrica Sigma de 33 magnitudes químicas y 26 magnitudes de inmunoensayo en un analizador Roche Cobas 6,000. El sesgo se calculó mediante tres métodos: a) calculando la media del sesgo mensual obtenida en los estudios de control de calidad externo (EQA, por sus siglas en inglés); 2) calculando los valores de sesgo mediante una ecuación de regresión a partir de datos obtenidos del EQA; y 3) calculando la media de los valores de sesgo mensual de los eventos de control de calidad internos (IQC, por sus siglas en inglés). Se realizó una métrica Sigma para cada uno de los dos niveles de muestras de IQC empleando tres métodos para calcular el sesgo. Los valores Sigma obtenidos se clasificaron en cinco categorías, en función de las reglas Sigma de Westgard, siendo ≥6, <6 y ≥5, <5 y ≥4, <4 y ≥3, y <3.

Resultados

Al clasificar la métrica Sigma, calculada aplicando tres métodos de estimación del sesgo para cada magnitud, se observó que 16 magnitudes químicas en los niveles 1 y 2 de IQC fueron clasificadas en categorías Sigma diferentes por al menos uno de los métodos de estimación de la desviación. Del mismo modo, dependiendo del método de estimación del sesgo empleado, se clasificaba en diferentes categorías a 12 magnitudes de inmunoensayo con niveles 1 y 2 de IQC.

Conclusiones

La métrica Sigma puede variar dependiendo del método empleado para calcular el sesgo, lo cual debe ser tenido en cuenta a la hora de evaluar el rendimiento analítico o programar eventos de IQC aplicando el método Seis Sigma.

Sigma Metrics in Laboratory Medicine: A Call for Harmonization

BACKGROUND AND AIM

Sigma metrics are applied in clinical laboratories to assess the quality of analytical processes. A parameter associated to a Sigma >6 is considered "world class" whereas a Sigma <3 is "poor" or "unacceptable". The aim of this retrospective study was to quantify the impact of different approaches for Sigma metrics calculation.

MATERIAL AND METHODS

Two IQC levels of 20 different parameters were evaluated for a 12-month period. Sigma metrics were calculated using the formula: (allowable total error (TEa) (%) - bias (%))/(coefficient of variation (CV) (%)). Method precision was calculated monthly or annually. The bias was obtained from peer comparison program (PCP) or external quality assessment program (EQAP), and 9 different TEa sources were included.

RESULTS

There was a substantial monthly variation of Sigma metrics for all combinations, with a median variation of 32% (IQR, 25.6-41.3%). Variation across multiple analyzers and IQC levels were also observed. Furthermore, TEa source had the highest impact on Sigma calculation with proportions of Sigma >6 ranging from 17.5% to 84.4%. The nature of bias was less decisive.

CONCLUSION

In absence of a clear consensus, we recommend that laboratories calculate Sigma metrics on a sufficiently long period of time (>6 months) and carefully evaluate the choice of TEa source.

"Six Sigma" standard as a level of quality of biochemical laboratories

The principal role of biochemical laboratories is responsibility for reliable, reproducible, accurate, timely, and accurately interpreted analysis results that help in making clinical decisions, while ensuring the desired clinical outcomes. To achieve this goal, the laboratory should introduce and maintain quality control in all phases of work. The importance of applying the Six SIGMA quality model has been analyzed in a large number of scientific studies. The purpose of this review is to highlight the importance of using six SIGMA metrics in biochemical laboratories and the current application of six SIGMA metrics in all laboratory work procedures. It has been shown that the six SIGMA model can be very useful in improving all phases of laboratory work, as well as that a detailed assessment of all procedures of the phases of work and improvement of the laboratory's quality control system is crucial for the laboratory to have the highest level of six SIGMA. Clinical laboratories should use SIGMA metrics to monitor their performance, as it makes it easier to identify gaps in their performance, thereby improving their efficiency and patient safety. Medical laboratory quality managers should provide a systematic methodology for analyzing and correcting quality assurance systems to achieve Six SIGMA quality-level standards.

Application of a six sigma model to evaluate the analytical performance of urinary biochemical analytes and design a risk-based statistical quality control strategy for these assays: A multicenter study

Background

The six sigma model has been widely used in clinical laboratory quality management. In this study, we first applied the six sigma model to (a) evaluate the analytical performance of urinary biochemical analytes across five laboratories, (b) design risk‐based statistical quality control (SQC) strategies, and (c) formulate improvement measures for each of the analytes when needed.

Methods

Internal quality control (IQC) and external quality assessment (EQA) data for urinary biochemical analytes were collected from five laboratories, and the sigma value of each analyte was calculated based on coefficients of variation, bias, and total allowable error (TEa). Normalized sigma method decision charts for these urinary biochemical analytes were then generated. Risk‐based SQC strategies and improvement measures were formulated for each laboratory according to the flowchart of Westgard sigma rules, including run sizes and the quality goal index (QGI).

Results

Sigma values of urinary biochemical analytes were significantly different at different quality control levels. Although identical detection platforms with matching reagents were used, differences in these analytes were also observed between laboratories. Risk‐based SQC strategies for urinary biochemical analytes were formulated based on the flowchart of Westgard sigma rules, including run size and analytical performance. Appropriate improvement measures were implemented for urinary biochemical analytes with analytical performance lower than six sigma according to the QGI calculation.

Conclusions

In multilocation laboratory systems, a six sigma model is an excellent quality management tool and can quantitatively evaluate analytical performance and guide risk‐based SQC strategy development and improvement measure implementation.

Application of Sigma metrics in the quality control strategies of immunology and protein analytes

Background

Six Sigma (6σ) is an efficient laboratory management method. We aimed to analyze the performance of immunology and protein analytes in terms of Six Sigma.

Methods

Assays were evaluated for these 10 immunology and protein analytes: Immunoglobulin G (IgG), Immunoglobulin A (IgA), Immunoglobulin M (IgM), Complement 3 (C3), Complement 4 (C4), Prealbumin (PA), Rheumatoid factor (RF), Anti streptolysin O (ASO), C‐reactive protein (CRP), and Cystatin C (Cys C). The Sigma values were evaluated based on bias, four different allowable total error (TEa) and coefficient of variation (CV) at QC materials levels 1 and 2 in 2020. Sigma Method Decision Charts were established. Improvement measures of analytes with poor performance were recommended according to the quality goal index (QGI), and appropriate quality control rules were given according to the Sigma values.

Results

While using the TEa_NCCL, 90% analytes had a world‐class performance with σ>6, Cys C showed marginal performance with σ<4. While using minimum, desirable, and optimal biological variation of TEa, only three (IgG, IgM, and CRP), one (CRP), and one (CRP) analytes reached 6σ level, respectively. Based on σ_NCCL that is calculated from TEa_NCCL, Sigma Method Decision Charts were constructed. For Cys C, five multi‐rules (1_3s/2_2s/R_4s/4_1s/6_X, N = 6, R = 1, Batch length: 45) were adopted for QC management. The remaining analytes required only one QC rule (1_3s, N = 2, R = 1, Batch length: 1000). Cys C need to improve precision (QGI = 0.12).

Conclusions

The laboratories should choose appropriate TEa goals and make judicious use of Sigma metrics as a quality improvement tool.

A hybrid model for ranking critical successful factors of Lean Six Sigma in the oil and gas industry

Purpose

The aim of this paper is to find and prioritise multiple critical success factors (CSFs) for the implementation of LSS in the oil and gas industry.

Design/methodology/approach

Based on a preselected list of possible CFSs, experts are involved in screening them with the Delphi method. As a result, 22 customised CSFs are selected. To prioritise these CSFs, the step-wise weight assessment ratio analysis (SWARA) method is applied to find weights corresponding to the decision-making preferences. Since the regular permutation-based weight assessment can be classified as NP-hard, the problem is solved by a metaheuristic method. For this purpose, a genetic algorithm (GA) is used.

Findings

The resulting prioritisation of CSFs helps companies find out which factors have a high priority in order to focus on them. The less important factors can be neglected and thus do not require limited resources.

Research limitations/implications

Only a specific set of methods have been considered.

Practical implications

The resulting prioritisation of CSFs helps companies find out which factors have a high priority in order to focus on them.

Social implications

The methodology supports respective evaluations in general.

Originality/value

The paper contributes to the very limited research on the implementation of LSS in the oil and gas industry, and, in addition, it suggests the usage of SWARA, a permutation method and a GA, which have not yet been researched, for the prioritisation of CSFs of LSS.

Practical application of Westgard Sigma rules with run size in analytical biochemistry processes in clinical settings

BACKGROUND

The performance of 18 routine chemical detection methods was evaluated by the sigma (σ) metric, and Westgard Sigma rules with run size were used to establish internal quality control (IQC) standards to reduce patient risks.

MATERIALS AND METHODS

External quality assessment (EQA) and internal quality control data from 18 assays in a biochemical laboratory were collected from January to June 2020. The sigma values of each assay were calculated, based on the bias, total error allowable, and coefficient of variation, appropriate quality control rules were selected. According to the quality goal index, the main causes of poor performance were determined to guide quality improvement.

RESULTS

At IQC material level 1, seven of the 18 assays achieved five sigma (excellent), and five assays (UA, Crea, AMY, TC and Na) showed world-class performance. At IQC material level 2, 14 of the 18 assays achieved 5 sigma (excellent), and thirteen assays (UA, ALT, CK, Crea, AMY, K, AST, ALP, Na, LDH, Mg, TC and GGT) showed world-class performance. The quality goal index (QGI) was calculated for items with analysis performance <5 sigma, and the main causes of poor performance were determined to guide quality improvement.

CONCLUSIONS

Westgard sigma rules with run size are an effective tool for evaluating the performance of biochemical assays. These rules can be used to more simply and intuitively select the quality control strategy of related items and reduce the risk to patients.

Application of a six sigma model to the evaluation of the analytical performance of serum enzyme assays and the design of a quality control strategy for these assays: A multicentre study

BACKGROUND

Six medical testing laboratories at six different sites in China participated in this study. We applied a six sigma model for (a) the evaluation of the analytical performance of serum enzyme assays at each of the laboratories, (b) the design of individualized quality control programs and (c) the development of improvement measures for each of the assays, as appropriate.

METHODS

Internal quality control (IQC) and external quality assessment (EQA) data for selected serum enzyme assays were collected from each of the laboratories. Sigma values for these assays were calculated using coefficients of variation, bias, and total allowable error (TEa). Normalized sigma method decision charts were generated using these parameters. IQC design and improvement measures were defined using the Westgard sigma rules. The quality goal index (QGI) was used to assist with identification of deficiencies (bias problems, precision problems, or their combination) affecting the analytical performance of assays with sigma values <6.

RESULTS

Sigma values for the selected serum enzyme assays were significantly different at different levels of enzyme activity. Differences in assay quality in different laboratories were also seen, despite the use of identical testing instruments and reagents. Based on the six sigma data, individualized quality control programs were outlined for each assay with sigma <6 at each laboratory.

CONCLUSIONS

In multi-location laboratory systems, a six sigma model can evaluate the quality of the assays being performed, allowing management to design individualized IQC programs and strategies for continuous improvement as appropriate for each laboratory. This will improve patient care, especially for patients transferred between sites within multi-hospital systems.

Practical application of Six Sigma management in analytical biochemistry processes in clinical settings

BACKGROUND

Six Sigma methodology with a zero-defect goal has long been applied in commercial settings and was utilized in this study to assure/improve the quality of various analytes.

METHODS

Daily internal quality control (QC) and external quality assessment data were collected and analyzed by calculating the sigma (σ) values for 19 analytes based on the coefficient of variation, bias, and total error allowable. Standardized QC sigma charts were established with these parameters. Quality goal index (QGI) analysis and root cause analysis (RCA) were used to discover potential problems for the analytes.

RESULTS

Five analytes with σ ≥ 6 achieved world-class performance, and only the Westgard rule (1_3s ) with one control measurement at two QC material levels (N2) per QC event and a run size of 1000 patient samples between QC events (R1000) was needed for QC. In contrast, more control rules (2_2s /R_4s /4_1s ) along with high N values and low R values were needed for quality assurance for five analytes with 4 ≤ σ < 6. However, the sigma levels of nine analytes were σ < 4 at one or more QC levels, and a more rigorous QC procedure (1_3s /2_2s /R_4s /4_1s /8_x with N4 and R45) was implemented. The combination of QGI analysis and RCA further revealed inaccuracy or imprecision problems for these analytes with σ < 4 and discovered five aspects of potential causes considered for quality improvement.

CONCLUSIONS

Six Sigma methodology is an effective tool for evaluating the performance of biochemical analytes and is conducive to quality assurance and improvement.

Evaluation of Analytical Performance of Variant II Turbo HbA1c Analyzer According to Sigma Metrics

Background

Hemoglobin A1c, (HbA1c) which is the major constituent of glycated hemoglobin, has been used in the follow-up of retrospective glycemia for years and in the diagnosis of diabetes mellitus nowadays. Since the analytical performance of HbA1c should be high likewise all laboratory tests, various quality control measures are used. Sigma metrics is one of these measures and it is the combination of bias, precision and total allowable error that ensures a general evaluation of analytical quality. The aim of our study was to evaluate the analytical performance of Bio-Rad’s Variant Turbo II HbA1c analyzer according to sigma metrics.

Methods

Sigma levels were calculated using the data obtained from two levels of internal and 12 external quality control materials (Bio-Rad) of Variant II Turbo HbA1c analyzer according to σ = (TEa% - Bias%) / CV% formula.

Results

The mean sigma levels for low and high quality control materials were found to be 3.0 and 4.1, respectively.

Conclusions

The annual mean analytical performance of Variant II Turbo HbA1c analyzer was found to be acceptable according to sigma metrics. In order to be sure of the difference in HbA1c results indicating the success or failure in treatment but not arise from analytical variation, it is thought that more stringent quality control measures should be applied to reach higher sigma levels.

Analytical quality assessment and method comparison of immunoassays for the measurement of serum cobalamin and folate in dogs and cats

Serum cobalamin and folate are often measured in cats and dogs as part of laboratory testing for intestinal disease, small intestinal dysbiosis, or exocrine pancreatic deficiency. We performed an analytical validation of human immunoassays for cobalamin and folate measurement (AIA-900 analyzer, Tosoh Bioscience) and compared results with those obtained using chemiluminescence assays (Immulite 2000 analyzer, Siemens Medical Solutions Diagnostics). Accuracy, precision, total observable error (TE_obs%), and σ values were calculated for the immunoassays. Correlation and agreement were evaluated with Deming regression, Passing-Bablok regression, and Bland-Altman analysis. Cobalamin intra-assay and inter-assay CVs were 1.8-9.3% and 2.6-6.8%, respectively. Folate intra-assay and inter-assay CVs were 1.5-9.1% and 3.4-8.1%, respectively. TE_obs (%) were ≤19 and ≤31 for cobalamin and folate, respectively. Sigma values were 3.60-11.50 for cobalamin and 2.90-7.50 for folate. Regression analysis demonstrated very high or high correlations for cobalamin [ r = 0.98 (dogs), 0.97 (cats)] and folate [ r = 0.88 (dogs), 0.92 (cats)] but Bland-Altman analysis revealed poor agreement for both. The immunoassays had good analytical performance for measuring cobalamin and folate in both species. Results obtained by the 2 analyzers cannot be used interchangeably and should be interpreted using instrument-specific reference intervals. Further studies are required to establish immunoassay-specific reference intervals and to evaluate the diagnostic performance and clinical utility of the analyzer for these analytes.