APS calculator: a data-driven tool for setting outcome-based analytical performance specifications for measurement uncertainty using specific clinical requirements and population data

OBJECTIVES

According to ISO 15189:2022, analytical performance specifications (APS) should relate to intended clinical use and impact on patient care. Therefore, we aimed to develop a web application for laboratory professionals to calculate APS based on a simulation of the impact of measurement uncertainty (MU) on the outcome using the chosen decision limits, agreement thresholds, and data of the population of interest.

METHODS

We developed the "APS Calculator" allowing users to upload and select data of concern, specify decision limits and agreement thresholds, and conduct simulations to determine APS for MU. The simulation involved categorizing original measurand concentrations, generating measured (simulated) results by introducing different degrees of MU, and recategorizing measured concentrations based on clinical decision limits and acceptable clinical misclassification rates. The agreements between original and simulated result categories were assessed, and values that met or exceeded user-specified agreement thresholds that set goals for the between-category agreement were considered acceptable. The application generates contour plots of agreement rates and corresponding MU values. We tested the application using National Health and Nutrition Examination Survey data, with decision limits from relevant guidelines.

RESULTS

We determined APS for MU of six measurands (blood total hemoglobin, plasma fasting glucose, serum total and high-density lipoprotein cholesterol, triglycerides, and total folate) to demonstrate the potential of the application to generate APS.

CONCLUSIONS

The developed data-driven web application offers a flexible tool for laboratory professionals to calculate APS for MU using their chosen decision limits and agreement thresholds, and the data of the population of interest.

ISO 15189 is a sufficient instrument to guarantee high-quality manufacture of laboratory developed tests for in-house-use conform requirements of the European In-Vitro-Diagnostics Regulation

The EU In-Vitro Diagnostic Device Regulation (IVDR) aims for transparent risk-and purpose-based validation of diagnostic devices, traceability of results to uniquely identified devices, and post-market surveillance. The IVDR regulates design, manufacture and putting into use of devices, but not medical services using these devices. In the absence of suitable commercial devices, the laboratory can resort to laboratory-developed tests (LDT) for in-house use. Documentary obligations (IVDR Art 5.5), the performance and safety specifications of ANNEX I, and development and manufacture under an ISO 15189-equivalent quality system apply. LDTs serve specific clinical needs, often for low volume niche applications, or correspond to the translational phase of new tests and treatments, often extremely relevant for patient care. As some commercial tests may disappear with the IVDR roll-out, many will require urgent LDT replacement. The workload will also depend on which modifications to commercial tests turns them into an LDT, and on how national legislators and competent authorities (CA) will handle new competences and responsibilities. We discuss appropriate interpretation of ISO 15189 to cover IVDR requirements. Selected cases illustrate LDT implementation covering medical needs with commensurate management of risk emanating from intended use and/or design of devices. Unintended collateral damage of the IVDR comprises loss of non-profitable niche applications, increases of costs and wasted resources, and migration of innovative research to more cost-efficient environments. Taking into account local specifics, the legislative framework should reduce the burden on and associated opportunity costs for the health care system, by making diligent use of existing frameworks.

Clinical decision limits as criteria for setting analytical performance specifications for laboratory tests

BACKGROUND

The biological (CV_I), preanalytical (CV_PRE), and analytical variation (CV_A) are inherent to clinical laboratory testing and consequently, interpretation of clinical test results.

METHODS

The sum of the CV_I, CV_PRE, and CV_A, called diagnostic variation (CV_D), was used to derive clinically acceptable analytical performance specifications (CAAPS) for clinical chemistry measurands. The reference change concept was applied to clinically significant differences (CD) between two measurements, with the formula CD = z*√2* CV_D. CD for six measurands were sought from international guidelines. The CAAPS were calculated by subtracting variances of CV_I and CV_PRE from CV_D. Modified formulae were applied to consider statistical power (1-β) and repeated measurements.

RESULTS

The obtained CAAPS were 44.9% for urine albumin, 0.6% for plasma sodium, 22.9% for plasma pancreatic amylase, and 8.0% for plasma creatinine (z = 3, α = 2.5%, 1-β = 85%). For blood HbA_1c and plasma low-density lipoprotein cholesterol, replicate measurements were necessary to reach CAAPS for patient monitoring. The derived CAAPS were compared with analytical performance specifications, APS, based on biological variation.

CONCLUSIONS

The CAAPS models pose a new tool for assessing APS in a clinical laboratory. Their usability depends on the relevance of CD limits, required statistical power and the feasibility of repeated measurements.

An approach for determining allowable between reagent lot variation

Clinicians trust medical laboratories to provide reliable results on which they rely for clinical decisions. Laboratories fulfil their responsibility for accurate and consistent results by utilizing an arsenal of approaches, ranging from validation and verification experiments to daily quality control procedures. All these procedures verify, on different moments, that the results of a certain examination procedure have analytical performance characteristics (APC) that meet analytical performance specifications (APS) set for a particular intended use. The APC can in part be determined by estimating the measurement uncertainty component under conditions of within-laboratory precision (uRw), which comprises all components influencing the measurement uncertainty of random sources. To maintain the adequacy of their measurement procedures, laboratories need to distinguish aspects that are manageable vs. those that are not. One of the aspects that may influence uRw is the momentary significant bias caused by shifts in reagent and/or calibrator lots, which, when accepted or unnoticed, become a factor of the APC. In this paper, we postulate a model for allocating a part of allowable uRw to between-reagent lot variation, based on the need for long-term consistency of the measurement variability for that specific measurand. The allocation manages the ratio between short-term and long-term variation and indicates laboratories when to reject or correct certain variations due to reagent lots.

Improving the laboratory result release process in the light of ISO 15189:2012 standard

The ISO 15189:2012 standard section 5.9.1 requires laboratories to review results before release, considering quality control, previous results, and clinical information, if any, and to issue documented procedures about it. While laboratory result reporting is generally regarded as part of the post-analytical phase, the result release process requires a general view of the total examination process. Reviewing test results may follow with troubleshooting and test repetition, including reanalyzing an individual sample or resampling. A systematic understanding of the result release may help laboratory professionals carry out appropriate test repetition and ensure the plausibility of laboratory results. In this paper, we addressed the crucial steps in the result release process, including evaluation of sample quality, critical result notification, result reporting, and recommendations for the management of the result release, considering quality control alerts, instrument flags, warning messages, and interference indexes. Error detection tools and plausibility checks mentioned in the present paper can support the daily practice of results release.

Validation and verification of examination procedures in medical laboratories: opinion of the EFLM Working Group Accreditation and ISO/CEN standards (WG-A/ISO) on dealing with ISO 15189:2012 demands for method verification and validation

This paper reflects the opinion of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group Accreditation and ISO/CEN standards (WG-A/ISO). It aims to provide guidance for drawing up local/national documents about validation and verification of laboratory methods. We demonstrate how risk evaluation can be used to optimize laboratory policies to meet intended use requirements as well as requirements of standards. This is translated in a number of recommendations on how to introduce risk evaluation in various stages of the implementation of new methods ultimately covering the whole process cycle.

Documenting metrological traceability as intended by ISO 15189:2012: A consensus statement about the practice of the implementation and auditing of this norm element

ISO15189:2012 requires medical laboratories to document metrological traceability of their results. While the ISO17511:2003 standard on metrological traceability in laboratory medicine requires the use of the highest available level in the traceability chain, it recognizes that for many measurands there is no reference above the manufacturer’s selected measurement procedure and the manufacturer’s working calibrator. Some immunoassays, although they intend to measure the same quantity and may even refer to the same reference material, unfortunately produce different results because of differences in analytical selectivity as manufacturers select different epitopes and antibodies for the same analyte. In other cases, the cause is the use of reference materials, which are not commutable. The uncertainty associated with the result is another important aspect in metrological traceability implementation. As the measurement uncertainty on the clinical samples is influenced by the uncertainty of all steps higher in the traceability chain, laboratories should be provided with adequate and appropriate information on the uncertainty of the value assignment to the commercial calibrators that they use. Although the between-lot variation in value assignment will manifest itself as part of the long-term imprecision as estimated by the end-user, information on worst-case to be expected lot-lot variation has to be communicated to the end-user by the IVD provider. When laboratories use ancillary equipment that potentially could have a critical contribution to the reported results, such equipment needs verification of its proper calibration and criticality to the result uncertainty could be assessed by an approach based on risk analysis, which is a key element of ISO15189:2012 anyway. This paper discusses how the requirement for metrological traceability as stated in ISO15189 should be met by the medical laboratory and how this should be assessed by accreditation bodies.

Determination of testosterone in serum by liquid chromatography-tandem mass spectrometry

Commercial direct immunoassays for serum testosterone sometimes result in inaccuracies in samples from women and children, leading to misdiagnosis and inappropriate treatment. The diagnosis of male hypogonadism also requires an accurate testosterone assay method. We therefore developed a sensitive and specific stable-isotope dilution liquid chromatography-tandem mass spectrometric (LC-MS/MS) method for serum testosterone at the concentrations encountered in women and children. Testosterone was extracted with ether-ethyl acetate from 250 microL or 500 microL of serum. Instrumental analysis was performed on an API 2000 tandem mass spectrometer in the multiple-reaction monitoring (MRM) mode after separation on a reversed-phase column. The MRM transitions (m/z) were 289/97 for testosterone and 291/99 for d(2) testosterone. The calibration curves exhibited consistent linearity and repeatability in the range 0.2-100 nmol/L. Interassay CVs were 4.2-7.6 % at mean concentrations of testosterone of 3.3-45 nmol/L. Total measurement uncertainty (U, k = 2) was 12.9 % and 13.4 % at testosterone levels of 2.0 nmol/L and 20 nmol/L, respectively. The limit of detection was 0.05 nmol/L (signal-to-noise ratio = 3) and the overall method recovery of testosterone was 95 %. Correlation (r) with our in-house extraction RIA was 0.98 and with a commercial RIA 0.92. Reference intervals for adult males and females in age groups 18-30, 31-50, 51-70 and over 70 years were established. Sensitivity and specificity of the LC-MS/MS method offer advantages over immunoassay and make it suitable for use as a high-throughput assay in routine clinical laboratories. The high equipment costs are balanced by higher throughput together with shorter chromatographic run times.

Boron detection from blood samples by ICP-AES and ICP-MS during boron neutron capture therapy

OBJECTIVE

The concept of boron neutron capture therapy (BNCT) involves infusion of a (10)B containing tracer into the patient's bloodstream followed by local neutron irradiation(s). Accurate estimation of the blood boron level for the treatment field before irradiation is required. Boron concentration can be quantified by inductively coupled plasma atomic emission spectrometry (ICP-AES), mass spectrometry (ICP-MS), spectrofluorometric and direct current atomic emission spectrometry (DCP-AES) or by prompt gamma photon detection methods.

MATERIAL AND METHODS

The blood boron concentrations were analysed and compared using ICP-AES and ICP-MS to ensure congruency of the results if the analysis had to be changed during the treatment, e.g. for technical reasons. The effect of wet-ashing on the results was studied in addition.

RESULTS

The mean of all samples analysed with ICP-MS was 5.8 % lower than with ICP-AES coupled to wet-ashing (R (2) = 0.88). Without wet-ashing, the mean of all samples analysed with ICP-MS was 9.1 % higher than with ICP-AES (R (2) = 0.99).

CONCLUSIONS

Boron concentration analysed from whole blood samples with ICP-AES correlated well with the values of ICP-MS with wet-ashing of the sample matrix, which is generally considered the reference method. When using these methods in parallel at certain intervals during the treatments, reliability of the blood boron concentration values remains satisfactory, taking into account the required accuracy of dose determination in the irradiation of cancer patients.

Analytical performance of the Iris iQ200 automated urine microscopy analyzer

BACKGROUND

We evaluated the Iris iQ200 Automated Urine Microscopy Analyzer to find out if the instrument performed better than traditional visual bright field microscopy in detecting basic urine particles, as assessed against reference phase contrast microscopy.

METHODS

The HUSLAB quality system was followed in planning and completing the evaluation process. The iQ200 instrument results from 167 mid-stream, uncentrifuged urine specimens were compared to those obtained with phase contrast reference microscopy, and to those with routine bright field microscopy. Linearity, carry-over and precision were tested according to well-established protocols.

RESULTS

The iQ200 counted erythrocytes (RBC) at r=0.894 (R(2)=0.799) with Automated Particle Recognition (APR) software alone and at r=0.948 (R(2)=0.898) after re-classification. The performance for leukocytes (WBC) was r=0.885 with APR and r=0.978 after re-classification. The correlations of counting after user re-classification were r=0.927 for squamous epithelial cells (SQEP), r=0.856 for casts, and r=0.706 for non-squamous epithelial cells. The iQ200 showed good linearity and precision and no carry-over was detected.

CONCLUSIONS

The Iris iQ200 was capable to count reliably RBC, WBC, and SQEP cells and to identify a fraction of bacteria and renal elements. Counting results equalled or exceeded that of routine bright field microscopy or earlier flow cytometric technology. The instrument eliminates manual sample preparation but requires a well-trained technologist for re-grouping of findings.

Evaluation of uncertainty of measurement in routine clinical chemistry--applications to determination of the substance concentration of calcium and glucose in serum

We studied the uncertainty of measurement for the calcium and glucose (amount of) substance concentrations in serum. The evaluation follows a four-step procedure, which complies with the ISO document Guide to the Expression of Uncertainty in Measurement (GUM). The applications were chosen to represent commonly used measuring systems in medical laboratories. The uncertainty components are quantified using observations of the measuring system, and information from calibration certificates, instrument specifications and literature. The evaluation focuses on the measurement step but empirical terms are used to illustrate how the pre-analytical phase and patient-related issues can be accounted for. The software GUM Workbench was used to facilitate calculations and to visualize the importance of each uncertainty component. The combined standard uncertainties (u(c)) for the measurands were < or =2% including the pre-analytical uncertainty sources. The patient-related source is discussed in relation to clinician's diagnosis and decision-making. The evaluation, as carried out here for calcium and glucose substance concentration measurements, can easily be applied to many other measurands in clinical chemistry. This work emphasizes that the internal quality control can provide much of the information needed in the uncertainty evaluation, and that external quality assessment (EQA) schemes are important in the control of the uncertainty evaluated by the individual laboratories. Due to statistical and metrological limitations routine EQA schemes should themselves not be used as a means of uncertainty evaluation.

Factors affecting the cytology outcome of Pap smears--a brief approach to internal quality control in private cytopathology laboratory practice

We evaluated the effect of intrauterine device (IUD), patient age and hormone replacement therapy (HRT) on cytology outcome in Pap smears together with the important IQC procedures: 1) manual double-screening by cytotechnologists, and 2) retrospective senior pathologist review during 1996-1999. The results from primary double-screening (119 of 87,409 Pap smears) showed an excellent inter-observer correlation. The estimation of hormonal effects showed higher incidence of disagreements (p=0.013) in patients <47 yr. Some individual trends were found in the assessments of both cellular atypia (p=0.012) and Papanicolaou classification (p=0.018). The IUD had no influence on the accuracy when the degree of inflammatory reaction was evaluated (p>0.050), but showed an adverse effect on the estimation of cellular atypia (p=0.001). HRT distinctly equalized the entire sample material, since fewer disagreements were found in the age groups <47 yr and >47 yr when estimating the hormonal effects (p=0.013), inflammatory reaction (p=0.044) or cellular atypia (p=0.006) compared to those without HRT. The continuous cytopathologist supervision had a positive impact on the accuracy of hormonal effect estimation during the 4 years. The senior cytopathologists' reviews (354 of 87,409 Pap smears) showed mutually good interobserver correlation, and diagnostic conclusions of the same specimens differed only slightly between the cytopathologists. We found these state-of-the-art cytopathological IQC procedures to be effective and fit-for-purpose when evaluating hormonal effects, inflammatory reaction and cellular atypia.

Optimisation of media and cultivation conditions for L(+)(S)-lactic acid production by Lactobacillus casei NRRL B-441

Process variables and concentration of carbon in media were optimised for lactic acid production by Lactobacillus casei NRRL B-441. Lactic acid yield was inversely proportional to initial glucose concentration within the experimental area (80-160 g l(-1)). The highest lactic acid concentration in batch fermentation, 118.6 g l(-1), was obtained with 160 g 1(-1) glucose. The maximum volumetric productivity, 4.4 g 1(-1) h(-1) at 15 h, was achieved at an initial glucose concentration of 100 g l(-1). Similar lactic acid concentrations were reached with a fedbatch approach using growing cells, in which case the fermentation time was much shorter. Statistical experimental design and response surface methodology were used for optimising the process variables. The temperature and pH optima for lactic acid production were 35 degrees C, pH 6.3. Malt sprout extract supplemented with yeast extract (4 g l(-1)) appeared to be an economical alternative to yeast extract alone (22 g l(-1)) although the fermentation time was a little longer. The results demonstrated both the separation of the growth and lactic acid production phases and lactic acid production by non-growing cells without any nutrient supplements. Resting L. casei cells converted 120 g l(-1) glucose to lactic acid with 100% yield and a maximum volumetric productivity of 3.5 g l(-1) h(-1).

Applying neural networks as software sensors for enzyme engineering

The on-line control of enzyme-production processes is difficult, owing to the uncertainties typical of biological systems and to the lack of suitable on-line sensors for key process variables. For example, intelligent methods to predict the end point of fermentation could be of great economic value. Computer-assisted control based on artificial-neural-network models offers a novel solution in such situations. Well-trained feedforward-backpropagation neural networks can be used as software sensors in enzyme-process control; their performance can be affected by a number of factors.

Neural networks as "software sensors" in enzyme engineering

Industrial applications of enzyme technology are rapidly increasing. On-line control of enzyme production processes, however, is difficult owing to the uncertainties typical of biological systems and to the lack of suitable on-line sensors for key process variables and quality attributes. We demonstrate that well-trained feedforward backpropagation neural networks with one hidden layer can be employed to overcome such problems with no need for a priori knowledge of the relationships of the process variables involved. Neural network programs were written in Microsoft Visual C++ for Windows and implemented in a personal computer. The goodness of fit of the trained neural network to the reference data was determined by the coefficient of determination, R2. Case studies of beta-galactosidase, glucoamylase, lipase, and xylanase production processes will be used as examples.

Assessment of the state-of-the-art trueness and precision of serum total-calcium and glucose measurements in Finnish laboratories--the QSL-Finland study

The purpose of the QSL-Finland study was to assess the state-of-the-art trueness and precision of serum total-calcium and glucose measurements in Finnish clinical laboratories. For this purpose, 21 hospitals and clinical institutes were selected. They measured six single donation sera, the total-calcium (t-calcium) and glucose content of which had been determined by ion chromatography and isotope dilution-gas chromatography-mass spectrometry (ID-GC-MS) reference methods. The results were interpreted in light of specifications for imprecision, bias and total error of routine methods that have been proposed in the past. The data revealed that the performance of t-calcium and glucose methods is generally acceptable in Finnish clinical laboratories. This study did not lead to a separation of laboratories according to accreditation. In consequence, it seems that accreditation, in its present form, cannot substitute dedicated quality assurance practices.

Neural networks as 'software sensors' in enzyme production

Industrial applications of enzyme technology are rapidly increasing. On-line control of enzyme production processes, however, is difficult, owing to the uncertainties typical of biological reactions and to the lack of suitable sensors. We demonstrate that well-trained feedforward backpropagation neural networks with one hidden layer can be employed to overcome such problems with no need for a priori knowledge of the relationships of the process variables involved. Neural network programs were written in Microsoft Visual C+2 for Windows and implemented in a personal computer. The goodness of fit of the trained neural network to the reference data was determined by the coefficient of determination R2. On-line state estimation and multi-step ahead prediction of enzyme activity and biomass concentration, both in a yeast lipase and fungal glucoamylase production could be satisfactorily carried out. Results showed an excellent fit for estimated lipase activity (R2 = 0.988) and biomass concentration (R2 = 0.989). In glucoamylase production, both enzyme activity and biomass concentration could also be reliably predicted for 2 time intervals (10 h) ahead with only on-line measurable parameter values as the input data.

Production of Phanerochaete chrysosporium lignin peroxidase

Liginin peroxidase (ligninase) of the white rot fungus Phanerochaete chrysosporium Burdsall was discovered in 1982 as a secondary metabolite. Today multiple isoenzymes are known, which are often collectively called as lignin peroxidase. Lignin peroxidase has been characterized as a veratryl alcohol oxidizing enzyme, but it is a relatively unspecific enzyme catalyzing a variety of reactions with hydrogen peroxide as the electron acceptor. P. chrysosporium ligninases are heme glycoproteins. At least a number of isoenzymes are also phosphorylated. Two of the major isoenzymes have been crystallized. Until recently lignin peroxidase could only be produced in low yields in very small scale stationary cultures owing to shear sensitivity. Most strains produce the enzyme only after grown under nitrogen or carbon limitation, although strains producing lignin peroxidase under nutrient sufficiency have also been isolated. Activities over 2000 U dm(-3) (as determined at 30 degrees to 37 degrees C) have been reported in small scale Erlenmeyer cultures with the strain INA-12 grown on glycerol in the presence of soybean phospholipids under nitrogen sufficiency. In about 8 dm(3) liquid volume pilot scale higher than 100 U dm(-3) (as determined at 23 degrees C) have been obtained under agitation with immobilized P. chrysosporium strains ATCC 24725 or TKK 20512. Good results have been obtained for example with nylon web, polyurethane foam, sintered glass or silicon tubing as the carrier. The immobilized biocatalyst systems have also made large scale repeated batch and semicontinuous production possible. With nylon web as the carrier, lignin peroxidase production has recently been scaled up to 800 dm(3) liquid volume semicontinuous industrial production process.