Machine Learning in Laboratory Medicine: Recommendations of the IFCC Working Group

Machine Learning-Based Sample Misidentification Error Detection in Clinical Laboratory Tests: A Retrospective Multicenter Study

BACKGROUND

In clinical laboratories, the precision and sensitivity of autoverification technologies are crucial for ensuring reliable diagnostics. Conventional methods have limited sensitivity and applicability, making error detection challenging and reducing laboratory efficiency. This study introduces a machine learning (ML)-based autoverification technology to enhance tumor marker test error detection.

METHODS

The effectiveness of various ML models was evaluated by analyzing a large data set of 397 751 for model training and internal validation and 215 339 for external validation. Sample misidentification was simulated by random shuffling error-free test results with a 1% error rate to achieve a real-world approximation. The ML models were developed with Bayesian optimization for tuning. Model validation was performed internally at the primary institution and externally at other institutions, comparing the ML models' performance with conventional delta check methods.

RESULTS

Deep neural networks and extreme gradient boosting achieved an area under the receiver operating characteristic curve of 0.834 to 0.903, outperforming that of conventional methods (0.705 to 0.816). External validation by 3 independent laboratories showed that the balanced accuracy of the ML model ranged from 0.760 to 0.836, outperforming the balanced accuracy of 0.670 to 0.773 of the conventional models.

CONCLUSIONS

This study addresses limitations regarding the sensitivity of current delta check methods for detection of sample misidentification errors and provides versatile models that mitigate the operational challenges faced by smaller laboratories. Our findings offer a pathway toward more efficient and reliable clinical laboratory testing.

Non-Invasive Endometrial Cancer Screening through Urinary Fluorescent Metabolome Profile Monitoring and Machine Learning Algorithms

Endometrial cancer is becoming increasingly common, highlighting the need for improved diagnostic methods that are both effective and non-invasive. This study investigates the use of urinary fluorescence spectroscopy as a potential diagnostic tool for endometrial cancer. Urine samples were collected from endometrial cancer patients (n = 77), patients with benign uterine tumors (n = 23), and control gynecological patients attending regular checkups or follow-ups (n = 96). These samples were analyzed using synchronous fluorescence spectroscopy to measure the total fluorescent metabolome profile, and specific fluorescence ratios were created to differentiate between control, benign, and malignant samples. These spectral markers demonstrated potential clinical applicability with AUC as high as 80%. Partial Least Squares Discriminant Analysis (PLS-DA) was employed to reduce data dimensionality and enhance class separation. Additionally, machine learning models, including Random Forest (RF), Logistic Regression (LR), Support Vector Machine (SVM), and Stochastic Gradient Descent (SGD), were utilized to distinguish between controls and endometrial cancer patients. PLS-DA achieved an overall accuracy of 79% and an AUC of 90%. These promising results indicate that urinary fluorescence spectroscopy, combined with advanced machine learning models, has the potential to revolutionize endometrial cancer diagnostics, offering a rapid, accurate, and non-invasive alternative to current methods.

A vision to the future: value-based laboratory medicine

The ultimate goal of value-based laboratory medicine is maximizing the effectiveness of laboratory tests in improving patient outcomes, optimizing resources and minimizing unnecessary costs. This approach abandons the oversimplified notion of test volume and cost, in favor of emphasizing the clinical utility and quality of diagnostic tests in the clinical decision-making. Several key elements characterize value-based laboratory medicine, which can be summarized in some basic concepts, such as organization of in vitro diagnostics (including appropriateness, integrated diagnostics, networking, remote patient monitoring, disruptive innovations), translation of laboratory data into clinical information and measurable outcomes, sustainability, reimbursement, ethics (e.g., patient empowerment and safety, data protection, analysis of big data, scientific publishing). Education and training are also crucial, along with considerations for the future of the profession, which will be largely influenced by advances in automation, information technology, artificial intelligence, and regulations concerning in vitro diagnostics. This collective opinion paper, composed of summaries from presentations given at the two-day European Federation of Laboratory Medicine (EFLM) Strategic Conference "A vision to the future: value-based laboratory medicine" (Padova, Italy; September 23-24, 2024), aims to provide a comprehensive overview of value-based laboratory medicine, projecting the profession into a more clinically effective and sustainable future.

Validating, Implementing, and Monitoring Machine Learning Solutions in the Clinical Laboratory Safely and Effectively

BACKGROUND

Machine learning solutions offer tremendous promise for improving clinical and laboratory operations in pathology. Proof-of-concept descriptions of these approaches have become commonplace in laboratory medicine literature, but only a scant few of these have been implemented within clinical laboratories, owing to the often substantial barriers in validating, implementing, and monitoring these applications in practice. This mini-review aims to highlight the key considerations in each of these steps.

CONTENT

Effective and responsible applications of machine learning in clinical laboratories require robust validation prior to implementation. A comprehensive validation study involves a critical evaluation of study design, data engineering and interoperability, target label definition, metric selection, generalizability and applicability assessment, algorithmic fairness, and explainability. While the main text highlights these concepts in broad strokes, a supplementary code walk-through is also provided to facilitate a more practical understanding of these topics using a real-world classification task example, the detection of saline-contaminated chemistry panels.Following validation, the laboratorian's role is far from over. Implementing machine learning solutions requires an interdisciplinary effort across several roles in an organization. We highlight the key roles, responsibilities, and terminologies for successfully deploying a validated solution into a live production environment. Finally, the implemented solution must be routinely monitored for signs of performance degradation and updated if necessary.

SUMMARY

This mini-review aims to bridge the gap between theory and practice by highlighting key concepts in validation, implementation, and monitoring machine learning solutions effectively and responsibly in the clinical laboratory.

Machine Learning for Patient-Based Real-Time Quality Control (PBRTQC), Analytical and Preanalytical Error Detection in Clinical Laboratory

The rapidly evolving field of machine learning (ML), along with artificial intelligence in a broad sense, is revolutionising many areas of healthcare, including laboratory medicine. The amalgamation of the fields of ML and patient-based real-time quality control (PBRTQC) processes could improve the traditional PBRTQC and error detection algorithms in the laboratory. This narrative review discusses published studies on using ML for the detection of systematic errors, non-systematic errors, and combinations of different types of errors in clinical laboratories. The studies discussed used ML for detecting bias, the requirement for re-calibration, samples contaminated with intravenous fluid or EDTA, delayed sample analysis, wrong-blood-in-tube errors, interference or a combination of different types of errors, by comparing the performance of ML models with human validators or traditional PBRTQC algorithms. Advantages, limitations, the creation of standardised ML models, ethical and regulatory aspects and potential future developments have also been discussed in brief.

An artificial intelligence-driven support tool for prediction of urine culture test results

BACKGROUND AND AIMS

We aimed to develop an easily deployable artificial intelligence (AI)-driven model for rapid prediction of urine culture test results.

MATERIAL AND METHODS

We utilized a training dataset (n = 34,584 urine samples) and two separate, unseen test sets (n = 10,083 and 9,289 samples). Various machine learning models were compared for diagnostic performance. Predictive parameters included urinalysis results (dipstick and flow cytometry), patient demographics (age and gender), and sample collection method.

RESULTS

Although more complex models achieved the highest AUCs for predicting positive cultures (highest: multilayer perceptron (MLP) with AUC of 0.884, 95% CI 0.878-0.89), multiple logistic regression (MLR) using only flow cytometry parameters achieved a very good AUC (0.858, 95% CI 0.852-0.865). To aid interpretation, prediction results of the MLP and MLR models were categorized based on likelihood ratio (LR) for positivity: highly unlikely (LR 0.1), unlikely (LR 0.3), grey zone (LR 0.9), likely (LR 5.0), and highly likely (LR 40). This resulted in 17%, 28%, 34%, 9%, and 13% of samples falling into each respective category for the MLR model and 20%, 26%, 31%, 7%, and 16% for the MLP model.

CONCLUSIONS

In conclusion, this robust model has the potential to assist clinicians in their decision-making process by providing insights prior to the availability of urine culture results in a significant portion of samples (∼2/3rd).

Biomarkers vs Machines: The Race to Predict Acute Kidney Injury

BACKGROUND

Acute kidney injury (AKI) is a serious complication affecting up to 15% of hospitalized patients. Early diagnosis is critical to prevent irreversible kidney damage that could otherwise lead to significant morbidity and mortality. However, AKI is a clinically silent syndrome, and current detection primarily relies on measuring a rise in serum creatinine, an imperfect marker that can be slow to react to developing AKI. Over the past decade, new innovations have emerged in the form of biomarkers and artificial intelligence tools to aid in the early diagnosis and prediction of imminent AKI.

CONTENT

This review summarizes and critically evaluates the latest developments in AKI detection and prediction by emerging biomarkers and artificial intelligence. Main guidelines and studies discussed herein include those evaluating clinical utilitiy of alternate filtration markers such as cystatin C and structural injury markers such as neutrophil gelatinase-associated lipocalin and tissue inhibitor of metalloprotease 2 with insulin-like growth factor binding protein 7 and machine learning algorithms for the detection and prediction of AKI in adult and pediatric populations. Recommendations for clinical practices considering the adoption of these new tools are also provided.

SUMMARY

The race to detect AKI is heating up. Regulatory approval of select biomarkers for clinical use and the emergence of machine learning algorithms that can predict imminent AKI with high accuracy are all promising developments. But the race is far from being won. Future research focusing on clinical outcome studies that demonstrate the utility and validity of implementing these new tools into clinical practice is needed.

Artificial intelligence in the clinical laboratory

Laboratory medicine has become a highly automated medical discipline. Nowadays, artificial intelligence (AI) applied to laboratory medicine is also gaining more and more attention, which can optimize the entire laboratory workflow and even revolutionize laboratory medicine in the future. However, only a few commercially available AI models are currently approved for use in clinical laboratories and have drawbacks such as high cost, lack of accuracy, and the need for manual review of model results. Furthermore, there are a limited number of literature reviews that comprehensively address the research status, challenges, and future opportunities of AI applications in laboratory medicine. Our article begins with a brief introduction to AI and some of its subsets, then reviews some AI models that are currently being used in clinical laboratories or that have been described in emerging studies, and explains the existing challenges associated with their application and possible solutions, finally provides insights into the future opportunities of the field. We highlight the current status of implementation and potential applications of AI models in different stages of the clinical testing process.

Machine learning-based clinical decision support using laboratory data

Artificial intelligence (AI) and machine learning (ML) are becoming vital in laboratory medicine and the broader context of healthcare. In this review article, we summarized the development of ML models and how they contribute to clinical laboratory workflow and improve patient outcomes. The process of ML model development involves data collection, data cleansing, feature engineering, model development, and optimization. These models, once finalized, are subjected to thorough performance assessments and validations. Recently, due to the complexity inherent in model development, automated ML tools were also introduced to streamline the process, enabling non-experts to create models. Clinical Decision Support Systems (CDSS) use ML techniques on large datasets to aid healthcare professionals in test result interpretation. They are revolutionizing laboratory medicine, enabling labs to work more efficiently with less human supervision across pre-analytical, analytical, and post-analytical phases. Despite contributions of the ML tools at all analytical phases, their integration presents challenges like potential model uncertainties, black-box algorithms, and deskilling of professionals. Additionally, acquiring diverse datasets is hard, and models' complexity can limit clinical use. In conclusion, ML-based CDSS in healthcare can greatly enhance clinical decision-making. However, successful adoption demands collaboration among professionals and stakeholders, utilizing hybrid intelligence, external validation, and performance assessments.

Analytical Performance Specifications for Input Variables: Investigation of the Model of End-Stage Liver Disease

BACKGROUND

Artificial intelligence models constitute specific uses of analysis results and, therefore, necessitate evaluation of analytical performance specifications (APS) for this context specifically. The Model of End-stage Liver Disease (MELD) is a clinical prediction model based on measurements of bilirubin, creatinine, and the international normalized ratio (INR). This study evaluates the propagation of error through the MELD, to inform choice of APS for the MELD input variables.

METHODS

A total of 6093 consecutive MELD scores and underlying analysis results were retrospectively collected. "Desirable analytical variation" based on biological variation as well as current local analytical variation was simulated onto the data set as well as onto a constructed data set, representing a worst-case scenario. Resulting changes in MELD score and risk classification were calculated.

RESULTS

Biological variation-based APS in the worst-case scenario resulted in 3.26% of scores changing by ≥1 MELD point. In the patient-derived data set, the same variation resulted in 0.92% of samples changing by ≥1 MELD point, and 5.5% of samples changing risk category. Local analytical performance resulted in lower reclassification rates.

CONCLUSIONS

Error propagation through MELD is complex and includes population-dependent mechanisms. Biological variation-derived APS were acceptable for all uses of the MELD score. Other combinations of APS can yield equally acceptable results. This analysis exemplifies how error propagation through artificial intelligence models can become highly complex. This complexity will necessitate that both model suppliers and clinical laboratories address analytical performance specifications for the specific use case, as these may differ from performance specifications for traditional use of the analyses.

Working with the Electronic Health Record and Laboratory Information System to Maximize Ordering and Reporting of Molecular Microbiology Results

Molecular microbiology assays have a higher cost of testing compared to traditional methods and need to be utilized appropriately. Results from these assays may also require interpretation and appropriate follow-up. Electronic tools available in the electronic health record and laboratory information system can be deployed both preanalytically and postanalytically to influence ordering behaviors and positively impact diagnostic stewardship. Next generation technologies, such as machine learning and artificial intelligence, have the potential to expand upon the capabilities currently available and warrant additional study and development but also require regulation around their use in health care.

Effect size and inferential statistical techniques coupled with machine learning for assessing the association between prolactin concentration and metabolic homeostasis

BACKGROUND AND OBJECTIVE

Recent guidelines classify low prolactin levels as low as <7 ng/mL and high levels as >25 ng/mL, while the "Homeostatically Functionally Increased Transient Prolactinemia" (HomeoFIT-PRL) range (25-100 ng/mL) suggests that a temporary increase in prolactin could be metabolically beneficial if no related health issues are present. The aim of this study was to investigate the association between mean prolactin concentrations and disturbances in glycidic and lipidic metabolism and to identify the gray zone associated with prolactin inflection points that correlate with these metabolic changes.

METHODS

This cross-sectional study involved 65,795 adults who underwent HOMA-IR, glucose, insulin, total cholesterol, HDL-c, LDL-c, and triglyceride tests. Data was categorized into 106 partitions based on prolactin results. Employing an approach referred to in this study as "Hierarchical Multicriteria Analysis of Differences Between Groups - Statistical and Effect Size Approach" (HiMADiG-SESA) comparing the mean concentrations of metabolic tests across prolactin ranges. A machine learning model was utilized to determine inflection points and their corresponding confidence intervals (CIs). These CIs helped establish gray zones in mean prolactin results related to metabolic changes.

RESULTS

Statistically and clinically, metabolic test means differed for prolactin <7 ng/mL, except insulin. In the HomeoFIT-PRL range, means were lower except for HDL-c. The gray zones of the mean prolactin results associated with changes in glycidic and lipidic metabolism were 9.58-12.87 ng/mL and 13.81-18.73 ng/mL, respectively.

CONCLUSION

A strong correlation was identified between mean prolactin concentrations and the results of metabolism tests below the gray zones associated with inflection points, indicating the potential role of prolactin in the appearance of metabolic disorders. Mean prolactin results can provide deeper insight into metabolic balance.

Applications of Artificial Intelligence in Urinalysis: Is the Future Already Here?

BACKGROUND

Artificial intelligence (AI) has emerged as a promising and transformative tool in the field of urinalysis, offering substantial potential for advancements in disease diagnosis and the development of predictive models for monitoring medical treatment responses.

CONTENT

Through an extensive examination of relevant literature, this narrative review illustrates the significance and applicability of AI models across the diverse application area of urinalysis. It encompasses automated urine test strip and sediment analysis, urinary tract infection screening, and the interpretation of complex biochemical signatures in urine, including the utilization of cutting-edge techniques such as mass spectrometry and molecular-based profiles.

SUMMARY

Retrospective studies consistently demonstrate good performance of AI models in urinalysis, showcasing their potential to revolutionize clinical practice. However, to comprehensively evaluate the real clinical value and efficacy of AI models, large-scale prospective studies are essential. Such studies hold the potential to enhance diagnostic accuracy, improve patient outcomes, and optimize medical treatment strategies. By bridging the gap between research and clinical implementation, AI can reshape the landscape of urinalysis, paving the way for more personalized and effective patient care.

Generalizability of a Machine Learning Model for Improving Utilization of Parathyroid Hormone-Related Peptide Testing across Multiple Clinical Centers

BACKGROUND

Measuring parathyroid hormone-related peptide (PTHrP) helps diagnose the humoral hypercalcemia of malignancy, but is often ordered for patients with low pretest probability, resulting in poor test utilization. Manual review of results to identify inappropriate PTHrP orders is a cumbersome process.

METHODS

Using a dataset of 1330 patients from a single institute, we developed a machine learning (ML) model to predict abnormal PTHrP results. We then evaluated the performance of the model on two external datasets. Different strategies (model transporting, retraining, rebuilding, and fine-tuning) were investigated to improve model generalizability. Maximum mean discrepancy (MMD) was adopted to quantify the shift of data distributions across different datasets.

RESULTS

The model achieved an area under the receiver operating characteristic curve (AUROC) of 0.936, and a specificity of 0.842 at 0.900 sensitivity in the development cohort. Directly transporting this model to two external datasets resulted in a deterioration of AUROC to 0.838 and 0.737, with the latter having a larger MMD corresponding to a greater data shift compared to the original dataset. Model rebuilding using site-specific data improved AUROC to 0.891 and 0.837 on the two sites, respectively. When external data is insufficient for retraining, a fine-tuning strategy also improved model utility.

CONCLUSIONS

ML offers promise to improve PTHrP test utilization while relieving the burden of manual review. Transporting a ready-made model to external datasets may lead to performance deterioration due to data distribution shift. Model retraining or rebuilding could improve generalizability when there are enough data, and model fine-tuning may be favorable when site-specific data is limited.