Identifying mislabelled samples: Machine learning models exceed human performance

Machine Learning for Patient-Based Real-Time Quality Control (PBRTQC), Analytical and Preanalytical Error Detection in Clinical Laboratory

The rapidly evolving field of machine learning (ML), along with artificial intelligence in a broad sense, is revolutionising many areas of healthcare, including laboratory medicine. The amalgamation of the fields of ML and patient-based real-time quality control (PBRTQC) processes could improve the traditional PBRTQC and error detection algorithms in the laboratory. This narrative review discusses published studies on using ML for the detection of systematic errors, non-systematic errors, and combinations of different types of errors in clinical laboratories. The studies discussed used ML for detecting bias, the requirement for re-calibration, samples contaminated with intravenous fluid or EDTA, delayed sample analysis, wrong-blood-in-tube errors, interference or a combination of different types of errors, by comparing the performance of ML models with human validators or traditional PBRTQC algorithms. Advantages, limitations, the creation of standardised ML models, ethical and regulatory aspects and potential future developments have also been discussed in brief.

Machine learning-based delta check method for detecting misidentification errors in tumor marker tests

OBJECTIVES

Misidentification errors in tumor marker tests can lead to serious diagnostic and treatment errors. This study aims to develop a method for detecting these errors using a machine learning (ML)-based delta check approach, overcoming limitations of conventional methods.

METHODS

We analyzed five tumor marker test results: alpha-fetoprotein (AFP), cancer antigen 19-9 (CA19-9), cancer antigen 125 (CA125), carcinoembryonic antigen (CEA), and prostate-specific antigen (PSA). A total of 246,261 records were used in the analysis. Of these, 179,929 records were used for model training and 66,332 records for performance evaluation. We developed a misidentification error detection model based on the random forest (RF) and deep neural network (DNN) methods. We performed an in silico simulation with 1 % random sample shuffling. The performance of the developed models was evaluated and compared to conventional delta check methods such as delta percent change (DPC), absolute DPC (absDPC), and reference change values (RCV).

RESULTS

The DNN model outperformed the RF, DPC, absDPC, and RCV methods in detecting sample misidentification errors. It achieved balanced accuracies of 0.828, 0.842, 0.792, 0.818, and 0.833 for AFP, CA19-9, CA125, CEA, and PSA, respectively. Although the RF method performed better than DPC and absDPC, it showed similar or lower performance compared to RCV.

CONCLUSIONS

Our research results demonstrate that an ML-based delta check method can more effectively detect sample misidentification errors compared to conventional delta check methods. In particular, the DNN model demonstrated superior and stable detection performance compared to the RF, DPC, absDPC, and RCV methods.

Artificial intelligence in the clinical laboratory

Laboratory medicine has become a highly automated medical discipline. Nowadays, artificial intelligence (AI) applied to laboratory medicine is also gaining more and more attention, which can optimize the entire laboratory workflow and even revolutionize laboratory medicine in the future. However, only a few commercially available AI models are currently approved for use in clinical laboratories and have drawbacks such as high cost, lack of accuracy, and the need for manual review of model results. Furthermore, there are a limited number of literature reviews that comprehensively address the research status, challenges, and future opportunities of AI applications in laboratory medicine. Our article begins with a brief introduction to AI and some of its subsets, then reviews some AI models that are currently being used in clinical laboratories or that have been described in emerging studies, and explains the existing challenges associated with their application and possible solutions, finally provides insights into the future opportunities of the field. We highlight the current status of implementation and potential applications of AI models in different stages of the clinical testing process.

Machine learning-based clinical decision support using laboratory data

Artificial intelligence (AI) and machine learning (ML) are becoming vital in laboratory medicine and the broader context of healthcare. In this review article, we summarized the development of ML models and how they contribute to clinical laboratory workflow and improve patient outcomes. The process of ML model development involves data collection, data cleansing, feature engineering, model development, and optimization. These models, once finalized, are subjected to thorough performance assessments and validations. Recently, due to the complexity inherent in model development, automated ML tools were also introduced to streamline the process, enabling non-experts to create models. Clinical Decision Support Systems (CDSS) use ML techniques on large datasets to aid healthcare professionals in test result interpretation. They are revolutionizing laboratory medicine, enabling labs to work more efficiently with less human supervision across pre-analytical, analytical, and post-analytical phases. Despite contributions of the ML tools at all analytical phases, their integration presents challenges like potential model uncertainties, black-box algorithms, and deskilling of professionals. Additionally, acquiring diverse datasets is hard, and models' complexity can limit clinical use. In conclusion, ML-based CDSS in healthcare can greatly enhance clinical decision-making. However, successful adoption demands collaboration among professionals and stakeholders, utilizing hybrid intelligence, external validation, and performance assessments.

A Machine Learning Model for the Routine Detection of "Wrong Blood in Complete Blood Count Tube" Errors

BACKGROUND

Current laboratory procedures may fail to detect wrong blood in tube (WBIT) errors. Machine learning models have the potential to improve WBIT error detection, as demonstrated by proof-of-concept studies. The models developed so far, however, are not appropriate for routine use because they are unable to handle missing values and have low positive predictive value (PPV). In this study, a machine learning model suitable for routine use was developed.

METHODS

A model was trained and a preliminary evaluation performed on a retrospective data set of 135 128 current and previous patient complete blood count (CBC) results. The model was then applied prospectively to routine samples tested in a public hospital laboratory over a period of 22 weeks. Each week, the 5 samples identified by the model as most likely to be WBIT errors underwent further investigation by testing blood group and red cell phenotype. The study assessed the number of WBIT errors that were missed by current procedures but detected by the model, as well as the PPV of the model.

RESULTS

The model was applied prospectively to 38 187 CBC results that had passed routine laboratory checks. One hundred and ten samples were identified for further testing and 12 WBIT errors were detected. The PPV of the model was 10.9%.

CONCLUSION

A machine learning model suitable for routine use was able to identify WBIT errors missed by the laboratory's current procedures. Machine learning models are valuable for the identification of WBIT errors, and their validation and deployment in clinical laboratories would improve patient safety.

Delta checks

Delta check is an electronic error detection tool. It compares the difference in sequential results within a patient against a predefined limit, and when exceeded, the delta check rule is considered triggered. The patient results should be withheld for review and troubleshooting before releasing to the clinical team for patient management. Delta check was initially developed as a tool to detect wrong-blood-in-tube (sample misidentification) errors. It is now applied to detect errors more broadly within the total testing process. Recent advancements in the theoretical understanding of delta check has allowed for more precise application of this tool to achieve the desired clinical performance and operational set up. In this Chapter, we review the different pre-implementation considerations, the foundation concepts of delta check, the process of setting up key delta check parameters, performance verification and troubleshooting of a delta check flag.

Disruption vs. evolution in laboratory medicine. Current challenges and possible strategies, making laboratories and the laboratory specialist profession fit for the future

Since beginning of medical diagnostics, laboratory specialists have done an amazing job, continuously improving quality, spectrum and speed of laboratory tests, currently contributing to the majority of medical decision making. These improvements are mostly of an incremental evolutionary fashion, meaning improvements of current processes. Sometimes these evolutionary innovations are of a radical fashion, such as the invention of automated analyzers replacing manual testing or the implementation of mass spectrometry, leading to one big performance leap instead of several small ones. In few cases innovations may be of disruptive nature. In laboratory medicine this would be applicable to digitalization of medicine or the decoding of the human genetic material. Currently, laboratory medicine is again facing disruptive innovations or technologies, which need to be adapted to as soon as possible. One of the major disruptive technologies is the increasing availability and medical use of artificial intelligence. It is necessary to rethink the position of the laboratory specialist within healthcare settings and the added value he or she can provide to patient care. The future of the laboratory specialist profession is bright, as it the only medical profession comprising such vast experience in patient diagnostics. However, laboratory specialists need to develop strategies to provide this expertise, by adopting to the quickly evolving technologies and demands. This opinion paper summarizes some of the disruptive technologies as well as strategies to secure and/or improve the quality of diagnostic patient care and the laboratory specialist profession.

Artificial Intelligence Applications in Clinical Chemistry

Artificial intelligence (AI) applications are an area of active investigation in clinical chemistry. Numerous publications have demonstrated the promise of AI across all phases of testing including preanalytic, analytic, and postanalytic phases; this includes novel methods for detecting common specimen collection errors, predicting laboratory results and diagnoses, and enhancing autoverification workflows. Although AI applications pose several ethical and operational challenges, these technologies are expected to transform the practice of the clinical chemistry laboratory in the near future.

The preanalytical phase – from an instrument-centred to a patient-centred laboratory medicine

In order to guarantee patient safety, medical laboratories around the world strive to provide highest quality in the shortest amount of time. A major leap in quality improvement was achieved by aiming to avoid preanalytical errors within the total testing process. Although these errors were first described in the 1970s, it took additional years/decades for large-scale efforts, aiming to improve preanalytical quality by standardisation and/or harmonisation. Initially these initiatives were mostly on the local or national level. Aiming to fill this void, in 2011 the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) working group “Preanalytical Phase” (WG-PRE) was founded. In the 11 years of its existence this group was able to provide several recommendations on various preanalytical topics. One major achievement of the WG-PRE was the development of an European consensus guideline on venous blood collection. In recent years the definition of the preanalytical phase has been extended, including laboratory test selection, thereby opening a huge field for improvement, by implementing strategies to overcome misuse of laboratory testing, ideally with the support of artificial intelligence models. In this narrative review, we discuss important aspects and milestones in the endeavour of preanalytical process improvement, which would not have been possible without the support of the Clinical Chemistry and Laboratory Medicine (CCLM) journal, which was one of the first scientific journals recognising the importance of the preanalytical phase and its impact on laboratory testing quality and ultimately patient safety.

Clinlabomics: leveraging clinical laboratory data by data mining strategies

The recent global focus on big data in medicine has been associated with the rise of artificial intelligence (AI) in diagnosis and decision-making following recent advances in computer technology. Up to now, AI has been applied to various aspects of medicine, including disease diagnosis, surveillance, treatment, predicting future risk, targeted interventions and understanding of the disease. There have been plenty of successful examples in medicine of using big data, such as radiology and pathology, ophthalmology cardiology and surgery. Combining medicine and AI has become a powerful tool to change health care, and even to change the nature of disease screening in clinical diagnosis. As all we know, clinical laboratories produce large amounts of testing data every day and the clinical laboratory data combined with AI may establish a new diagnosis and treatment has attracted wide attention. At present, a new concept of radiomics has been created for imaging data combined with AI, but a new definition of clinical laboratory data combined with AI has lacked so that many studies in this field cannot be accurately classified. Therefore, we propose a new concept of clinical laboratory omics (Clinlabomics) by combining clinical laboratory medicine and AI. Clinlabomics can use high-throughput methods to extract large amounts of feature data from blood, body fluids, secretions, excreta, and cast clinical laboratory test data. Then using the data statistics, machine learning, and other methods to read more undiscovered information. In this review, we have summarized the application of clinical laboratory data combined with AI in medical fields. Undeniable, the application of Clinlabomics is a method that can assist many fields of medicine but still requires further validation in a multi-center environment and laboratory.

Application of artificial intelligence and machine learning technology for the prediction of postmortem interval: A systematic review of preclinical and clinical studies

BACKGROUND /PURPOSE

Establishing an accurate postmortem interval (PMI) is exceptionally crucial in forensic investigation. Artificial intelligence (AI) and Machine learning (ML) models are widely employed in forensic practice. ML is a part of AI, both terms are highly associated and sometimes used interchangeably. This systematic review aims to evaluate the application and performance of AI technology for the prediction of PMI.

METHODS

Systematic literature search across different electronic databases using PubMed/Google Scholar/EMBASE/Scopus/CINAHL/Web of Science/Cochrane library was conducted from inception to 3 December 2021 for preclinical and clinical studies reported ML models for PMI estimation.

RESULTS

We identified 18 studies (12 preclinical and 06 clinical) that met the inclusion criteria in the qualitative analysis. Most of the studies employed supervised learning (N = 15), and others employed unsupervised learning (N = 3). Due to the heterogeneity of the samples, quantitative analysis was not performed.

CONCLUSION

In this systematic review, we discussed the performance of AI-based automated systems in PMI estimation. ML models have demonstrated accuracy and precision and the ability to overcome human errors and bias. However, the research is limited, conducted in primarily small, selected human populations. In addition, we suggest further research in larger population-based studies is needed to fully understand the extent of integrated ML models.

Explainability does not improve biochemistry staff trust in artificial intelligence-based decision support

Background: Explainability, the aspect of artificial intelligence-based decision support (ADS) systems that allows users to understand why predictions are made, offers many potential benefits. One common claim is that explainability increases user trust, yet this has not been established in healthcare contexts. For advanced algorithms such as artificial neural networks, the generation of explanations is not trivial, but requires the use of a second algorithm. The assumption of improved user trust should therefore be investigated to determine if it justifies the additional complexity. Methods: Biochemistry staff completed a wrong blood in tube (WBIT) error identification task with the help of an ADS system. One-half of the volunteers were provided with both ADS predictions and explanations for those predictions, while the other half received predictions alone. The two groups were compared in terms of their rate of agreement with ADS predictions, as an index of user trust, and WBIT error detection performance. Since the AI model used to generate predictions was known to out-perform laboratory staff, increased trust was expected to improve user performance. Results: Volunteers reviewed 1,590 sets of results. The volunteers provided with explanations demonstrated no difference in their rate of agreement with the ADS system compared to volunteers receiving predictions alone (83.3% versus 81.8%, p=0.46). The two volunteer groups were also equivalent in accuracy, sensitivity and specificity for WBIT error identification (p-values >0.78). Conclusions: For a WBIT error identification task, there was no evidence to justify the additional complexity of explainability on the grounds of increased user trust

Applications of machine learning in routine laboratory medicine: Current state and future directions

Machine learning is able to leverage large amounts of data to infer complex patterns that are otherwise beyond the capabilities of rule-based systems and human experts. Its application to laboratory medicine is particularly exciting, as laboratory testing provides much of the foundation for clinical decision making. In this article, we provide a brief introduction to machine learning for the medical professional in addition to a comprehensive literature review outlining the current state of machine learning as it has been applied to routine laboratory medicine. Although still in its early stages, machine learning has been used to automate laboratory tasks, optimize utilization, and provide personalized reference ranges and test interpretation. The published literature leads us to believe that machine learning will be an area of increasing importance for the laboratory practitioner. We envision the laboratory of the future will utilize these methods to make significant improvements in efficiency and diagnostic precision.

Machine learning models outperform manual result review for the identification of wrong blood in tube errors in complete blood count results

INTRODUCTION

Wrong blood in tube (WBIT) errors are a significant patient-safety issue encountered by clinical laboratories. This study assessed the performance of machine learning models for the identification of WBIT errors affecting complete blood count (CBC) results against the benchmark of manual review of results by laboratory staff.

METHODS

De-identified current and previous (within seven days) CBC results were used in the computer simulation of WBIT errors. 101 015 sets of samples were used to develop machine learning models using artificial neural network, extreme gradient boosting, support vector machine, random forest, logistic regression, decision trees (one complex and one simple) and k-nearest neighbours algorithms. The performance of these models, and of manual review by laboratory staff, was assessed on a separate data set of 1940 samples.

RESULTS

Volunteers manually reviewing results identified WBIT errors with an accuracy of 85.7%, sensitivity of 80.1% and specificity of 92.1%. All machine learning models exceeded human-level performance (p-values for all metrics were <.001). The artificial neural network model was the most accurate (99.1%), and the simple decision tree was the least accurate (96.8%). Sensitivity for the machine learning models varied from 95.7% to 99.3%, and specificity varied from 96.3% to 98.9%.

CONCLUSION

This study provides preliminary evidence supporting the value of machine learning for detecting WBIT errors affecting CBC results. Although further work addressing practical issues is required, substantial patient-safety benefits await the successful deployment of machine learning models for WBIT error detection.

Decision support or autonomous artificial intelligence? The case of wrong blood in tube errors

OBJECTIVES

Artificial intelligence (AI) models are increasingly being developed for clinical chemistry applications, however, it is not understood whether human interaction with the models, which may occur once they are implemented, improves or worsens their performance. This study examined the effect of human supervision on an artificial neural network trained to identify wrong blood in tube (WBIT) errors.

METHODS

De-identified patient data for current and previous (within seven days) electrolytes, urea and creatinine (EUC) results were used in the computer simulation of WBIT errors at a rate of 50%. Laboratory staff volunteers reviewed the AI model's predictions, and the EUC results on which they were based, before making a final decision regarding the presence or absence of a WBIT error. The performance of this approach was compared to the performance of the AI model operating without human supervision.

RESULTS

Laboratory staff supervised the classification of 510 sets of EUC results. This workflow identified WBIT errors with an accuracy of 81.2%, sensitivity of 73.7% and specificity of 88.6%. However, the AI model classifying these samples autonomously was superior on all metrics (p-values<0.05), including accuracy (92.5%), sensitivity (90.6%) and specificity (94.5%).

CONCLUSIONS

Human interaction with AI models can significantly alter their performance. For computationally complex tasks such as WBIT error identification, best performance may be achieved by autonomously functioning AI models.