1
|
Hanna MG, Olson NH, Zarella M, Dash RC, Herrmann MD, Furtado LV, Stram MN, Raciti PM, Hassell L, Mays A, Pantanowitz L, Sirintrapun JS, Krishnamurthy S, Parwani A, Lujan G, Evans A, Glassy EF, Bui MM, Singh R, Souers RJ, de Baca ME, Seheult JN. Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists. Arch Pathol Lab Med 2024; 148:e335-e361. [PMID: 38041522 DOI: 10.5858/arpa.2023-0042-cp] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/11/2023] [Indexed: 12/03/2023]
Abstract
CONTEXT.— Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. OBJECTIVE.— To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. DATA SOURCES.— An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. CONCLUSIONS.— Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.
Collapse
Affiliation(s)
- Matthew G Hanna
- From the Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York (Hanna, Sirintrapun)
| | - Niels H Olson
- The Defense Innovation Unit, Mountain View, California (Olson)
- The Department of Pathology, Uniformed Services University, Bethesda, Maryland (Olson)
| | - Mark Zarella
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, Minnesota (Zarella, Seheult)
| | - Rajesh C Dash
- Department of Pathology, Duke University Health System, Durham, North Carolina (Dash)
| | - Markus D Herrmann
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston (Herrmann)
| | - Larissa V Furtado
- Department of Pathology, St. Jude Children's Research Hospital, Memphis, Tennessee (Furtado)
| | - Michelle N Stram
- The Department of Forensic Medicine, New York University, and Office of Chief Medical Examiner, New York (Stram)
| | | | - Lewis Hassell
- Department of Pathology, Oklahoma University Health Sciences Center, Oklahoma City (Hassell)
| | - Alex Mays
- The MITRE Corporation, McLean, Virginia (Mays)
| | - Liron Pantanowitz
- Department of Pathology & Clinical Labs, University of Michigan, Ann Arbor (Pantanowitz)
| | - Joseph S Sirintrapun
- From the Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York (Hanna, Sirintrapun)
| | | | - Anil Parwani
- Department of Pathology, The Ohio State University Wexner Medical Center, Columbus (Parwani, Lujan)
| | - Giovanni Lujan
- Department of Pathology, The Ohio State University Wexner Medical Center, Columbus (Parwani, Lujan)
| | - Andrew Evans
- Laboratory Medicine, Mackenzie Health, Toronto, Ontario, Canada (Evans)
| | - Eric F Glassy
- Affiliated Pathologists Medical Group, Rancho Dominguez, California (Glassy)
| | - Marilyn M Bui
- Departments of Pathology and Machine Learning, Moffitt Cancer Center, Tampa, Florida (Bui)
| | - Rajendra Singh
- Department of Dermatopathology, Summit Health, Summit Woodland Park, New Jersey (Singh)
| | - Rhona J Souers
- Department of Biostatistics, College of American Pathologists, Northfield, Illinois (Souers)
| | | | - Jansen N Seheult
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, Minnesota (Zarella, Seheult)
| |
Collapse
|
2
|
Çubukçu HC, Topcu Dİ, Yenice S. Machine learning-based clinical decision support using laboratory data. Clin Chem Lab Med 2024; 62:793-823. [PMID: 38015744 DOI: 10.1515/cclm-2023-1037] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2023] [Accepted: 11/17/2023] [Indexed: 11/30/2023]
Abstract
Artificial intelligence (AI) and machine learning (ML) are becoming vital in laboratory medicine and the broader context of healthcare. In this review article, we summarized the development of ML models and how they contribute to clinical laboratory workflow and improve patient outcomes. The process of ML model development involves data collection, data cleansing, feature engineering, model development, and optimization. These models, once finalized, are subjected to thorough performance assessments and validations. Recently, due to the complexity inherent in model development, automated ML tools were also introduced to streamline the process, enabling non-experts to create models. Clinical Decision Support Systems (CDSS) use ML techniques on large datasets to aid healthcare professionals in test result interpretation. They are revolutionizing laboratory medicine, enabling labs to work more efficiently with less human supervision across pre-analytical, analytical, and post-analytical phases. Despite contributions of the ML tools at all analytical phases, their integration presents challenges like potential model uncertainties, black-box algorithms, and deskilling of professionals. Additionally, acquiring diverse datasets is hard, and models' complexity can limit clinical use. In conclusion, ML-based CDSS in healthcare can greatly enhance clinical decision-making. However, successful adoption demands collaboration among professionals and stakeholders, utilizing hybrid intelligence, external validation, and performance assessments.
Collapse
Affiliation(s)
- Hikmet Can Çubukçu
- General Directorate of Health Services, Rare Diseases Department, Turkish Ministry of Health, Ankara, Türkiye
- Hacettepe University Institute of Informatics, Ankara, Türkiye
| | - Deniz İlhan Topcu
- Health Sciences University İzmir Tepecik Education and Research Hospital, Medical Biochemistry, İzmir, Türkiye
| | - Sedef Yenice
- Florence Nightingale Hospital, Istanbul, Türkiye
| |
Collapse
|
3
|
Bunch DR, Durant TJ, Rudolf JW. Artificial Intelligence Applications in Clinical Chemistry. Clin Lab Med 2023; 43:47-69. [PMID: 36764808 DOI: 10.1016/j.cll.2022.09.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
Artificial intelligence (AI) applications are an area of active investigation in clinical chemistry. Numerous publications have demonstrated the promise of AI across all phases of testing including preanalytic, analytic, and postanalytic phases; this includes novel methods for detecting common specimen collection errors, predicting laboratory results and diagnoses, and enhancing autoverification workflows. Although AI applications pose several ethical and operational challenges, these technologies are expected to transform the practice of the clinical chemistry laboratory in the near future.
Collapse
Affiliation(s)
- Dustin R Bunch
- Department of Pathology and Laboratory Medicine, Nationwide Children's Hospital, 700 Children's Drive, C1923, Columbus, OH 43205-2644, USA; Department of Pathology, College of Medicine, The Ohio State University, Columbus, OH 43210, USA
| | - Thomas Js Durant
- Department of Laboratory Medicine, Yale School of Medicine, 55 Park Street, Room PS 502A, New Haven, CT 06510, USA
| | - Joseph W Rudolf
- Department of Pathology, University of Utah School of Medicine, Salt Lake City, UT 84112, USA; ARUP Laboratories, 500 Chipeta Way, MC 115, Salt Lake City, UT 84108, USA.
| |
Collapse
|
4
|
Lee ES, Durant TJ. Supervised machine learning in the mass spectrometry laboratory: A tutorial. J Mass Spectrom Adv Clin Lab 2022; 23:1-6. [PMID: 34984411 PMCID: PMC8692990 DOI: 10.1016/j.jmsacl.2021.12.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Revised: 12/02/2021] [Accepted: 12/06/2021] [Indexed: 11/19/2022] Open
Abstract
As the demand for laboratory testing by mass spectrometry increases, so does the need for automated methods for data analysis. Clinical mass spectrometry (MS) data is particularly well-suited for machine learning (ML) methods, which deal nicely with structured and discrete data elements. The alignment of these two fields offers a promising synergy that can be used to optimize workflows, improve result quality, and enhance our understanding of high-dimensional datasets and their inherent relationship with disease. In recent years, there has been an increasing number of publications that examine the capabilities of ML-based software in the context of chromatography and MS. However, given the historically distant nature between the fields of clinical chemistry and computer science, there is an opportunity to improve technological literacy of ML-based software within the clinical laboratory scientist community. To this end, we present a basic overview of ML and a tutorial of an ML-based experiment using a previously published MS dataset. The purpose of this paper is to describe the fundamental principles of supervised ML, outline the steps that are classically involved in an ML-based experiment, and discuss the purpose of good ML practice in the context of a binary MS classification problem.
Collapse
Key Words
- Amino acid
- Artificial intelligence
- CART, Classification and Regression Trees
- ML, Machine Learning
- MS, Mass Spectrometry
- Mass spectrometry
- NLL, Negative Log Loss
- PAA, Plasma Amino Acid
- PR, Precision-Recall
- PRAUC, Area Under the Precision-Recall Curve
- RL, Reinforcement Learning
- ROC, Receiver Operator Curve
- SCF, Supplemental Code File
- Supervised machine learning
- XGBT, Extreme Gradient Boosted Trees
- Xgboost
Collapse
Affiliation(s)
- Edward S. Lee
- Department of Laboratory Medicine, at Yale School of Medicine, New Haven, CT, USA
- Department of Laboratory Medicine, at Yale New Haven Hospital, New Haven, CT, USA
| | - Thomas J.S. Durant
- Department of Laboratory Medicine, at Yale School of Medicine, New Haven, CT, USA
- Department of Laboratory Medicine, at Yale New Haven Hospital, New Haven, CT, USA
- Corresponding author at: Department of Laboratory Medicine, 55 Park Street PS345D, New Haven, CT 06511, USA.
| |
Collapse
|
5
|
Herman DS, Rhoads DD, Schulz WL, Durant TJS. Artificial Intelligence and Mapping a New Direction in Laboratory Medicine: A Review. Clin Chem 2021; 67:1466-1482. [PMID: 34557917 DOI: 10.1093/clinchem/hvab165] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2021] [Accepted: 07/26/2021] [Indexed: 12/21/2022]
Abstract
BACKGROUND Modern artificial intelligence (AI) and machine learning (ML) methods are now capable of completing tasks with performance characteristics that are comparable to those of expert human operators. As a result, many areas throughout healthcare are incorporating these technologies, including in vitro diagnostics and, more broadly, laboratory medicine. However, there are limited literature reviews of the landscape, likely future, and challenges of the application of AI/ML in laboratory medicine. CONTENT In this review, we begin with a brief introduction to AI and its subfield of ML. The ensuing sections describe ML systems that are currently in clinical laboratory practice or are being proposed for such use in recent literature, ML systems that use laboratory data outside the clinical laboratory, challenges to the adoption of ML, and future opportunities for ML in laboratory medicine. SUMMARY AI and ML have and will continue to influence the practice and scope of laboratory medicine dramatically. This has been made possible by advancements in modern computing and the widespread digitization of health information. These technologies are being rapidly developed and described, but in comparison, their implementation thus far has been modest. To spur the implementation of reliable and sophisticated ML-based technologies, we need to establish best practices further and improve our information system and communication infrastructure. The participation of the clinical laboratory community is essential to ensure that laboratory data are sufficiently available and incorporated conscientiously into robust, safe, and clinically effective ML-supported clinical diagnostics.
Collapse
Affiliation(s)
- Daniel S Herman
- Department of Pathology and Laboratory Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Daniel D Rhoads
- Department of Laboratory Medicine, Cleveland Clinic, Cleveland, OH, USA.,Department of Pathology, Cleveland Clinic Lerner College of Medicine, Case Western Reserve University, Cleveland, OH, USA
| | - Wade L Schulz
- Department of Laboratory Medicine, Yale University, New Haven, CT, USA
| | - Thomas J S Durant
- Department of Laboratory Medicine, Yale University, New Haven, CT, USA
| |
Collapse
|
6
|
Haymond S, McCudden C. Rise of the Machines: Artificial Intelligence and the Clinical Laboratory. J Appl Lab Med 2021; 6:1640-1654. [PMID: 34379752 DOI: 10.1093/jalm/jfab075] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Accepted: 06/08/2021] [Indexed: 11/14/2022]
Abstract
BACKGROUND Artificial intelligence (AI) is rapidly being developed and implemented to augment and automate decision-making across healthcare systems. Being an essential part of these systems, laboratories will see significant growth in AI applications for the foreseeable future. CONTENT In laboratory medicine, AI can be used for operational decision-making and automating or augmenting human-based workflows. Specific applications include instrument automation, error detection, forecasting, result interpretation, test utilization, genomics, and image analysis. If not doing so today, clinical laboratories will be using AI routinely in the future, therefore, laboratory experts should understand their potential role in this new area and the opportunities for AI technologies. The roles of laboratorians range from passive provision of data to fuel algorithms to developing entirely new algorithms, with subject matter expertise as a perfect fit in the middle. The technical development of algorithms is only a part of the overall picture, where the type, availability, and quality of data are at least as important. Implementation of AI algorithms also offers technical and usability challenges that need to be understood to be successful. Finally, as AI algorithms continue to become available, it is important to understand how to evaluate their validity and utility in the real world. SUMMARY This review provides an overview of what AI is, examples of how it is currently being used in laboratory medicine, different ways for laboratorians to get involved in algorithm development, and key considerations for AI algorithm implementation and critical evaluation.
Collapse
Affiliation(s)
- Shannon Haymond
- Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL.,Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL
| | - Christopher McCudden
- Department of Pathology & Laboratory Medicine, University of Ottawa, Canada, The Ottawa Hospital, and the Eastern Ontario Regional Laboratory Association, Canada
| |
Collapse
|
7
|
Harrison JH, Gilbertson JR, Hanna MG, Olson NH, Seheult JN, Sorace JM, Stram MN. Introduction to Artificial Intelligence and Machine Learning for Pathology. Arch Pathol Lab Med 2021; 145:1228-1254. [PMID: 33493264 DOI: 10.5858/arpa.2020-0541-cp] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/10/2020] [Indexed: 11/06/2022]
Abstract
CONTEXT.— Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools. OBJECTIVE.— To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation. DATA SOURCES.— Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development. CONCLUSIONS.— Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.
Collapse
Affiliation(s)
- James H Harrison
- From the Department of Pathology, University of Virginia School of Medicine, Charlottesville (Harrison)
| | - John R Gilbertson
- the Departments of Biomedical Informatics and Pathology, University of Pittsburgh, Pittsburgh, Pennsylvania (Gilbertson)
| | - Matthew G Hanna
- the Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, New York (Hanna)
| | - Niels H Olson
- the Defense Innovation Unit, Mountain View, California (Olson).,the Department of Pathology, Uniformed Services University, Bethesda, Maryland (Olson)
| | - Jansen N Seheult
- the Department of Pathology, University of Pittsburgh, and Vitalant Specialty Labs, Pittsburgh, Pennsylvania (Seheult)
| | - James M Sorace
- the US Department of Health and Human Services, retired, Lutherville, Maryland (Sorace)
| | - Michelle N Stram
- the Department of Forensic Medicine, New York University, and Office of Chief Medical Examiner, New York, New York (Stram)
| |
Collapse
|
8
|
De Bruyne S, Speeckaert MM, Van Biesen W, Delanghe JR. Recent evolutions of machine learning applications in clinical laboratory medicine. Crit Rev Clin Lab Sci 2020; 58:131-152. [PMID: 33045173 DOI: 10.1080/10408363.2020.1828811] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
Machine learning (ML) is gaining increased interest in clinical laboratory medicine, mainly triggered by the decreased cost of generating and storing data using laboratory automation and computational power, and the widespread accessibility of open source tools. Nevertheless, only a handful of ML-based products are currently commercially available for routine clinical laboratory practice. In this review, we start with an introduction to ML by providing an overview of the ML landscape, its general workflow, and the most commonly used algorithms for clinical laboratory applications. Furthermore, we aim to illustrate recent evolutions (2018 to mid-2020) of the techniques used in the clinical laboratory setting and discuss the associated challenges and opportunities. In the field of clinical chemistry, the reviewed applications of ML algorithms include quality review of lab results, automated urine sediment analysis, disease or outcome prediction from routine laboratory parameters, and interpretation of complex biochemical data. In the hematology subdiscipline, we discuss the concepts of automated blood film reporting and malaria diagnosis. At last, we handle a broad range of clinical microbiology applications, such as the reduction of diagnostic workload by laboratory automation, the detection and identification of clinically relevant microorganisms, and the detection of antimicrobial resistance.
Collapse
Affiliation(s)
- Sander De Bruyne
- Department of Diagnostic Sciences, Ghent University, Ghent, Belgium
| | | | - Wim Van Biesen
- Department of Nephrology, Ghent University Hospital, Ghent, Belgium
| | - Joris R Delanghe
- Department of Diagnostic Sciences, Ghent University, Ghent, Belgium
| |
Collapse
|