1
|
Alrawashdeh A, Alqahtani S, Alkhatib ZI, Kheirallah K, Melhem NY, Alwidyan M, Al-Dekah AM, Alshammari T, Nehme Z. Applications and Performance of Machine Learning Algorithms in Emergency Medical Services: A Scoping Review. Prehosp Disaster Med 2024:1-11. [PMID: 38757150 DOI: 10.1017/s1049023x24000414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/18/2024]
Abstract
OBJECTIVE The aim of this study was to summarize the literature on the applications of machine learning (ML) and their performance in Emergency Medical Services (EMS). METHODS Four relevant electronic databases were searched (from inception through January 2024) for all original studies that employed EMS-guided ML algorithms to enhance the clinical and operational performance of EMS. Two reviewers screened the retrieved studies and extracted relevant data from the included studies. The characteristics of included studies, employed ML algorithms, and their performance were quantitively described across primary domains and subdomains. RESULTS This review included a total of 164 studies published from 2005 through 2024. Of those, 125 were clinical domain focused and 39 were operational. The characteristics of ML algorithms such as sample size, number and type of input features, and performance varied between and within domains and subdomains of applications. Clinical applications of ML algorithms involved triage or diagnosis classification (n = 62), treatment prediction (n = 12), or clinical outcome prediction (n = 50), mainly for out-of-hospital cardiac arrest/OHCA (n = 62), cardiovascular diseases/CVDs (n = 19), and trauma (n = 24). The performance of these ML algorithms varied, with a median area under the receiver operating characteristic curve (AUC) of 85.6%, accuracy of 88.1%, sensitivity of 86.05%, and specificity of 86.5%. Within the operational studies, the operational task of most ML algorithms was ambulance allocation (n = 21), followed by ambulance detection (n = 5), ambulance deployment (n = 5), route optimization (n = 5), and quality assurance (n = 3). The performance of all operational ML algorithms varied and had a median AUC of 96.1%, accuracy of 90.0%, sensitivity of 94.4%, and specificity of 87.7%. Generally, neural network and ensemble algorithms, to some degree, out-performed other ML algorithms. CONCLUSION Triaging and managing different prehospital medical conditions and augmenting ambulance performance can be improved by ML algorithms. Future reports should focus on a specific clinical condition or operational task to improve the precision of the performance metrics of ML models.
Collapse
Affiliation(s)
- Ahmad Alrawashdeh
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | - Saeed Alqahtani
- Department of Emergency Medical Services, Prince Sultan Military College for Health Sciences, Dhahran, Saudi Arabia
| | - Zaid I Alkhatib
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | - Khalid Kheirallah
- Department of Public Health and Family Medicine, Faculty of Medicine, Jordan University of Science and Technology, Irbid, Jordan
| | - Nebras Y Melhem
- Department of Anatomy, Physiology and Biochemistry, Faculty of Medicine, The Hashemite University, Zarqa, Jordan
| | - Mahmoud Alwidyan
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | | | - Talal Alshammari
- Department of Emergency Medical Care, College of Applied Medical Sciences, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Ziad Nehme
- Ambulance Victoria, Doncaster, Victoria, Australia
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
2
|
Collins GS. Making the black box more transparent: improving the reporting of artificial intelligence studies in healthcare. BMJ 2024; 385:q832. [PMID: 38626954 DOI: 10.1136/bmj.q832] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 04/19/2024]
Affiliation(s)
- Gary S Collins
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| |
Collapse
|
3
|
Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, van Smeden M, Boulesteix AL, Camaradou JC, Celi LA, Denaxas S, Denniston AK, Glocker B, Golub RM, Harvey H, Heinze G, Hoffman MM, Kengne AP, Lam E, Lee N, Loder EW, Maier-Hein L, Mateen BA, McCradden MD, Oakden-Rayner L, Ordish J, Parnell R, Rose S, Singh K, Wynants L, Logullo P. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024; 385:e078378. [PMID: 38626948 PMCID: PMC11019967 DOI: 10.1136/bmj-2023-078378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/17/2024] [Indexed: 04/19/2024]
Affiliation(s)
- Gary S Collins
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Paula Dhiman
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Andrew L Beam
- Department of Epidemiology, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Science, Leiden University Medical Centre, Leiden, Netherlands
| | - Marzyeh Ghassemi
- Department of Electrical Engineering and Computer Science, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Xiaoxuan Liu
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Johannes B Reitsma
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Maarten van Smeden
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Anne-Laure Boulesteix
- Institute for Medical Information Processing, Biometry and Epidemiology, Faculty of Medicine, Ludwig-Maximilians-University of Munich and Munich Centre of Machine Learning, Germany
| | - Jennifer Catherine Camaradou
- Patient representative, Health Data Research UK patient and public involvement and engagement group
- Patient representative, University of East Anglia, Faculty of Health Sciences, Norwich Research Park, Norwich, UK
| | - Leo Anthony Celi
- Beth Israel Deaconess Medical Center, Boston, MA, USA
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
- British Heart Foundation Data Science Centre, London, UK
| | - Alastair K Denniston
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Ben Glocker
- Department of Computing, Imperial College London, London, UK
| | - Robert M Golub
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | | | - Georg Heinze
- Section for Clinical Biometrics, Centre for Medical Data Science, Medical University of Vienna, Vienna, Austria
| | - Michael M Hoffman
- Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada
- Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
- Vector Institute for Artificial Intelligence, Toronto, ON, Canada
| | | | - Emily Lam
- Patient representative, Health Data Research UK patient and public involvement and engagement group
| | - Naomi Lee
- National Institute for Health and Care Excellence, London, UK
| | - Elizabeth W Loder
- The BMJ, London, UK
- Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Lena Maier-Hein
- Department of Intelligent Medical Systems, German Cancer Research Centre, Heidelberg, Germany
| | - Bilal A Mateen
- Institute of Health Informatics, University College London, London, UK
- Wellcome Trust, London, UK
- Alan Turing Institute, London, UK
| | - Melissa D McCradden
- Department of Bioethics, Hospital for Sick Children Toronto, ON, Canada
- Genetics and Genome Biology, SickKids Research Institute, Toronto, ON, Canada
| | - Lauren Oakden-Rayner
- Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia
| | - Johan Ordish
- Medicines and Healthcare products Regulatory Agency, London, UK
| | - Richard Parnell
- Patient representative, Health Data Research UK patient and public involvement and engagement group
| | - Sherri Rose
- Department of Health Policy and Center for Health Policy, Stanford University, Stanford, CA, USA
| | - Karandeep Singh
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
| | - Laure Wynants
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
| | - Patricia Logullo
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| |
Collapse
|
4
|
Hassan A, Critelli B, Lahooti I, Lahooti A, Matzko N, Adams JN, Liss L, Quion J, Restrepo D, Nikahd M, Culp S, Noh L, Tong K, Park JS, Akshintala V, Windsor JA, Mull NK, Papachristou GI, Celi LA, Lee PJ. Critical appraisal of machine learning prognostic models for acute pancreatitis: protocol for a systematic review. Diagn Progn Res 2024; 8:6. [PMID: 38561864 PMCID: PMC10986113 DOI: 10.1186/s41512-024-00169-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 02/15/2024] [Indexed: 04/04/2024] Open
Abstract
Acute pancreatitis (AP) is an acute inflammatory disorder that is common, costly, and is increasing in incidence worldwide with over 300,000 hospitalizations occurring yearly in the United States alone. As its course and outcomes vary widely, a critical knowledge gap in the field has been a lack of accurate prognostic tools to forecast AP patients' outcomes. Despite several published studies in the last three decades, the predictive performance of published prognostic models has been found to be suboptimal. Recently, non-regression machine learning models (ML) have garnered intense interest in medicine for their potential for better predictive performance. Each year, an increasing number of AP models are being published. However, their methodologic quality relating to transparent reporting and risk of bias in study design has never been systematically appraised. Therefore, through collaboration between a group of clinicians and data scientists with appropriate content expertise, we will perform a systematic review of papers published between January 2021 and December 2023 containing artificial intelligence prognostic models in AP. To systematically assess these studies, the authors will leverage the CHARMS checklist, PROBAST tool for risk of bias assessment, and the most current version of the TRIPOD-AI. (Research Registry ( http://www.reviewregistry1727 .).
Collapse
Affiliation(s)
- Amier Hassan
- Division of Gastroenterology and Hepatology, Weill Cornell Medical College, New York, USA
| | - Brian Critelli
- Division of Gastroenterology and Hepatology, Weill Cornell Medical College, New York, USA
| | - Ila Lahooti
- Division of Gastroenterology and Hepatology, Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Ali Lahooti
- Division of Gastroenterology and Hepatology, Weill Cornell Medical College, New York, USA
| | - Nate Matzko
- Division of Gastroenterology and Hepatology, Weill Cornell Medical College, New York, USA
| | - Jan Niklas Adams
- Division of Process and Data Science, Rheinisch-Westfälische Technische Hochschule Aachen University, Aachen, Germany
| | - Lukas Liss
- Division of Process and Data Science, Rheinisch-Westfälische Technische Hochschule Aachen University, Aachen, Germany
| | - Justin Quion
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, USA
| | - David Restrepo
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, USA
| | - Melica Nikahd
- Division of Bioinformatics, Ohio State University Wexner Medical Center, Columbus, USA
| | - Stacey Culp
- Division of Bioinformatics, Ohio State University Wexner Medical Center, Columbus, USA
| | - Lydia Noh
- Northeast Ohio Medical School, Rootstown, USA
| | - Kathleen Tong
- Division of Gastroenterology and Hepatology, Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Jun Sung Park
- Division of Gastroenterology and Hepatology, Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Venkata Akshintala
- Division of Gastroenterology, Johns Hopkins Medical Center, Baltimore, USA
| | - John A Windsor
- Department of Surgery, University of Auckland, Auckland, New Zealand
| | - Nikhil K Mull
- Division of Hospital Medicine and Penn Medicine Center for Evidence-based Practice, University of Pennsylvania, Philadelphia, USA
| | - Georgios I Papachristou
- Division of Gastroenterology and Hepatology, Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Leo Anthony Celi
- Department of Surgery, University of Auckland, Auckland, New Zealand
- Division of Critical Care, Beth Israel Medical Center, Boston, USA
| | - Peter J Lee
- Division of Gastroenterology and Hepatology, Ohio State University Wexner Medical Center, Columbus, OH, USA.
| |
Collapse
|
5
|
Talimtzi P, Ntolkeras A, Kostopoulos G, Bougioukas KI, Pagkalidou E, Ouranidis A, Pataka A, Haidich AB. The reporting completeness and transparency of systematic reviews of prognostic prediction models for COVID-19 was poor: a methodological overview of systematic reviews. J Clin Epidemiol 2024; 167:111264. [PMID: 38266742 DOI: 10.1016/j.jclinepi.2024.111264] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Revised: 01/08/2024] [Accepted: 01/13/2024] [Indexed: 01/26/2024]
Abstract
OBJECTIVES To conduct a methodological overview of reviews to evaluate the reporting completeness and transparency of systematic reviews (SRs) of prognostic prediction models (PPMs) for COVID-19. STUDY DESIGN AND SETTING MEDLINE, Scopus, Cochrane Database of Systematic Reviews, and Epistemonikos (epistemonikos.org) were searched for SRs of PPMs for COVID-19 until December 31, 2022. The risk of bias in systematic reviews tool was used to assess the risk of bias. The protocol for this overview was uploaded in the Open Science Framework (https://osf.io/7y94c). RESULTS Ten SRs were retrieved; none of them synthesized the results in a meta-analysis. For most of the studies, there was absence of a predefined protocol and missing information on study selection, data collection process, and reporting of primary studies and models included, while only one SR had its data publicly available. In addition, for the majority of the SRs, the overall risk of bias was judged as being high. The overall corrected covered area was 6.3% showing a small amount of overlapping among the SRs. CONCLUSION The reporting completeness and transparency of SRs of PPMs for COVID-19 was poor. Guidance is urgently required, with increased awareness and education of minimum reporting standards and quality criteria. Specific focus is needed in predefined protocol, information on study selection and data collection process, and in the reporting of findings to improve the quality of SRs of PPMs for COVID-19.
Collapse
Affiliation(s)
- Persefoni Talimtzi
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, University Campus, 54124, Thessaloniki, Greece
| | - Antonios Ntolkeras
- School of Biology, Aristotle University of Thessaloniki, University Campus, 54636, Thessaloniki, Greece
| | | | - Konstantinos I Bougioukas
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, University Campus, 54124, Thessaloniki, Greece
| | - Eirini Pagkalidou
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, University Campus, 54124, Thessaloniki, Greece
| | - Andreas Ouranidis
- Department of Pharmaceutical Technology, School of Pharmacy, Aristotle University of Thessaloniki, 54124, Thessaloniki, Greece
| | - Athanasia Pataka
- Department of Respiratory Deficiency, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, University Campus, 54124, Thessaloniki, Greece
| | - Anna-Bettina Haidich
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, University Campus, 54124, Thessaloniki, Greece.
| |
Collapse
|
6
|
Kyriacou C, Ledger A, Bobdiwala S, Ayim F, Kirk E, Abughazza O, Guha S, Vathanan V, Gould D, Timmerman D, Van Calster B, Bourne T. Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors. ULTRASOUND IN OBSTETRICS & GYNECOLOGY : THE OFFICIAL JOURNAL OF THE INTERNATIONAL SOCIETY OF ULTRASOUND IN OBSTETRICS AND GYNECOLOGY 2024; 63:408-418. [PMID: 37842861 DOI: 10.1002/uog.27515] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 09/19/2023] [Accepted: 10/05/2023] [Indexed: 10/17/2023]
Abstract
OBJECTIVES Ectopic pregnancy (EP) is a major high-risk outcome following a pregnancy of unknown location (PUL) classification. Biochemical markers are used to triage PUL as high vs low risk to guide appropriate follow-up. The M6 model is currently the best risk-prediction model. We aimed to update the M6 model and evaluate whether performance can be improved by including clinical factors. METHODS This prospective cohort study recruited consecutive PUL between January 2015 and January 2017 at eight units (Phase 1), with two centers continuing recruitment between January 2017 and March 2021 (Phase 2). Serum samples were collected routinely and sent for β-human chorionic gonadotropin (β-hCG) and progesterone measurement. Clinical factors recorded were maternal age, pain score, bleeding score and history of EP. Based on transvaginal ultrasonography and/or biochemical confirmation during follow-up, PUL were classified subsequently as failed PUL (FPUL), intrauterine pregnancy (IUP) or EP (including persistent PUL (PPUL)). The M6 models with (M6P ) and without (M6NP ) progesterone were refitted and extended with clinical factors. Model validation was performed using internal-external cross-validation (IECV) (Phase 1) and temporal external validation (EV) (Phase 2). Missing values were handled using multiple imputation. RESULTS Overall, 5473 PUL were recruited over both phases. A total of 709 PUL were excluded because maternal age was < 16 years or initial β-hCG was ≤ 25 IU/L, leaving 4764 (87%) PUL for analysis (2894 in Phase 1 and 1870 in Phase 2). For the refitted M6P model, the area under the receiver-operating-characteristics curve (AUC) for EP/PPUL vs IUP/FPUL was 0.89 for IECV and 0.84-0.88 for EV, with respective sensitivities of 94% and 92-93%. For the refitted M6NP model, the AUCs were 0.85 for IECV and 0.82-0.86 for EV, with respective sensitivities of 92% and 93-94%. Calibration performance was good overall, but with heterogeneity between centers. Net Benefit confirmed clinical utility. The change in AUC when M6P was extended to include maternal age, bleeding score and history of EP was between -0.02 and 0.01, depending on center and phase. The corresponding change in AUC when M6NP was extended was between -0.01 and 0.03. At the 5% threshold to define high risk of EP/PPUL, extending M6P altered sensitivity by -0.02 to -0.01, specificity by 0.03 to 0.04 and Net Benefit by -0.005 to 0.006. Extending M6NP altered sensitivity by -0.03 to -0.01, specificity by 0.05 to 0.07 and Net Benefit by -0.005 to 0.006. CONCLUSIONS The updated M6 model offers accurate diagnostic performance, with excellent sensitivity for EP. Adding clinical factors to the model improved performance in some centers, especially when progesterone levels were not suitable or unavailable. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.
Collapse
Affiliation(s)
- C Kyriacou
- Tommy's National Centre for Miscarriage Research, Queen Charlotte's and Chelsea Hospital, Imperial College London, London, UK
| | - A Ledger
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - S Bobdiwala
- Tommy's National Centre for Miscarriage Research, Queen Charlotte's and Chelsea Hospital, Imperial College London, London, UK
| | - F Ayim
- Department of Gynaecology, Hillingdon Hospital NHS Trust, London, UK
| | - E Kirk
- Department of Gynaecology, Royal Free NHS Foundation Trust, London, UK
| | - O Abughazza
- Department of Gynaecology, Royal Surrey County Hospital, Guildford, UK
| | - S Guha
- Department of Gynaecology, Chelsea and Westminster NHS Trust, London, UK
| | - V Vathanan
- Department of Gynaecology, Wexham Park Hospital, London, UK
| | - D Gould
- Department of Gynaecology, St Mary's Hospital, London, UK
| | - D Timmerman
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Gynecology, University Hospital Leuven, Leuven, Belgium
| | - B Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - T Bourne
- Tommy's National Centre for Miscarriage Research, Queen Charlotte's and Chelsea Hospital, Imperial College London, London, UK
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Gynecology, University Hospital Leuven, Leuven, Belgium
| |
Collapse
|
7
|
Giddings R, Joseph A, Callender T, Janes SM, van der Schaar M, Sheringham J, Navani N. Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review. Lancet Digit Health 2024; 6:e131-e144. [PMID: 38278615 DOI: 10.1016/s2589-7500(23)00241-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 10/20/2023] [Accepted: 11/14/2023] [Indexed: 01/28/2024]
Abstract
Machine learning (ML)-based risk prediction models hold the potential to support the health-care setting in several ways; however, use of such models is scarce. We aimed to review health-care professional (HCP) and patient perceptions of ML risk prediction models in published literature, to inform future risk prediction model development. Following database and citation searches, we identified 41 articles suitable for inclusion. Article quality varied with qualitative studies performing strongest. Overall, perceptions of ML risk prediction models were positive. HCPs and patients considered that models have the potential to add benefit in the health-care setting. However, reservations remain; for example, concerns regarding data quality for model development and fears of unintended consequences following ML model use. We identified that public views regarding these models might be more negative than HCPs and that concerns (eg, extra demands on workload) were not always borne out in practice. Conclusions are tempered by the low number of patient and public studies, the absence of participant ethnic diversity, and variation in article quality. We identified gaps in knowledge (particularly views from under-represented groups) and optimum methods for model explanation and alerts, which require future research.
Collapse
Affiliation(s)
- Rebecca Giddings
- Lungs for Living Research Centre, UCL Respiratory, University College London, London, UK.
| | - Anabel Joseph
- Lungs for Living Research Centre, UCL Respiratory, University College London, London, UK
| | - Thomas Callender
- Lungs for Living Research Centre, UCL Respiratory, University College London, London, UK
| | - Sam M Janes
- Lungs for Living Research Centre, UCL Respiratory, University College London, London, UK
| | - Mihaela van der Schaar
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK; The Alan Turing Institute, London, UK
| | - Jessica Sheringham
- Department of Applied Health Research, University College London, London, UK
| | - Neal Navani
- Lungs for Living Research Centre, UCL Respiratory, University College London, London, UK
| |
Collapse
|
8
|
Collins GS, Whittle R, Bullock GS, Logullo P, Dhiman P, de Beyer JA, Riley RD, Schlussel MM. Open science practices need substantial improvement in prognostic model studies in oncology using machine learning. J Clin Epidemiol 2024; 165:111199. [PMID: 37898461 DOI: 10.1016/j.jclinepi.2023.10.015] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 10/06/2023] [Accepted: 10/20/2023] [Indexed: 10/30/2023]
Abstract
OBJECTIVE To describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology. STUDY DESIGN AND SETTING We conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices. RESULTS We identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing statements, with 21 (46%) indicating data were available on request to the authors and seven declaring data sharing was not applicable. Two studies (4%) shared data. Only 12 studies (26%) provided code sharing statements, including 2 (4%) that indicated the code was available on request to the authors. Only 11 studies (24%) provided sufficient information to allow their model to be used in practice. The use of reporting guidelines was rare: eight studies (18%) mentioning using a reporting guideline, with 4 (10%) using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement, 1 (2%) using Minimum Information About Clinical Artificial Intelligence Modeling and Consolidated Standards Of Reporting Trials-Artificial Intelligence, 1 (2%) using Strengthening The Reporting Of Observational Studies In Epidemiology, 1 (2%) using Standards for Reporting Diagnostic Accuracy Studies, and 1 (2%) using Transparent Reporting of Evaluations with Nonrandomized Designs. CONCLUSION The adoption of open science principles in oncology studies developing prognostic models using machine learning methods is poor. Guidance and an increased awareness of benefits and best practices of open science are needed for prediction research in oncology.
Collapse
Affiliation(s)
- Gary S Collins
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom.
| | - Rebecca Whittle
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Garrett S Bullock
- Department of Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC, USA; Centre for Sport, Exercise and Osteoarthritis Research Versus Arthritis, University of Oxford, Oxford, United Kingdom
| | - Patricia Logullo
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Paula Dhiman
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Jennifer A de Beyer
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom
| | - Michael M Schlussel
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
9
|
Vasey B, Collins GS. Invited Commentary: Transparent reporting of artificial intelligence models development and evaluation in surgery: The TRIPOD and DECIDE-AI checklists. Surgery 2023; 174:727-729. [PMID: 37244769 DOI: 10.1016/j.surg.2023.04.037] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 04/27/2023] [Indexed: 05/29/2023]
Affiliation(s)
- Baptiste Vasey
- Nuffield Department of Surgical Sciences, University of Oxford, UK; Department of Surgery, Geneva University Hospital, Switzerland.
| | - Gary S Collins
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, UK. http://www.twitter.com/GSCollins
| |
Collapse
|
10
|
Hofmann AG, Shoumariyeh T, Domenig C, Skrabal F, Kovarik JJ. Abdominal Aortic Aneurysm Detection in Bioelectrical Impedance Cardiovascular Screenings-A Pilot Study. J Clin Med 2023; 12:jcm12113726. [PMID: 37297921 DOI: 10.3390/jcm12113726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 05/20/2023] [Accepted: 05/23/2023] [Indexed: 06/12/2023] Open
Abstract
Screening and diagnosing abdominal aortic aneurysms (AAA) are currently dependent on imaging studies such as ultrasound or computed tomography angiography. All imaging studies offer distinct advantages but also suffer from inherent limitations such as examiner dependency or ionizing radiation. Bioelectrical impedance analysis has previously been investigated with respect to its use in the detection of several cardiovascular and renal pathologies. The present pilot study assessed the feasibility of AAA detection based on bioimpedance analysis. In this single-center exploratory pilot study, measurements were conducted among three different cohorts: patients with AAA, end-stage renal disease patients without AAA, and healthy controls. The device used in the study, CombynECG, is an open-market accessible device for segmental bioelectrical impedance analysis. The data was preprocessed and used to train four different machine learning models on a randomized training sample (80% of the full dataset). Each model was then evaluated on a test set (20% of the full dataset). The total sample included 22 patients with AAA, 16 chronic kidney disease patients, and 23 healthy controls. All four models showed strong predictive performance in the test partitions. Specificity ranged from 71.4 to 100%, while sensitivity ranged from 66.7 to 100%. The best-performing model had 100% accuracy for classification when applied to the test sample. Additionally, an exploratory analysis to approximate the maximum AAA diameter was conducted. An association analysis revealed several impedance parameters that might possess predictive ability with respect to aneurysm size. AAA detection via bioelectrical impedance analysis is technically feasible and appears to be a promising technology for large-scale clinical studies and routine clinical screening assessments.
Collapse
Affiliation(s)
- Amun G Hofmann
- Department of Internal Medicine III, Division of Nephrology and Dialysis, Medical University of Vienna, 1090 Vienna, Austria
| | - Tarik Shoumariyeh
- Department of Internal Medicine III, Division of Nephrology and Dialysis, Medical University of Vienna, 1090 Vienna, Austria
| | - Christoph Domenig
- Department of General Surgery, Division of Vascular Surgery, Medical University of Vienna, 1090 Vienna, Austria
| | - Falko Skrabal
- Institute of Cardiovascular & Metabolic Medicine, 8010 Graz, Austria
| | - Johannes J Kovarik
- Department of Internal Medicine III, Division of Nephrology and Dialysis, Medical University of Vienna, 1090 Vienna, Austria
| |
Collapse
|
11
|
Interpretable prognostic modeling of endometrial cancer. Sci Rep 2022; 12:21543. [PMID: 36513790 PMCID: PMC9747711 DOI: 10.1038/s41598-022-26134-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Accepted: 12/09/2022] [Indexed: 12/15/2022] Open
Abstract
Endometrial carcinoma (EC) is one of the most common gynecological cancers in the world. In this work we apply Cox proportional hazards (CPH) and optimal survival tree (OST) algorithms to the retrospective prognostic modeling of disease-specific survival in 842 EC patients. We demonstrate that linear CPH models are preferred for the EC risk assessment based on clinical features alone, while interpretable, non-linear OST models are favored when patient profiles can be supplemented with additional biomarker data. We show how visually interpretable tree models can help generate and explore novel research hypotheses by studying the OST decision path structure, in which L1 cell adhesion molecule expression and estrogen receptor status are correctly indicated as important risk factors in the p53 abnormal EC subgroup. To aid further clinical adoption of advanced machine learning techniques, we stress the importance of quantifying model discrimination and calibration performance in the development of explainable clinical prediction models.
Collapse
|