1
|
Fares MY, Liu HH, da Silva Etges APB, Zhang B, Warner JJP, Olson JJ, Fedorka CJ, Khan AZ, Best MJ, Kirsch JM, Simon JE, Sanders B, Costouros JG, Zhang X, Jones P, Haas DA, Abboud JA. Utility of Machine Learning, Natural Language Processing, and Artificial Intelligence in Predicting Hospital Readmissions After Orthopaedic Surgery: A Systematic Review and Meta-Analysis. JBJS Rev 2024; 12:01874474-202408000-00011. [PMID: 39172864 DOI: 10.2106/jbjs.rvw.24.00075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/24/2024]
Abstract
BACKGROUND Numerous applications and strategies have been utilized to help assess the trends and patterns of readmissions after orthopaedic surgery in an attempt to extrapolate possible risk factors and causative agents. The aim of this work is to systematically summarize the available literature on the extent to which natural language processing, machine learning, and artificial intelligence (AI) can help improve the predictability of hospital readmissions after orthopaedic and spine surgeries. METHODS This is a systematic review and meta-analysis. PubMed, Embase and Google Scholar were searched, up until August 30, 2023, for studies that explore the use of AI, natural language processing, and machine learning tools for the prediction of readmission rates after orthopedic procedures. Data regarding surgery type, patient population, readmission outcomes, advanced models utilized, comparison methods, predictor sets, the inclusion of perioperative predictors, validation method, size of training and testing sample, accuracy, and receiver operating characteristics (C-statistic), among other factors, were extracted and assessed. RESULTS A total of 26 studies were included in our final dataset. The overall summary C-statistic showed a mean of 0.71 across all models, indicating a reasonable level of predictiveness. A total of 15 articles (57%) were attributed to the spine, making it the most commonly explored orthopaedic field in our study. When comparing accuracy of prediction models between different fields, models predicting readmissions after hip/knee arthroplasty procedures had a higher prediction accuracy (mean C-statistic = 0.79) than spine (mean C-statistic = 0.7) and shoulder (mean C-statistic = 0.67). In addition, models that used single institution data, and those that included intraoperative and/or postoperative outcomes, had a higher mean C-statistic than those utilizing other data sources, and that include only preoperative predictors. According to the Prediction model Risk of Bias Assessment Tool, the majority of the articles in our study had a high risk of bias. CONCLUSION AI tools perform reasonably well in predicting readmissions after orthopaedic procedures. Future work should focus on standardizing study methodologies and designs, and improving the data analysis process, in an attempt to produce more reliable and tangible results. LEVEL OF EVIDENCE Level III. See Instructions for Authors for a complete description of levels of evidence.
Collapse
Affiliation(s)
- Mohamad Y Fares
- Rothman Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania
| | | | | | | | - Jon J P Warner
- Department of Orthopaedic Surgery, Harvard Medical School, Boston Shoulder Institute, Massachusetts General Hospital, Boston, Massachusetts
| | | | - Catherine J Fedorka
- Cooper Bone and Joint Institute, Cooper University Hospital, Camden, New Jersey
| | - Adam Z Khan
- Department of Orthopaedic Surgery, Southern California Permanente Medical Group, Panorama City, California
| | - Matthew J Best
- Department of Orthopaedic Surgery, Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Jacob M Kirsch
- Department of Orthopaedic Surgery, New England Baptist Hospital, Tufts University School of Medicine, Boston, Massachusetts
| | - Jason E Simon
- Department of Orthopaedic Surgery, Massachusetts General Hospital/Newton-Wellesley Hospital, Boston, Massachusetts
| | - Brett Sanders
- Center for Sports Medicine and Orthopaedics, Chattanooga, Tennessee
| | - John G Costouros
- Institute for Joint Restoration and Research, California Shoulder Center, Menlo Park, California
| | | | | | | | - Joseph A Abboud
- Rothman Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania
| |
Collapse
|
2
|
Ghanem M, Ghaith AK, El-Hajj VG, Bhandarkar A, de Giorgio A, Elmi-Terander A, Bydon M. Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review. Brain Sci 2023; 13:1723. [PMID: 38137171 PMCID: PMC10741524 DOI: 10.3390/brainsci13121723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Revised: 12/12/2023] [Accepted: 12/15/2023] [Indexed: 12/24/2023] Open
Abstract
Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models' effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model's area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.
Collapse
Affiliation(s)
- Marc Ghanem
- Mayo Clinic Neuro-Informatics Laboratory, Mayo Clinic, Rochester, MN 55902, USA; (M.G.); (A.K.G.); (V.G.E.-H.); (A.B.); (M.B.)
- Department of Neurological Surgery, Mayo Clinic, Rochester, MN 55902, USA
- School of Medicine, Lebanese American University, Byblos 4504, Lebanon
| | - Abdul Karim Ghaith
- Mayo Clinic Neuro-Informatics Laboratory, Mayo Clinic, Rochester, MN 55902, USA; (M.G.); (A.K.G.); (V.G.E.-H.); (A.B.); (M.B.)
- Department of Neurological Surgery, Mayo Clinic, Rochester, MN 55902, USA
| | - Victor Gabriel El-Hajj
- Mayo Clinic Neuro-Informatics Laboratory, Mayo Clinic, Rochester, MN 55902, USA; (M.G.); (A.K.G.); (V.G.E.-H.); (A.B.); (M.B.)
- Department of Neurological Surgery, Mayo Clinic, Rochester, MN 55902, USA
- Department of Clinical Neuroscience, Karolinska Institutet, 17177 Stockholm, Sweden
| | - Archis Bhandarkar
- Mayo Clinic Neuro-Informatics Laboratory, Mayo Clinic, Rochester, MN 55902, USA; (M.G.); (A.K.G.); (V.G.E.-H.); (A.B.); (M.B.)
- Department of Neurological Surgery, Mayo Clinic, Rochester, MN 55902, USA
| | - Andrea de Giorgio
- Artificial Engineering, Via del Rione Sirignano, 80121 Naples, Italy;
| | - Adrian Elmi-Terander
- Department of Clinical Neuroscience, Karolinska Institutet, 17177 Stockholm, Sweden
- Department of Surgical Sciences, Uppsala University, 75236 Uppsala, Sweden
| | - Mohamad Bydon
- Mayo Clinic Neuro-Informatics Laboratory, Mayo Clinic, Rochester, MN 55902, USA; (M.G.); (A.K.G.); (V.G.E.-H.); (A.B.); (M.B.)
- Department of Neurological Surgery, Mayo Clinic, Rochester, MN 55902, USA
| |
Collapse
|
3
|
Saravi B, Hassel F, Ülkümen S, Zink A, Shavlokhova V, Couillard-Despres S, Boeker M, Obid P, Lang GM. Artificial Intelligence-Driven Prediction Modeling and Decision Making in Spine Surgery Using Hybrid Machine Learning Models. J Pers Med 2022; 12:jpm12040509. [PMID: 35455625 PMCID: PMC9029065 DOI: 10.3390/jpm12040509] [Citation(s) in RCA: 38] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2022] [Revised: 03/18/2022] [Accepted: 03/19/2022] [Indexed: 12/22/2022] Open
Abstract
Healthcare systems worldwide generate vast amounts of data from many different sources. Although of high complexity for a human being, it is essential to determine the patterns and minor variations in the genomic, radiological, laboratory, or clinical data that reliably differentiate phenotypes or allow high predictive accuracy in health-related tasks. Convolutional neural networks (CNN) are increasingly applied to image data for various tasks. Its use for non-imaging data becomes feasible through different modern machine learning techniques, converting non-imaging data into images before inputting them into the CNN model. Considering also that healthcare providers do not solely use one data modality for their decisions, this approach opens the door for multi-input/mixed data models which use a combination of patient information, such as genomic, radiological, and clinical data, to train a hybrid deep learning model. Thus, this reflects the main characteristic of artificial intelligence: simulating natural human behavior. The present review focuses on key advances in machine and deep learning, allowing for multi-perspective pattern recognition across the entire information set of patients in spine surgery. This is the first review of artificial intelligence focusing on hybrid models for deep learning applications in spine surgery, to the best of our knowledge. This is especially interesting as future tools are unlikely to use solely one data modality. The techniques discussed could become important in establishing a new approach to decision-making in spine surgery based on three fundamental pillars: (1) patient-specific, (2) artificial intelligence-driven, (3) integrating multimodal data. The findings reveal promising research that already took place to develop multi-input mixed-data hybrid decision-supporting models. Their implementation in spine surgery may hence be only a matter of time.
Collapse
Affiliation(s)
- Babak Saravi
- Department of Orthopedics and Trauma Surgery, Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, 79108 Freiburg, Germany; (S.Ü.); (P.O.); (G.M.L.)
- Department of Spine Surgery, Loretto Hospital, 79100 Freiburg, Germany; (F.H.); (A.Z.)
- Institute of Experimental Neuroregeneration, Spinal Cord Injury and Tissue Regeneration Center Salzburg (SCI-TReCS), Paracelsus Medical University, 5020 Salzburg, Austria;
- Correspondence:
| | - Frank Hassel
- Department of Spine Surgery, Loretto Hospital, 79100 Freiburg, Germany; (F.H.); (A.Z.)
| | - Sara Ülkümen
- Department of Orthopedics and Trauma Surgery, Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, 79108 Freiburg, Germany; (S.Ü.); (P.O.); (G.M.L.)
- Department of Spine Surgery, Loretto Hospital, 79100 Freiburg, Germany; (F.H.); (A.Z.)
| | - Alisia Zink
- Department of Spine Surgery, Loretto Hospital, 79100 Freiburg, Germany; (F.H.); (A.Z.)
| | - Veronika Shavlokhova
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany;
| | - Sebastien Couillard-Despres
- Institute of Experimental Neuroregeneration, Spinal Cord Injury and Tissue Regeneration Center Salzburg (SCI-TReCS), Paracelsus Medical University, 5020 Salzburg, Austria;
- Austrian Cluster for Tissue Regeneration, 1200 Vienna, Austria
| | - Martin Boeker
- Intelligence and Informatics in Medicine, Medical Center Rechts der Isar, School of Medicine, Technical University of Munich, 81675 Munich, Germany;
| | - Peter Obid
- Department of Orthopedics and Trauma Surgery, Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, 79108 Freiburg, Germany; (S.Ü.); (P.O.); (G.M.L.)
| | - Gernot Michael Lang
- Department of Orthopedics and Trauma Surgery, Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, 79108 Freiburg, Germany; (S.Ü.); (P.O.); (G.M.L.)
| |
Collapse
|