1
|
Bozzo A, Hollingsworth A, Chatterjee S, Apte A, Deng J, Sun S, Tap W, Aoude A, Bhatnagar S, Healey JH. A multimodal neural network with gradient blending improves predictions of survival and metastasis in sarcoma. NPJ Precis Oncol 2024; 8:188. [PMID: 39237726 PMCID: PMC11377835 DOI: 10.1038/s41698-024-00695-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Accepted: 08/30/2024] [Indexed: 09/07/2024] Open
Abstract
The objective of this study is to develop a multimodal neural network (MMNN) model that analyzes clinical variables and MRI images of a soft tissue sarcoma (STS) patient, to predict overall survival and risk of distant metastases. We compare the performance of this MMNN to models based on clinical variables alone, radiomics models, and an unimodal neural network. We include patients aged 18 or older with biopsy-proven STS who underwent primary resection between January 1st, 2005, and December 31st, 2020 with complete outcome data and a pre-treatment MRI with both a T1 post-contrast sequence and a T2 fat-sat sequence available. A total of 9380 MRI slices containing sarcomas from 287 patients are available. Our MMNN accepts the entire 3D sarcoma volume from T1 and T2 MRIs and clinical variables. Gradient blending allows the clinical and image sub-networks to optimally converge without overfitting. Heat maps were generated to visualize the salient image features. Our MMNN outperformed all other models in predicting overall survival and the risk of distant metastases. The C-Index of our MMNN for overall survival is 0.77 and the C-Index for risk of distant metastases is 0.70. The provided heat maps demonstrate areas of sarcomas deemed most salient for predictions. Our multimodal neural network with gradient blending improves predictions of overall survival and risk of distant metastases in patients with soft tissue sarcoma. Future work enabling accurate subtype-specific predictions will likely utilize similar end-to-end multimodal neural network architecture and require prospective curation of high-quality data, the inclusion of genomic data, and the involvement of multiple centers through federated learning.
Collapse
Affiliation(s)
- Anthony Bozzo
- Orthopaedic Service of the Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA.
- Division of Orthopaedic Surgery, McGill University, Montreal, QC, Canada.
| | - Alex Hollingsworth
- AI/ML and NextGen Analytics, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Subrata Chatterjee
- AI/ML and NextGen Analytics, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Aditya Apte
- Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Jiawen Deng
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Simon Sun
- Musculoskeletal Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - William Tap
- Medical Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Ahmed Aoude
- Division of Orthopaedic Surgery, McGill University, Montreal, QC, Canada
| | - Sahir Bhatnagar
- Department of Epidemiology and Biostatistics, McGill University, Montreal, QC, Canada
| | - John H Healey
- Orthopaedic Service of the Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| |
Collapse
|
2
|
Bozzo A, Tsui JMG, Bhatnagar S, Forsberg J. Deep Learning and Multimodal Artificial Intelligence in Orthopaedic Surgery. J Am Acad Orthop Surg 2024; 32:e523-e532. [PMID: 38652882 PMCID: PMC11075751 DOI: 10.5435/jaaos-d-23-00831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Revised: 02/13/2024] [Accepted: 03/01/2024] [Indexed: 04/25/2024] Open
Abstract
This review article focuses on the applications of deep learning with neural networks and multimodal neural networks in the orthopaedic domain. By providing practical examples of how artificial intelligence (AI) is being applied successfully in orthopaedic surgery, particularly in the realm of imaging data sets and the integration of clinical data, this study aims to provide orthopaedic surgeons with the necessary tools to not only evaluate existing literature but also to consider AI's potential in their own clinical or research pursuits. We first review standard deep neural networks which can analyze numerical clinical variables, then describe convolutional neural networks which can analyze image data, and then introduce multimodal AI models which analyze various types of different data. Then, we contrast these deep learning techniques with related but more limited techniques such as radiomics, describe how to interpret deep learning studies, and how to initiate such studies at your institution. Ultimately, by empowering orthopaedic surgeons with the knowledge and know-how of deep learning, this review aspires to facilitate the translation of research into clinical practice, thereby enhancing the efficacy and precision of real-world orthopaedic care for patients.
Collapse
Affiliation(s)
- Anthony Bozzo
- From the Division of Orthopaedic Surgery, McGill University, Canada (Bozzo), the Division of Radiation Oncology, McGill University, Canada (Tsui), the Department of Epidemiology and Biostatistics, Department of Diagnostic Radiology, McGill University, Canada (Bhatnagar), and the Memorial Sloan Kettering Cancer Center (Forsberg)
| | - James M. G. Tsui
- From the Division of Orthopaedic Surgery, McGill University, Canada (Bozzo), the Division of Radiation Oncology, McGill University, Canada (Tsui), the Department of Epidemiology and Biostatistics, Department of Diagnostic Radiology, McGill University, Canada (Bhatnagar), and the Memorial Sloan Kettering Cancer Center (Forsberg)
| | - Sahir Bhatnagar
- From the Division of Orthopaedic Surgery, McGill University, Canada (Bozzo), the Division of Radiation Oncology, McGill University, Canada (Tsui), the Department of Epidemiology and Biostatistics, Department of Diagnostic Radiology, McGill University, Canada (Bhatnagar), and the Memorial Sloan Kettering Cancer Center (Forsberg)
| | - Jonathan Forsberg
- From the Division of Orthopaedic Surgery, McGill University, Canada (Bozzo), the Division of Radiation Oncology, McGill University, Canada (Tsui), the Department of Epidemiology and Biostatistics, Department of Diagnostic Radiology, McGill University, Canada (Bhatnagar), and the Memorial Sloan Kettering Cancer Center (Forsberg)
| |
Collapse
|
3
|
Zhang Y, Wu Z, Chang J, Jiang W, Wang Y, Wang H, Li J, Li C, Li X. An updated incidence trends of soft-tissue sarcoma and cancer-specific survival of patients with primary soft-tissue sarcoma of liver: a population-based study. Expert Rev Gastroenterol Hepatol 2021; 15:689-698. [PMID: 33115276 DOI: 10.1080/17474124.2021.1842193] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
Objectives: This study aimed to evaluate and update incidence trends of soft-tissue sarcoma (STS) and to develop a nomogram to predict cancer-specific survival (CSS) in patients diagnosed with primary STS of the liver.Methods: Patients with hepatic STS were extracted from the Surveillance, Epidemiology, and End Results (SEER) database. Joinpoint regression analyses were performed to assess the incidence trends of STS. A nomogram was developed based on the independent risk factors chosen by Cox regression models. The calibration curve, area under the receiver operating characteristic curve (AUC), C-index, and decision curve analysis (DCA) were used to assess the predictive performance of the nomogram.Results: The incidence of STS increased between 1994 and 2012. There was a sudden decline in the incidence of STS from 2013. The incidence of STS was different in distinct races and genders. The nomogram for predicting the CSS of hepatic STS according to the independent factors was well calibrated and it displayed optimal discrimination power.Conclusion: This study highlights that age, sex, tumor size, quality of surgery, and histologic subtypes may contribute to the prognosis of hepatic STS, and STS may be etiologically distinct and should be considered separately in different races and genders.
Collapse
Affiliation(s)
- Yaodong Zhang
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Zhengshan Wu
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Jiang Chang
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Wangjie Jiang
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Yirui Wang
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Hongwei Wang
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Jinyang Li
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Changxian Li
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| | - Xiangcheng Li
- Key Laboratory on Living Donor Transplantation, Ministry of Health, Department of Liver Surgery, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu Province, China
| |
Collapse
|
4
|
Gamboa AC, Gronchi A, Cardona K. Soft-tissue sarcoma in adults: An update on the current state of histiotype-specific management in an era of personalized medicine. CA Cancer J Clin 2020; 70:200-229. [PMID: 32275330 DOI: 10.3322/caac.21605] [Citation(s) in RCA: 273] [Impact Index Per Article: 68.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/12/2020] [Revised: 02/11/2020] [Accepted: 02/12/2020] [Indexed: 12/13/2022] Open
Abstract
Soft-tissue sarcomas (STS) are rare tumors that account for 1% of all adult malignancies, with over 100 different histologic subtypes occurring predominately in the trunk, extremity, and retroperitoneum. This low incidence is further complicated by their variable presentation, behavior, and long-term outcomes, which emphasize the importance of centralized care in specialized centers with a multidisciplinary team approach. In the last decade, there has been an effort to improve the quality of care for patients with STS based on anatomic site and histology, and multiple ongoing clinical trials are focusing on tailoring therapy to histologic subtype. This report summarizes the latest evidence guiding the histiotype-specific management of extremity/truncal and retroperitoneal STS with regard to surgery, radiation, and chemotherapy.
Collapse
Affiliation(s)
- Adriana C Gamboa
- Division of Surgical Oncology, Department of Surgery, Emory University, Atlanta, Georgia
| | - Alessandro Gronchi
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Kenneth Cardona
- Division of Surgical Oncology, Winship Cancer Institute, Emory University Hospital Midtown, Atlanta, Georgia
| |
Collapse
|
5
|
New research strategies in retroperitoneal sarcoma. The case of TARPSWG, STRASS and RESAR: making progress through collaboration. Curr Opin Oncol 2019; 31:310-316. [DOI: 10.1097/cco.0000000000000535] [Citation(s) in RCA: 41] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
6
|
Nizri E, Fiore M, Colombo C, Radaelli S, Callegaro D, Sanfilippo R, Sangalli C, Collini P, Morosi C, Stacchiotti S, Casali PG, Gronchi A. Completion surgery of residual disease after primary inadequate surgery of retroperitoneal sarcomas can salvage a selected subgroup of patients-A propensity score analysis. J Surg Oncol 2018; 119:318-323. [PMID: 30554403 DOI: 10.1002/jso.25337] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2018] [Accepted: 11/28/2018] [Indexed: 11/08/2022]
Abstract
BACKGROUND Patients with retroperitoneal sarcoma (RPSs) who undergo primary inadequate surgery before referral to specialized sarcoma centers may be considered for completion surgery (CS). We wanted to compare the outcome of these patients to those who underwent primary adequate surgery (PAS) at a single referral institution. METHODS We identified 34 patients who were referred for CS after primary inadequate surgery. Using a propensity score based on validated RPS outcome risk factors, we managed to match 28 patients to patients with PAS. RESULTS Median time lag between the first and second operation in CS patients was 5 months (2-15). Surgical extent was similar among groups (median number of organs resected = 3; P = 0.08), and macroscopically complete excision was achieved in all patients. The rate of severe complications did not differ between the groups (1 of 28 vs 3 of 28, respectively; P = 0.35) and no perioperative mortality was documented. Median follow-up was 43.5 months. Patients in the CS group had similar local recurrence-free survival (mean, 92.1 ± 9.7 vs 99.8 ± 12.4; P = 0.85) and relapse-free survival (mean, 88.7 ± 9.8 vs 80.9 ± 12.3; P = 0.3) to those with PAS. CONCLUSIONS CS has short- and long-term outcomes comparable to PAS. While primary surgery should always be carried out at a referral institution, some of the patients who undergo an initial incomplete resection at a non specialist center can still be offered a salvage procedure at a referral institution with comparable results.
Collapse
Affiliation(s)
- Eran Nizri
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy.,Department of Surgery A, Tel-Aviv Sourasky Medical Center, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Marco Fiore
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Chiara Colombo
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Stefano Radaelli
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Dario Callegaro
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Roberta Sanfilippo
- Department of Cancer Medicine, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Claudia Sangalli
- Department of Radiotherapy, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Paola Collini
- Department of Pathology, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Carlo Morosi
- Department of Radiology, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Silvia Stacchiotti
- Department of Cancer Medicine, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Paolo G Casali
- Department of Cancer Medicine, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy.,Oncology and Haemato-Oncology Department, University of Milan, Milan, Italy
| | - Alessandro Gronchi
- Department of Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| |
Collapse
|