1
|
Yang X, Yang S, Bao Y, Wang Q, Peng Z, Lu S. Novel machine-learning prediction tools for overall survival of patients with chondrosarcoma: Based on recursive partitioning analysis. Cancer Med 2024; 13:e70058. [PMID: 39123313 PMCID: PMC11315679 DOI: 10.1002/cam4.70058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 07/04/2024] [Accepted: 07/20/2024] [Indexed: 08/12/2024] Open
Abstract
BACKGROUND Chondrosarcoma (CHS), a bone malignancy, poses a significant challenge due to its heterogeneous nature and resistance to conventional treatments. There is a clear need for advanced prognostic instruments that can integrate multiple prognostic factors to deliver personalized survival predictions for individual patients. This study aimed to develop a novel prediction tool based on recursive partitioning analysis (RPA) to improve the estimation of overall survival for patients with CHS. METHODS Data from the Surveillance, Epidemiology, and End Results (SEER) database were analyzed, including demographic, clinical, and treatment details of patients diagnosed between 2000 and 2018. Using C5.0 algorithm, decision trees were created to predict survival probabilities at 12, 24, 60, and 120 months. The performance of the models was assessed through confusion scatter plot, accuracy rate, receiver operator characteristic (ROC) curve, and area under ROC curve (AUC). RESULTS The study identified tumor histology, surgery, age, visceral (brain/liver/lung) metastasis, chemotherapy, tumor grade, and sex as critical predictors. Decision trees revealed distinct patterns for survival prediction at each time point. The models showed high accuracy (82.40%-89.09% in training group, and 82.16%-88.74% in test group) and discriminatory power (AUC: 0.806-0.894 in training group, and 0.808-0.882 in test group) in both training and testing datasets. An interactive web-based shiny APP (URL: https://yangxg1209.shinyapps.io/chondrosarcoma_survival_prediction/) was developed, simplifying the survival prediction process for clinicians. CONCLUSIONS This study successfully employed RPA to develop a user-friendly tool for personalized survival predictions in CHS. The decision tree models demonstrated robust predictive capabilities, with the interactive application facilitating clinical decision-making. Future prospective studies are recommended to validate these findings and further refine the predictive model.
Collapse
Affiliation(s)
- Xiong‐Gang Yang
- Department of Orthopedics, The First People's Hospital of Yunnan ProvinceThe Affiliated Hospital of Kunming University of Science and TechnologyKunmingYunnanChina
- The Key Laboratory of Digital Orthopedics of Yunnan ProvinceKunmingYunnanChina
| | - Shan‐Shan Yang
- Department of ProsthodonticsAffiliated Stomatological Hospital of Zunyi Medical University, Zunyi Medical UniversityZunyiChina
| | - Yi Bao
- Department of Orthopedics, The First People's Hospital of Yunnan ProvinceThe Affiliated Hospital of Kunming University of Science and TechnologyKunmingYunnanChina
- The Key Laboratory of Digital Orthopedics of Yunnan ProvinceKunmingYunnanChina
| | - Qi‐Yang Wang
- Department of Orthopedics, The First People's Hospital of Yunnan ProvinceThe Affiliated Hospital of Kunming University of Science and TechnologyKunmingYunnanChina
- The Key Laboratory of Digital Orthopedics of Yunnan ProvinceKunmingYunnanChina
| | - Zhi Peng
- Department of Orthopedics, The First People's Hospital of Yunnan ProvinceThe Affiliated Hospital of Kunming University of Science and TechnologyKunmingYunnanChina
- The Key Laboratory of Digital Orthopedics of Yunnan ProvinceKunmingYunnanChina
| | - Sheng Lu
- Department of Orthopedics, The First People's Hospital of Yunnan ProvinceThe Affiliated Hospital of Kunming University of Science and TechnologyKunmingYunnanChina
- The Key Laboratory of Digital Orthopedics of Yunnan ProvinceKunmingYunnanChina
| |
Collapse
|
2
|
Yang SS, Yang XG, Hu XH, Yang XH. Novel decision tree models predict the overall survival of patients with submandibular gland cancer. Clin Oral Investig 2024; 28:395. [PMID: 38916666 DOI: 10.1007/s00784-024-05703-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2023] [Accepted: 05/06/2024] [Indexed: 06/26/2024]
Abstract
BACKGROUND While the accurate prediction of the overall survival (OS) in patients with submandibular gland cancer (SGC) is paramount for informed therapeutic planning, the development of reliable survival prediction models has been hindered by the rarity of SGC cases. The purpose of this study is to identify key prognostic factors for OS in SGC patients using a large database and construct decision tree models to aid the prediction of survival probabilities in 12, 24, 60 and 120 months. MATERIALS AND METHODS We performed a retrospective cohort study using the Surveillance, Epidemiology and End Result (SEER) program. Demographic and peri-operative predictor variables were identified. The outcome variables overall survival at 12-, 24-, 60, and 120 months. The C5.0 algorithm was utilized to establish the dichotomous decision tree models, with the depth of tree limited within 4 layers. To evaluate the performances of the novel models, the receiver operator characteristic (ROC) curves were generated, and the metrics such as accuracy rate, and area under ROC curve (AUC) were calculated. RESULTS A total of 1,705, 1,666, 1,543, and 1,413 SGC patients with a follow up of 12, 24, 60 and 120 months and exact survival status were identified from the SEER database. Predictor variables of age, sex, surgery, radiation, chemotherapy, tumor histology, summary stage, metastasis to distant lymph node, and marital status exerted substantial influence on overall survival. Decision tree models were then developed, incorporating these vital prognostic indicators. Favorable consistency was presented between the predicted and actual survival statuses. For the training dataset, the accuracy rates for the 12-, 24-, 60- and 120-month survival models were 0.866, 0.767, 0.737 and 0.797. Correspondingly, the AUC values were 0.841, 0.756, 0.725, and 0.774 for the same time points. CONCLUSIONS Based on the most important predictor variables identified using the large, SEER database, decision tree models were established that predict OS of SGC patients. The models offer a more exhaustive evaluation of mortality risk and may lead to more personalized treatment strategies.
Collapse
Affiliation(s)
- Shan-Shan Yang
- Hospital/School of Stomatology, Zunyi Medical University, No. 89, Wujiang East Road, Xinpu New District, Zunyi City, Guizhou Province, 563000, China
| | - Xiong-Gang Yang
- Department of Orthopaedics, The First People's Hospital of Yunnan Province, Affiliated Hospital of Kunming University of Science and Technology, Kunming, 650032, China
| | - Xiao-Hua Hu
- Hospital/School of Stomatology, Zunyi Medical University, No. 89, Wujiang East Road, Xinpu New District, Zunyi City, Guizhou Province, 563000, China.
| | - Xiao-Hong Yang
- Hospital/School of Stomatology, Zunyi Medical University, No. 89, Wujiang East Road, Xinpu New District, Zunyi City, Guizhou Province, 563000, China.
| |
Collapse
|
3
|
Dhiman P, Ma J, Andaur Navarro CL, Speich B, Bullock G, Damen JAA, Hooft L, Kirtley S, Riley RD, Van Calster B, Moons KGM, Collins GS. Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review. J Clin Epidemiol 2023; 157:120-133. [PMID: 36935090 DOI: 10.1016/j.jclinepi.2023.03.012] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 03/03/2023] [Accepted: 03/14/2023] [Indexed: 03/19/2023]
Abstract
OBJECTIVES In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction. STUDY DESIGN AND SETTING We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices. RESULTS We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion. CONCLUSION The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.
Collapse
Affiliation(s)
- Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Constanza L Andaur Navarro
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Benjamin Speich
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; Meta-Research Centre, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Garrett Bullock
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Johanna A A Damen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Shona Kirtley
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK, ST5 5BG
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands; EPI-centre, KU Leuven, Leuven, Belgium
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| |
Collapse
|
4
|
Dhiman P, Ma J, Andaur Navarro CL, Speich B, Bullock G, Damen JAA, Hooft L, Kirtley S, Riley RD, Van Calster B, Moons KGM, Collins GS. Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review. BMC Med Res Methodol 2022; 22:101. [PMID: 35395724 PMCID: PMC8991704 DOI: 10.1186/s12874-022-01577-x] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2021] [Accepted: 03/18/2022] [Indexed: 12/13/2022] Open
Abstract
BACKGROUND Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.
Collapse
Affiliation(s)
- Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK.
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| | - Constanza L Andaur Navarro
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Benjamin Speich
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
- Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Garrett Bullock
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Johanna A A Damen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Shona Kirtley
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, ST5 5BG, UK
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
- EPI-centre, KU Leuven, Leuven, Belgium
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| |
Collapse
|
5
|
Walker A, Bassale S, Shukla R, Kubicky CD. A Prognostic Index for Predicting Survival of Patients Undergoing Radiation Therapy for Spine Metastasis Using Recursive Partitioning Analysis. J Palliat Med 2021; 25:21-27. [PMID: 34382867 DOI: 10.1089/jpm.2020.0715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
Background: Physicians' estimate of life expectancy in patients with spine metastasis frequently impacts treatment decisions regarding surgery, radiation techniques, dose, and fractionation. Objective: We aimed to identify predictors of survival and generate a stratification schema that could guide clinical decision making. Materials and Methods: We identified 269 patients who have undergone surgery and/or radiation for spine metastasis from 2002 to 2013 at an academic medical institution in the United States. A univariate survival analysis was carried out using the Kaplan-Meier method. Differences in survival by histology were assessed using the log-rank test. Multivariate analysis was performed using the Cox proportional hazards model, then using the same variables, recursive partitioning analysis (RPA) was conducted to determine risk groups associated with survival. Results: The median overall survival was 4.76 months. Twenty percent, 40%, and 57% of patients died within one, three, and six months of radiation treatment, respectively. RPA analysis resulted in three classes; class I included patients with Karnofsky Performance Status (KPS) ≥80. Class II included patients with KPS <80 and radioresistant or favorable histologies. Class III included all other histologies. Median survival in months was 11.4, 6.3, and 2.0, respectively. Conclusion: We developed a stratification schema predictive of survival in patients with spine metastasis. This RPA classification should be validated in independent patient populations from several institutions and may ultimately identify patients who are good candidates for more complex treatment regimens, such as stereotactic body radiotherapy.
Collapse
Affiliation(s)
- Allison Walker
- Department of Obstetrics and Gynecology, Oregon Health and Science University School of Medicine, Portland, Oregon, USA
| | - Solange Bassale
- Knight Cancer Institute Biostatistics Shared Resource and Oregon Health and Science University, Portland, Oregon, USA
| | - Rakendu Shukla
- Department of Obstetrics and Gynecology, Oregon Health and Science University School of Medicine, Portland, Oregon, USA
| | - Charlotte Dai Kubicky
- Department of Radiation Medicine, Oregon Health and Science University, Portland, Oregon, USA
| |
Collapse
|
6
|
Gao ZY, Zhang T, Zhang H, Pang CG, Jiang WX. Establishment and validation of nomogram model for survival predicting in patients with spinal metastases secondary to lung cancer. Neurol Res 2020; 43:327-335. [PMID: 33377432 DOI: 10.1080/01616412.2020.1866244] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
OBJECTIVES To evaluate the prognostic effect of pre-treatment factors in patients with spinal metastases secondary to lung cancer, and establish a novel predicting nomogram for predicting the survival probability. METHODS A total of 209 patients operated for spinal metastases from lung cancer were consecutively enrolled, and divided into the training and validation samples with a ratio of 7:3, for model establishing and validating, respectively. Basing on the training sample, univariate and multivariate COX proportional hazard models were used for identifying the prognostic effect of pre-treatment factors, following which significant prognostic factors would be listed as items in nomogram to calculate the survival probabilities at 3, 6, 12 and 18 months. Then, the C-indexes and the calibration curves would be figured out to evaluate the discrimination ability and accuracy of the model both for the training and validation samples. RESULTS In the multivariate COX analysis, the gender, smoking history, location of spinal metastasis, visceral metastasis, Karnofsky performance status (KPS), adjuvant therapy, lymphocyte percentage and globulin were found to be significantly associated with the overall survival, and a novel nomogram was generated basing on these independent predictors. The C-indexes for the training and validation samples were 0.761 and 0.732, respectively. Favorable consistencies between the predicted and actual survival rates were demonstrated both in the internal and external validations. DISCUSSION Pre-treatment characteristics, including gender, smoking history, location of spinal metastasis, visceral metastasis, KPS, adjuvant therapy, percentage of lymphocyte, and serum globulin level, were identified to be significantly associated with overall survival of patients living with spinal metastases derived from lung cancer, and a user-friendly nomogram was established using these independent predictors.
Collapse
Affiliation(s)
- Zhong-Yu Gao
- Department of Orthopedic Surgery, Tianjin First Central Hospital, Tianjin, China
| | - Tao Zhang
- Department of Orthopedic Surgery, Tianjin First Central Hospital, Tianjin, China
| | - Hui Zhang
- Department of Orthopedic Surgery, Tianjin First Central Hospital, Tianjin, China
| | | | - Wen-Xue Jiang
- Department of Orthopedic Surgery, Tianjin First Central Hospital, Tianjin, China
| |
Collapse
|