A systematic review of methodological quality of model development studies predicting prognostic outcome for resectable pancreatic cancer

Systematic review highlights high risk of bias of clinical prediction models for blood transfusion in patients undergoing elective surgery

BACKGROUND

Blood transfusion can be a lifesaving intervention after perioperative blood loss. Many prediction models have been developed to identify patients most likely to require blood transfusion during elective surgery, but it is unclear whether any are suitable for clinical practice.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching MEDLINE, Embase, PubMed, The Cochrane Library, Transfusion Evidence Library, Scopus, and Web of Science databases for studies reporting the development or validation of a blood transfusion prediction model in elective surgery patients between January 1, 2000 and June 30, 2021. We extracted study characteristics, discrimination performance (c-statistics) of final models, and data, which we used to perform risk of bias assessment using the Prediction model risk of bias assessment tool (PROBAST).

RESULTS

We reviewed 66 studies (72 developed and 48 externally validated models). Pooled c-statistics of externally validated models ranged from 0.67 to 0.78. Most developed and validated models were at high risk of bias due to handling of predictors, validation methods, and too small sample sizes.

CONCLUSION

Most blood transfusion prediction models are at high risk of bias and suffer from poor reporting and methodological quality, which must be addressed before they can be safely used in clinical practice.

Glioblastoma glycolytic signature predicts unfavorable prognosis, immunological heterogeneity, and ENO1 promotes microglia M2 polarization and cancer cell malignancy

Glioblastomas are the most malignant brain tumors, whose progress was promoted by aberrate aerobic glycolysis. The immune environment was highly engaged in glioblastoma formation, while its interaction with aerobic glycolysis remained unclear. Herein, we build a 7-gene Glycolytic Score (GS) by Elastic Net in the training set and two independent validating sets. The GS predicted malignant features and poor survival with good performances. Immune functional analyses and Cibersort calculation identified depressed T cells, B cells, natural killer cells immunity, and high immunosuppressive cell infiltration in the high-GS group. Also, high expressions of the immune-escape genes were discovered. Subsequently, the single-cell analyses validated the glycolysis-related immunosuppression. The functional results manifested the high-GS neoplastic cells' association with T cells, NK cells, and macrophage function regulation. The intercellular cross-talk showed strong associations between high-GS neoplastic cells and M2 macrophages/microglia in several immunological pathways. We finally confirmed that ENO1, the key gene of the GS, promoted M2 microglia polarization and glioblastoma cell malignant behaviors via immunofluorescence, clone formation, CCK8, and transwell rescue experiments. These results indicated the interactions between cancerous glycolysis and immunosuppression and glycolysis' role in promoting glioblastoma progression. Conclusively, we built a robust model and discovered strong interaction between GS and immune, shedding light on prognosis management improvement and therapeutic strategies development for glioblastoma patients.

Preoperative data-based deep learning model for predicting postoperative survival in pancreatic cancer patients

BACKGROUND

Pancreatic ductal adenocarcinoma (PDAC) has a poor prognosis even after curative resection. A deep learning-based stratification of postoperative survival in the preoperative setting may aid the treatment decisions for improving prognosis. This study was aimed to develop a deep learning model based on preoperative data for predicting postoperative survival.

METHODS

The patients who underwent surgery for PDAC between January 2014 and May 2015. Clinical data-based machine learning models and computed tomography (CT) data-based deep learning models were developed separately, and ensemble learning was utilized to combine two models. The primary outcomes were the prediction of 2-year overall survival (OS) and 1-year recurrence-free survival (RFS). The model's performance was measured by area under the receiver operating curve (AUC) and was compared with that of American Joint Committee on Cancer (AJCC) 8th stage.

RESULTS

The median OS and RFS were 23 and 10 months in training dataset (n = 229), and 22 and 11 months in test dataset (n = 53), respectively. The AUC of the ensemble model for predicting 2-year OS and 1-year RFS in the test dataset was 0.76 and 0.74, respectively. The performance of the ensemble model was comparable to that of the AJCC in predicting 2-year OS (AUC, 0.67; P = 0.35) and superior to the AJCC in predicting 1-year RFS (AUC, 0.54; P = 0.049).

CONCLUSION

and relevance: Our ensemble model based on routine preoperative variables showed good performance for predicting prognosis for PDAC patients after surgery.

Risk of bias of prognostic models developed using machine learning: a systematic review in oncology

BACKGROUND

Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain.

METHODS

We conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately.

RESULTS

We included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation.

CONCLUSIONS

The quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models.

Consensus Statement on Mandatory Measurements for Pancreatic Cancer Trials for Patients With Resectable or Borderline Resectable Disease (COMM-PACT-RB): A Systematic Review and Delphi Consensus Statement

Importance

Pancreatic cancer is the third most common cause of cancer death; however, randomized clinical trials (RCTs) of survival in patients with resectable pancreatic cancer lack mandatory measures for reporting baseline and prognostic factors, which hampers comparisons between outcome measures.

Objective

To develop a consensus on baseline and prognostic factors to be used as mandatory measurements in RCTs of resectable and borderline resectable pancreatic cancer.

Evidence Review

We performed a systematic literature search of the Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, and Embase for RCTs on resectable and borderline resectable pancreatic cancer with overall survival as the primary outcome. We produced a systematic summary of all baseline and prognostic factors identified in the RCTs. A Delphi panel that included 13 experts was surveyed to reach a consensus on mandatory and recommended baseline and prognostic factors.

Findings

The 42 RCTs that met inclusion criteria reported a total of 60 baseline and 19 prognostic factors. After 2 Delphi rounds, agreement was reached on 50 mandatory baseline and 20 mandatory prognostic factors for future RCTs, with a distinction between studies of neoadjuvant vs adjuvant treatment.

Conclusion and Relevance

This findings of this systematic review and international expert consensus have produced this Consensus Statement on Mandatory Measurements in Pancreatic Cancer Trials for Resectable and Borderline Resectable Disease (COMM-PACT-RB). The baseline and prognostic factors comprising the mandatory measures will facilitate better comparison across RCTs and eventually will enable improved clinical practice among patients with resectable and borderline resectable pancreatic cancer.

Structured reporting to improve transparency of analyses in prognostic marker studies

BACKGROUND

Factors contributing to the lack of understanding of research studies include poor reporting practices, such as selective reporting of statistically significant findings or insufficient methodological details. Systematic reviews have shown that prognostic factor studies continue to be poorly reported, even for important aspects, such as the effective sample size. The REMARK reporting guidelines support researchers in reporting key aspects of tumor marker prognostic studies. The REMARK profile was proposed to augment these guidelines to aid in structured reporting with an emphasis on including all aspects of analyses conducted.

METHODS

A systematic search of prognostic factor studies was conducted, and fifteen studies published in 2015 were selected, three from each of five oncology journals. A paper was eligible for selection if it included survival outcomes and multivariable models were used in the statistical analyses. For each study, we summarized the key information in a REMARK profile consisting of details about the patient population with available variables and follow-up data, and a list of all analyses conducted.

RESULTS

Structured profiles allow an easy assessment if reporting of a study only has weaknesses or if it is poor because many relevant details are missing. Studies had incomplete reporting of exclusion of patients, missing information about the number of events, or lacked details about statistical analyses, e.g., subgroup analyses in small populations without any information about the number of events. Profiles exhibit severe weaknesses in the reporting of more than 50% of the studies. The quality of analyses was not assessed, but some profiles exhibit several deficits at a glance.

CONCLUSIONS

A substantial part of prognostic factor studies is poorly reported and analyzed, with severe consequences for related systematic reviews and meta-analyses. We consider inadequate reporting of single studies as one of the most important reasons that the clinical relevance of most markers is still unclear after years of research and dozens of publications. We conclude that structured reporting is an important step to improve the quality of prognostic marker research and discuss its role in the context of selective reporting, meta-analysis, study registration, predefined statistical analysis plans, and improvement of marker research.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

BACKGROUND

Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology.

METHODS

We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models.

RESULTS

Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available.

CONCLUSIONS

The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Appraisal of multivariable prognostic models for post-operative liver decompensation following partial hepatectomy: a systematic review

BACKGROUND

Few reports have evaluated prognostic modelling studies of tools used for surgical decision-making. This systematic review aimed to describe and critically appraise studies that have developed or validated multivariable prognostic models for post-operative liver decompensation following partial hepatectomy.

METHODS

This study was designed using the CHARMS checklist. Following a comprehensive literature search, two reviewers independently screened candidate references for inclusion and abstracted relevant study details. Qualitative assessment was performed using the PROBAST tool.

RESULTS

We identified 36 prognostic modelling studies; 25 focused on development only, 3 developed and validated models, and 8 validated pre-existing models. None compared routine use of a prognostic model against standard clinical practice. Most studies used single-institution, retrospective cohort designs, conducted in Eastern populations. In total, 15 different outcome definitions for post-operative liver decompensation events were used. Statistical concerns surrounding model overfitting, performance assessment, and internal validation led to high risk of bias for all studies.

CONCLUSIONS

Current prognostic models for post-operative liver decompensation following partial hepatectomy may not be valid for routine clinical use due to design and methodologic concerns. Landmark resources and reporting guidelines such as the TRIPOD statement may assist researchers, and additionally, model impact assessment studies represent opportunities for future research.

Development and Validation of Prognostic Models for Oral Squamous Cell Carcinoma: A Systematic Review and Appraisal of the Literature

(1) Background: An accurate prediction of cancer survival is very important for counseling, treatment planning, follow-up, and postoperative risk assessment in patients with Oral Squamous Cell Carcinoma (OSCC). There has been an increased interest in the development of clinical prognostic models and nomograms which are their graphic representation. The study aimed to revise the prognostic performance of clinical-pathological prognostic models with internal validation for OSCC. (2) Methods: This systematic review was performed according to the Cochrane Handbook for Diagnostic Test Accuracy Reviews chapter on searching, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines, and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). (3) Results: Six studies evaluating overall survival in patients with OSCC were identified. All studies performed internal validation, while only four models were externally validated. (4) Conclusions: Based on the results of this systematic review, it is possible to state that it is necessary to carry out internal validation and shrinkage to correct overfitting and provide an adequate performance for optimism. Moreover, calibration, discrimination and nonlinearity of continuous predictors should always be examined. To reduce the risk of bias the study design used should be prospective and imputation techniques should always be applied to handle missing data. In addition, the complete equation of the prognostic model must be reported to allow updating, external validation in a new context and the subsequent evaluation of the impact on health outcomes and on the cost-effectiveness of care.

Promoting Prognostic Model Application: A Review Based on Gliomas

Malignant neoplasms are characterized by poor therapeutic efficacy, high recurrence rate, and extensive metastasis, leading to short survival. Previous methods for grouping prognostic risks are based on anatomic, clinical, and pathological features that exhibit lower distinguishing capability compared with genetic signatures. The update of sequencing techniques and machine learning promotes the genetic panels-based prognostic model development, especially the RNA-panel models. Gliomas harbor the most malignant features and the poorest survival among all tumors. Currently, numerous glioma prognostic models have been reported. We systematically reviewed all 138 machine-learning-based genetic models and proposed novel criteria in assessing their quality. Besides, the biological and clinical significance of some highly overlapped glioma markers in these models were discussed. This study screened out markers with strong prognostic potential and 27 models presenting high quality. Conclusively, we comprehensively reviewed 138 prognostic models combined with glioma genetic panels and presented novel criteria for the development and assessment of clinically important prognostic models. This will guide the genetic models in cancers from laboratory-based research studies to clinical applications and improve glioma patient prognostic management.

Promoting Prognostic Model Application: A Review Based on Gliomas

Malignant neoplasms are characterized by poor therapeutic efficacy, high recurrence rate, and extensive metastasis, leading to short survival. Previous methods for grouping prognostic risks are based on anatomic, clinical, and pathological features that exhibit lower distinguishing capability compared with genetic signatures. The update of sequencing techniques and machine learning promotes the genetic panels-based prognostic model development, especially the RNA-panel models. Gliomas harbor the most malignant features and the poorest survival among all tumors. Currently, numerous glioma prognostic models have been reported. We systematically reviewed all 138 machine-learning-based genetic models and proposed novel criteria in assessing their quality. Besides, the biological and clinical significance of some highly overlapped glioma markers in these models were discussed. This study screened out markers with strong prognostic potential and 27 models presenting high quality. Conclusively, we comprehensively reviewed 138 prognostic models combined with glioma genetic panels and presented novel criteria for the development and assessment of clinically important prognostic models. This will guide the genetic models in cancers from laboratory-based research studies to clinical applications and improve glioma patient prognostic management.

A general presentation on how to carry out a CHARMS analysis for prognostic multivariate models

The CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist was created to provide methodological appraisals of predictive models, based on the best available scientific evidence and through systematic reviews. Our purpose is to give a general presentation on how to carry out a CHARMS analysis for prognostic multivariate models, making clear what the steps are and how they are applied individually to the studies included in the systematic review. This tutorial is aimed at providing such a resource. In addition to this explanation, we will apply the method to a real case: predictive models of atrial fibrillation in the community. This methodology could be applied to other predictive models using the steps provided in our review so as to have complete information for each included model and determine whether it can be implemented in daily clinical practice.

C-Reactive Protein and Neutrophil/Lymphocytes Ratio: Prognostic Indicator for Doubling overall survival Prediction in Pancreatic Cancer Patients

Background: Despite modern chemotherapy regimens, survival of patients with locally advanced/metastatic pancreatic cancer remains dismal. Long-term survivors are rare and there are no prognostic scores to identify patients benefitting most from chemotherapy. Methods: This retrospective study includes 240 patients with pancreatic cancer who were treated in a primary palliative setting between the years 2007 to 2016 in a single academic institution. Survival rates were analyzed using the Kaplan–Meier method. Prognostic models including laboratory and clinical parameters were calculated using Cox proportional models in univariate and multivariate analyses. Results: Median age at diagnosis was 67 years (range 29–90 years), 52% were female and a majority had an ECOG performance status of 0 or 1. Locally advanced pancreatic cancer was diagnosed in 23.3% (n = 56) and primary metastatic disease in 76.7% (n = 184) of all patients. Median overall survival of the whole study cohort was 8.3 months. Investigating potential risk factors like patient characteristics, tumor marker or inflammatory markers, multivariate survival analysis found CRP (c-reactive protein) and NLR (neutrophil to lymphocyte ratio) elevation before the start of palliative chemotherapy to be independent negative prognostic factors for OS (overall survival) (p < 0.001 and p < 0.01). Grouping patients with no risk factor versus patients with one or two of the above mentioned two risk factors, we found a median OS of 16.8 months and 9.4 months (p < 0.001) respectively. By combining these two factors, we were also able to identify pancreatic cancer patients that were more likely to receive any post first line therapy. These two risk factors are predictive for improved survival independent of disease stage (III or IV) and applied chemotherapy agents in first line. Conclusion: By combining these two factors, CRP and NLR, to create a score for OS, we propose a simple, new prognostic tool for OS prediction in pancreatic cancer.