Reporting performance of prognostic models in cancer: a review

Artificial intelligence methods available for cancer research

Cancer is a heterogeneous and multifaceted disease with a significant global footprint. Despite substantial technological advancements for battling cancer, early diagnosis and selection of effective treatment remains a challenge. With the convenience of large-scale datasets including multiple levels of data, new bioinformatic tools are needed to transform this wealth of information into clinically useful decision-support tools. In this field, artificial intelligence (AI) technologies with their highly diverse applications are rapidly gaining ground. Machine learning methods, such as Bayesian networks, support vector machines, decision trees, random forests, gradient boosting, and K-nearest neighbors, including neural network models like deep learning, have proven valuable in predictive, prognostic, and diagnostic studies. Researchers have recently employed large language models to tackle new dimensions of problems. However, leveraging the opportunity to utilize AI in clinical settings will require surpassing significant obstacles-a major issue is the lack of use of the available reporting guidelines obstructing the reproducibility of published studies. In this review, we discuss the applications of AI methods and explore their benefits and limitations. We summarize the available guidelines for AI in healthcare and highlight the potential role and impact of AI models on future directions in cancer research.

Prognostic models for predicting oncological outcomes after surgical resection of a nonmetastatic renal cancer: A critical review of current literature

Prognostic models can be valuable for clinicians in counseling and monitoring patients after the surgical resection of nonmetastatic renal cell carcinoma (nmRCC). Over the years, several risk prediction models have been developed, evolving significantly in their ability to predict recurrence and overall survival following surgery. This review comprehensively evaluates and critically appraises current prognostic models for nm-RCC after nephrectomy. The last 2 decades have witnessed a notable increase in the development of various prognostic risk models for RCC, incorporating clinical, pathological, genomic, and molecular factors, primarily using retrospective data. Only a limited number of these models have been developed using prospective data, and their performance has been less effective than expected when applied to broader, real-life patient populations. Recently, artificial intelligence (AI), especially machine learning and deep learning algorithms, has emerged as a significant tool in creating survival prediction models. However, their widespread application remains constrained due to limited external validation, a lack of cost-effectiveness analysis, and unconfirmed clinical utility. Although numerous models that integrate clinical, pathological, and molecular data have been proposed for nm-RCC risk stratification, none have conclusively demonstrated practical effectiveness. As a result, current guidelines do not endorse a specific model. The ongoing development and validation of AI algorithms in RCC risk prediction are crucial areas for future research.

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

BACKGROUND

External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential.

METHODS

A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule.

RESULTS

We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data.

DISCUSSION

Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.

Poor statistical reporting: do we have a reason for concern? A narrative review and recommendations

PURPOSE OF REVIEW

The aim of the review conducted was to present recent articles indicating the need to implement statistical recommendations in the daily work of biomedical journals.

RECENT FINDINGS

The most recent literature shows an unchanged percentage of journals using specialized statistical review over 20 years. The problems of finding statistical reviewers, the impractical way in which biostatistics is taught and the nonimplementation of published statistical recommendations contribute to the fact that a small percentage of accepted manuscripts contain correctly performed analysis. The statistical recommendations published for authors and editorial board members in recent years contain important advice, but more emphasis should be placed on their practical and rigorous implementation. If this is not the case, we will additionally continue to experience low reproducibility of the research.

SUMMARY

There is a low level of statistical reporting these days. Recommendations related to the statistical review of submitted manuscripts should be followed more rigorously.

Modified neoadjuvant rectal score as a novel prognostic model for rectal cancer patients who underwent chemoradiotherapy

BACKGROUND

The neoadjuvant rectal score (NAR score) has recently been proposed as a better prognostic model than the conventional TNM classification for rectal cancer patients that have undergone neoadjuvant chemoradiotherapy. We recently developed an apoptosis-detection technique for assessing the viability of residual tumors in resected specimens after chemoradiotherapy. This study aimed to establish an improved prognostic classification by combining the NAR score and the assessment of the apoptosis of residual cancer cells.

METHODS

We retrospectively enrolled 319 rectal cancer patients who underwent chemoradiotherapy followed by radical surgery. The recurrence-free survival and overall survival of the four models were compared: TNM stage, NAR score, modified TNM stage by re-staging according to cancer cell viability, and modified NAR score also by re-staging.

RESULTS

Downstaging of the ypT stage was observed in 15.5% of cases, whereas only 4.5% showed downstaging of ypN stage. C-index was highest for the modified NAR score (0.715), followed by the modified TNM, TNM, and NAR score. Similarly, Akaike's information criterion was smallest in the modified NAR score (926.2), followed by modified TNM, TNM, and NAR score, suggesting that the modified NAR score was the best among these four models. The overall survival results were similar: C-index was the highest (0.767) and Akaike's information criterion was the smallest (383.9) for the modified NAR score among the four models tested.

CONCLUSION

We established a novel prognostic model, for rectal cancer patients that have undergone neoadjuvant chemoradiotherapy, using a combination of apoptosis-detecting immunohistochemistry and neoadjuvant rectal scores.

Models to predict length of stay in the emergency department: a systematic literature review and appraisal

INTRODUCTION

Prolonged Length of Stay (LOS) in ED (Emergency Department) has been associated with poor clinical outcomes. Prediction of ED LOS may help optimize resource utilization, clinical management, and benchmarking. This study aims to systematically review models for predicting ED LOS and to assess the reporting and methodological quality about these models.

METHODS

The online database PubMed, Scopus, and Web of Science (10 Sep 2023) was searched for English language articles that reported prediction models of LOS in ED. Identified titles and abstracts were independently screened by two reviewers. All original papers describing either development (with or without internal validation) or external validation of a prediction model for LOS in ED were included.

RESULTS

Of 12,193 uniquely identified articles, 34 studies were included (29 describe the development of new models and five describe the validation of existing models). Different statistical and machine learning methods were applied to the papers. On the 39-point reporting score and 11-point methodological quality score, the highest reporting scores for development and validation studies were 39 and 8, respectively.

CONCLUSION

Various studies on prediction models for ED LOS were published but they are fairly heterogeneous and suffer from methodological and reporting issues. Model development studies were associated with a poor to a fair level of methodological quality in terms of the predictor selection approach, the sample size, reproducibility of the results, missing imputation technique, and avoiding dichotomizing continuous variables. Moreover, it is recommended that future investigators use the confirmed checklist to improve the quality of reporting.

Improving methods of clinical practice guidelines: From guidelines to pathways to fast-and-frugal trees and decision analysis to develop individualised patient care

BACKGROUND

Current methods for developing clinical practice guidelines have several limitations: they are characterised by the "black box" operation-a process with defined inputs and outputs but an incomplete understanding of its internal workings; they have "the integration problem"-a lack of framework for explicitly integrating factors such as patient preferences and trade-offs between benefits and harms; they generate one recommendation at a time that typically are not connected in a coherent analytical framework; and they apply to "average" patients, while clinicians and their patients seek advice tailored to individual circumstances.

METHODS

We propose augmenting the current guideline development method by converting evidence-based pathways into fast-and-frugal decision trees (FFTs) and integrating them with generalised decision curve analysis to formulate clear, individualised management recommendations.

RESULTS

We illustrate the process by developing recommendations for the management of heparin-induced thrombocytopenia (HIT). We converted evidence-based pathways for HIT, developed by the American Society of Hematology, into an FFT. Here, we consider only thrombotic complications and major bleeding. We leveraged the predictive potential of FFTs to compare the effects of argatroban, bivalirudin, fondaparinux, and direct oral anticoagulants (DOACs) using generalised decision curve analysis. We found that DOACs were superior to other treatments if the FFT-predicted probability of HIT exceeded 3%.

CONCLUSIONS

The proposed analytical framework connects guidelines, pathways, FFTs, and decision analysis, offering risk-tailored personalised recommendations and addressing current guideline development critiques.

Differences in adverse outcomes across race and ethnicity among Veterans with similar predicted risks of an overdose or suicide-related event

OBJECTIVE

To evaluate the degree to which differences in incidence of mortality and serious adverse events exist across patient race and ethnicity among Veterans Health Administration (VHA) patients receiving outpatient opioid prescriptions and who have similar predicted risks of adverse outcomes. Patients were assigned scores via the VHA Stratification Tool for Opioid Risk Mitigation (STORM), a model used to predict the risk of experiencing overdose- or suicide-related health care events or death. Individuals with the highest STORM risk scores are targeted for case review.

DESIGN

Retrospective cohort study of high-risk veterans who received an outpatient prescription opioid between 4/2018-3/2019.

SETTING

All VHA medical centers.

PARTICIPANTS

In total, 84 473 patients whose estimated risk scores were between 0.0420 and 0.0609, the risk scores associated with the top 5%-10% of risk in the STORM development sample.

METHODS

We examined the expected probability of mortality and serious adverse events (SAEs; overdose or suicide-related events) given a patient's risk score and race.

RESULTS

Given a similar risk score, Black patients were less likely than White patients to have a recorded SAE within 6 months of risk score calculation. Black, Hispanic, and Asian patients were less likely than White patients with similar risk scores to die within 6 months of risk score calculation. Some of the mortality differences were driven by age differences in the composition of racial and ethnic groups in our sample.

CONCLUSIONS

Our results suggest that relying on the STORM model to identify patients who may benefit from an interdisciplinary case review may identify patients with clinically meaningful differences in outcome risk across race and ethnicity.

Machine learning to improve interpretability of clinical, radiological and panel-based genomic data of glioma grade 4 patients undergoing surgical resection

BACKGROUND

Glioma grade 4 (GG4) tumors, including astrocytoma IDH-mutant grade 4 and the astrocytoma IDH wt are the most common and aggressive primary tumors of the central nervous system. Surgery followed by Stupp protocol still remains the first-line treatment in GG4 tumors. Although Stupp combination can prolong survival, prognosis of treated adult patients with GG4 still remains unfavorable. The introduction of innovative multi-parametric prognostic models may allow refinement of prognosis of these patients. Here, Machine Learning (ML) was applied to investigate the contribution in predicting overall survival (OS) of different available data (e.g. clinical data, radiological data, or panel-based sequencing data such as presence of somatic mutations and amplification) in a mono-institutional GG4 cohort.

METHODS

By next-generation sequencing, using a panel of 523 genes, we performed analysis of copy number variations and of types and distribution of nonsynonymous mutations in 102 cases including 39 carmustine wafer (CW) treated cases. We also calculated tumor mutational burden (TMB). ML was applied using eXtreme Gradient Boosting for survival (XGBoost-Surv) to integrate clinical and radiological information with genomic data.

RESULTS

By ML modeling (concordance (c)- index = 0.682 for the best model), the role of predicting OS of radiological parameters including extent of resection, preoperative volume and residual volume was confirmed. An association between CW application and longer OS was also showed. Regarding gene mutations, a role in predicting OS was defined for mutations of BRAF and of other genes involved in the PI3K-AKT-mTOR signaling pathway. Moreover, an association between high TMB and shorter OS was suggested. Consistently, when a cutoff of 1.7 mutations/megabase was applied, cases with higher TMB showed significantly shorter OS than cases with lower TMB.

CONCLUSIONS

The contribution of tumor volumetric data, somatic gene mutations and TBM in predicting OS of GG4 patients was defined by ML modeling.

Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement

OBJECTIVES

To (1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and (2) assess the inter-rater agreement of the PROBAST.

STUDY DESIGN AND SETTING

PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key publications. Inter-rater agreement was assessed using Cohen's Kappa.

RESULTS

One hundred and thirty nine systematic reviews were included, of which 85 reviews (containing 2,477 single studies) on domain level and 54 reviews (containing 2,458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04-0.26) and SQ level (Kappa -0.14 to 0.49).

CONCLUSION

Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key publications having no influence on ROB or recency of key publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST.

Pleural Fluid Resolution Is Associated with Improved Survival in Patients with Malignant Pleural Effusion

Malignant pleural effusion is associated with a poor prognosis and, while risk stratification models exist, prior studies have not evaluated pleural fluid resolution and its association with survival. We performed a retrospective review of patients diagnosed with malignant pleural effusion between 2013 and 2017, evaluating patient demographics, pleural fluid and serum composition, and procedural and treatment data using Cox regression analysis to evaluate associations with survival. In total, 123 patients were included in the study, with median survival from diagnosis being 4.8 months. Resolution of malignant pleural fluid was associated with a significant survival benefit, even when accounting for factors such as placement of an indwelling pleural catheter, anti-cancer therapy, pleural fluid cytology, cancer pheno/genotypes, and pleural fluid characteristics. Elevated fluid protein, placement of an indwelling pleural catheter, and treatment with targeted or hormone therapies were associated with pleural fluid resolution. We conclude that the resolution of pleural fluid accumulation in patients with malignant pleural effusion is associated with a survival benefit possibility representing a surrogate marker for treatment of the underlying metastatic cancer. These findings support the need to better understand the mechanism of fluid resolution in patients with malignant pleural effusion as well as the tumor-immune interplay occurring with the malignant pleural space.

Utility of gamma-glutamyl transpeptidase to lymphocyte count ratio in predicting prognosis of patients with oral cancer: A prospective cohort study in Southeastern China

BACKGROUND

To assess the prognostic role of gamma-glutamyl transpeptidase to lymphocyte count ratio (GLR) and develop a prognostic nomogram for patients with oral cancer.

METHODS

A prospective cohort (n = 1011) was conducted during July 2002 to March 2021 in Southeastern China.

RESULTS

The median follow-up time was 3.5 years. Multivariate Cox regression (OS: HR = 1.51, 95% CI: 1.04, 2.18) and Fine-Gray model (DSS: HR = 1.68, 95% CI: 1.14, 2.49) both showed that high GLR could act as an indicator of poor prognosis. A nonlinear dose-response relationship was observed between continuous GLR and the risk of all-cause mortality (p for overall = 0.028, p for nonlinear = 0.048). Compare with TNM stage, time-dependent ROC curve proved that GLR-based nomogram model performs better in predicting prognosis (the area under curve for 1-, 3-, and 5-years mortality: 0.63, 0.65, and 0.64 vs. 0.76, 0.77, and 0.78, p < 0.001).

CONCLUSION

GLR might be a useful tool in predicting prognosis for patients with oral cancer.

Performance of the Matsumiya scoring system in cervical cancer patients with bone metastasis: an external validation study

BACKGROUND

Accurate prognostic prediction of survival in cervical cancer patients with bone metastasis is important for treatment planning. We aimed to externally validate the Matsumiya scoring system using external patient data.

METHODS

We collected a retrospective cohort of patients with cervical cancer diagnosed with bone metastasis at Chiang Mai University Hospital from 1st January 2007 to 31st December 2016. The Matsumiya score was composed of 5 predictors, including the presence of extraskeletal metastasis, ECOG performance status, history of previous chemo- or radiotherapy, the presence of multiple bone metastasis, and bone metastasis-free interval < 12 months. Harrell's C-statistics and score calibration plots were used to evaluate the score performance. We also reconstructed the development study to estimate apparent performance values for comparison during external validation.

RESULTS

A total of 124 cervical cancer patients with bone metastasis were included in this study. The 13-, 26-, and 52-week survival probabilities in the validation study were 70.1%, 50.5%, and 25.7%, respectively. Several differences were identified between development and validation studies regarding clinical characteristics, case-mix, and predictor-outcome associations. Harrell's C-statistics in the development and validation study were 0.714 and 0.567. The score showed poor agreement between the observed and the predicted survival probabilities in the validation study. Score reweighting and refitting showed only modest improvement in performance.

CONCLUSION

A prognostic scoring system by Matsumiya et al. performed poorly in our cohort of Thai cervical cancer patients with bone metastasis. We suggested that the score should be sufficiently updated before being used.

External Validation of the Skin and UV Neoplasia Transplant Risk Assessment Calculator (SUNTRAC) in a Large European Solid Organ Transplant Recipient Cohort

Importance

The Skin and UV Neoplasia Transplant Risk Assessment Calculator (SUNTRAC) tool has been developed in the US to facilitate the identification of solid organ transplant recipients (SOTRs) at a higher risk of developing skin cancer. However, it has not yet been validated in populations other than the one used for its creation.

Objective

To provide an external validation of the SUNTRAC tool in different SOTR populations.

Design, Setting, and Participants

This retrospective external validation prognostic study used data from a prospectively collected cohort of European SOTRs from transplant centers at teaching hospitals in the Netherlands (1995-2016) and Spain (2011-2021). Participants were screened and followed up at dermatology departments. Data were analyzed from September to October 2021.

Main Outcomes and Measures

The discrimination ability of the SUNTRAC tool was assessed via a competing risk survival analysis, cumulative incidence plots, and Wolbers concordance index. Calibration of the SUNTRAC tool was assessed through comparison of projected skin cancer incidences. Skin cancer diagnoses included squamous cell carcinoma, basal cell carcinoma, melanoma, and Merkel cell carcinoma.

Results

A total of 3421 SOTRs (median age at transplant, 53 [quartile 1: 42; quartile 3: 62] years; 2132 [62.3%] men) were assessed, including 72 Asian patients (2.1%), 137 Black patients (4.0%), 275 Latinx patients (8.0%), 109 Middle Eastern and North African patients (3.2%), and 2828 White patients (82.7%). With a total of 23 213 years of follow-up time, 603 patients developed skin cancer. The SUNTRAC tool classified patients into 4 groups with significantly different risks of developing skin cancer during follow-up. Overall, the relative rate for developing skin cancer estimated using subdistribution hazard ratios (SHRs) and using the low-risk group as the reference group, increased according to the proposed risk group (medium-risk group: SHR, 6.8 [95% CI, 3.8-12.1]; P < .001; high-risk group: SHR, 15.9 [95% CI, 8.9-28.4]; P < .001; very-high-risk group: SHR, 54.8 [95% CI, 29.1-102.9]; P < .001), with a concordance index of 0.72. Actual skin cancer incidences were similar to those predicted by the SUNTRAC tool (5-year skin cancer cumulative incidence for medium-risk group: predicted, 6.2%; observed, 7.0%).

Conclusions and Relevance

The findings of this external validation prognostic study support the use of the SUNTRAC tool in European populations for stratifying SOTRs based on their skin cancer risk and also detecting patients at a high risk of developing skin cancer. This can be helpful in prioritizing and providing better screening and surveillance for these patients.

Impact of image filtering and assessment of volume-confounding effects on CT radiomic features and derived survival models in non-small cell lung cancer

Background

No evidence supports the choice of specific imaging filtering methodologies in radiomics. As the volume of the primary tumor is a well-recognized prognosticator, our purpose is to assess how filtering may impact the feature/volume dependency in computed tomography (CT) images of non-small cell lung cancer (NSCLC), and if such impact translates into differences in the performance of survival modeling. The role of lesion volume in model performances was also considered and discussed.

Methods

Four-hundred seventeen CT images NSCLC patients were retrieved from the NSCLC-Radiomics public repository. Pre-processing and features extraction were implemented using Pyradiomics v3.0.1. Features showing high correlation with volume across original and filtered images were excluded. Cox proportional hazards (PH) with least absolute shrinkage and selection operator (LASSO) regularization and CatBoost models were built with and without volume, and their concordance (C-) indices were compared using Wilcoxon signed-ranked test. The Mann Whitney U test was used to assess model performances after stratification into two groups based on low- and high-volume lesions.

Results

Radiomic models significantly outperformed models built on only clinical variables and volume. However, the exclusion/inclusion of volume did not generally alter the performances of radiomic models. Overall, performances were not substantially affected by the choice of either imaging filter (overall C-index 0.539-0.590 for Cox PH and 0.589-0.612 for CatBoost). The separation of patients with high-volume lesions resulted in significantly better performances in 2/10 and 7/10 cases for Cox PH and CatBoost models, respectively. Both low- and high-volume models performed significantly better with the inclusion of radiomic features (P<0.0001), but the improvement was largest in the high-volume group (+10.2% against +8.7% improvement for CatBoost models and +10.0% against +5.4% in Cox PH models).

Conclusions

Radiomic features complement well-known prognostic factors such as volume, but their volume-dependency is high and should be managed with vigilance. The informative content of radiomic features may be diminished in small lesion volumes, which could limit the applicability of radiomics in early-stage NSCLC, where tumors tend to be small. Our results also suggest an advantage of CatBoost models over the Cox PH models.

Machine learning models for identifying predictors of clinical outcomes with first-line immune checkpoint inhibitor therapy in advanced non-small cell lung cancer

Immune checkpoint inhibitors (ICIs) are standard-of-care as first-line (1L) therapy for advanced non-small cell lung cancer (aNSCLC) without actionable oncogenic driver mutations. While clinical trials demonstrated benefits of ICIs over chemotherapy, variation in outcomes across patients has been observed and trial populations may not be representative of clinical practice. Predictive models can help understand heterogeneity of treatment effects, identify predictors of meaningful clinical outcomes, and may inform treatment decisions. We applied machine learning (ML)-based survival models to a real-world cohort of patients with aNSCLC who received 1L ICI therapy extracted from a US-based electronic health record database. Model performance was evaluated using metrics including concordance index (c-index), and we used explainability techniques to identify significant predictors of overall survival (OS) and progression-free survival (PFS). The ML model achieved c-indices of 0.672 and 0.612 for OS and PFS, respectively, and Kaplan-Meier survival curves showed significant differences between low- and high-risk groups for OS and PFS (both log-rank test p < 0.0001). Identified predictors were mostly consistent with the published literature and/or clinical expectations and largely overlapped for OS and PFS; Eastern Cooperative Oncology Group performance status, programmed cell death-ligand 1 expression levels, and serum albumin were among the top 5 predictors for both outcomes. Prospective and independent data set evaluation is required to confirm these results.

Prognostic value of patient-reported outcome measures (PROMs) in adults with non-small cell Lung Cancer: a scoping review

Background

There is growing interest in the collection and use of patient-reported outcome measures (PROMs) to support clinical decision making in patients with non-small cell lung cancer (NSCLC). However, an overview of research into the prognostic value of PROMs is currently lacking.

Aim

To explore to what extent, how, and how robustly the value of PROMs for prognostic prediction has been investigated in adults diagnosed with NSCLC.

Methods

We systematically searched Medline, Embase, CINAHL Plus and Scopus for English-language articles published from 2011 to 2021 that report prognostic factor study, prognostic model development or validation study. Example data charting forms from the Cochrane Prognosis Methods Group guided our data charting on study characteristics, PROMs as predictors, predicted outcomes, and statistical methods. Two reviewers independently charted the data and critically appraised studies using the QUality In Prognosis Studies (QUIPS) tool for prognostic factor studies, and the risk of bias assessment section of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) for prognostic model studies.

Results

Our search yielded 2,769 unique titles of which we included 31 studies, reporting the results of 33 unique analyses and models. Out of the 17 PROMs used for prediction, the EORTC QLQ-C30 was most frequently used (16/33); 12/33 analyses used PROM subdomain scores instead of the overall scores. PROMs data was mostly collected at baseline (24/33) and predominantly used to predict survival (32/33) but seldom other clinical outcomes (1/33). Almost all prognostic factor studies (26/27) had moderate to high risk of bias and all four prognostic model development studies had high risk of bias.

Conclusion

There is an emerging body of research into the value of PROMs as a prognostic factor for survival in people with NSCLC but the methodological quality of this research is poor with significant bias. This warrants more robust studies into the prognostic value of PROMs, in particular for predicting outcomes other than survival. This will enable further development of PROM-based prediction models to support clinical decision making in NSCLC.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-022-10151-z.

Development and validation of a clinical prediction model of acute kidney injury in intensive care unit patients at a rural tertiary teaching hospital in South Africa: a study protocol

INTRODUCTION

Acute kidney injury (AKI) is a decline in renal function lasting hours to days. The rising global incidence of AKI, and associated costs of renal replacement therapy, is a public health priority. With the only therapeutic option being supportive therapy, prevention and early diagnosis will facilitate timely interventions to prevent progression to chronic kidney disease. While many factors have been identified as predictive of AKI, none have shown adequate sensitivity or specificity on their own. Many tools have been developed in developed-country cohorts with higher rates of non-communicable disease, and few have been validated and practically implemented. The development and validation of a predictive tool incorporating clinical, biochemical and imaging parameters, as well as quantification of their impact on the development of AKI, should make timely and improved prediction of AKI possible. This study is positioned to develop and validate an AKI prediction tool in critically ill patients at a rural tertiary hospital in South Africa.

METHOD AND ANALYSIS

Critically ill patients will be followed from admission until discharge or death. Risk factors for AKI will be identified and their impact quantified using statistical modelling. Internal validation of the developed model will be done on separate patients admitted at a different time. Furthermore, patients developing AKI will be monitored for 3 months to assess renal recovery and quality of life. The study will also explore the utility of endothelial monitoring using the biomarker Syndecan-1 and capillary leak measurements in predicting persistent AKI.

ETHICS AND DISSEMINATION

The study has been approved by the Walter Sisulu University Faculty of Health Science Research Ethics and Biosafety Committee (WSU No. 005/2021), and the Eastern Cape Department of Health Research Ethics (approval number: EC 202103006). The findings will be shared with facility management, and presented at relevant conferences and seminars.

Development and validation of a multivariable mortality risk prediction model for COPD in primary care

Risk stratification of chronic obstructive pulmonary disease (COPD) patients is important to enable targeted management. Existing disease severity classification systems, such as GOLD staging, do not take co-morbidities into account despite their high prevalence in COPD patients. We sought to develop and validate a prognostic model to predict 10-year mortality in patients with diagnosed COPD. We constructed a longitudinal cohort of 37,485 COPD patients (149,196 person-years) from a UK-wide primary care database. The risk factors included in the model pertained to demographic and behavioural characteristics, co-morbidities, and COPD severity. The outcome of interest was all-cause mortality. We fitted an extended Cox-regression model to estimate hazard ratios (HR) with 95% confidence intervals (CI), used machine learning-based data modelling approaches including k-fold cross-validation to validate the prognostic model, and assessed model fitting and discrimination. The inter-quartile ranges of the three metrics on the validation set suggested good performance: 0.90–1.06 for model fit, 0.80–0.83 for Harrel’s c-index, and 0.40–0.46 for Royston and Saurebrei’s \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_D^2$$\end{document}RD2 with a strong overlap of these metrics on the training dataset. According to the validated prognostic model, the two most important risk factors of mortality were heart failure (HR 1.92; 95% CI 1.87–1.96) and current smoking (HR 1.68; 95% CI 1.66–1.71). We have developed and validated a national, population-based prognostic model to predict 10-year mortality of patients diagnosed with COPD. This model could be used to detect high-risk patients and modify risk factors such as optimising heart failure management and offering effective smoking cessation interventions.

Applicability of predictive models for 30-day unplanned hospital readmission risk in paediatrics: a systematic review

OBJECTIVES

To summarise multivariable predictive models for 30-day unplanned hospital readmissions (UHRs) in paediatrics, describe their performance and completeness in reporting, and determine their potential for application in practice.

DESIGN

Systematic review.

DATA SOURCE

CINAHL, Embase and PubMed up to 7 October 2021.

ELIGIBILITY CRITERIA

English or German language studies aiming to develop or validate a multivariable predictive model for 30-day paediatric UHRs related to all-cause, surgical conditions or general medical conditions were included.

DATA EXTRACTION AND SYNTHESIS

Study characteristics, risk factors significant for predicting readmissions and information about performance measures (eg, c-statistic) were extracted. Reporting quality was addressed by the 'Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis' (TRIPOD) adherence form. The study quality was assessed by applying six domains of potential biases. Due to expected heterogeneity among the studies, the data were qualitatively synthesised.

RESULTS

Based on 28 studies, 37 predictive models were identified, which could potentially be used for determining individual 30-day UHR risk in paediatrics. The number of study participants ranged from 190 children to 1.4 million encounters. The two most common significant risk factors were comorbidity and (postoperative) length of stay. 23 models showed a c-statistic above 0.7 and are primarily applicable at discharge. The median TRIPOD adherence of the models was 59% (P₂₅-P₇₅, 55%-69%), ranging from a minimum of 33% to a maximum of 81%. Overall, the quality of many studies was moderate to low in all six domains.

CONCLUSION

Predictive models may be useful in identifying paediatric patients at increased risk of readmission. To support the application of predictive models, more attention should be placed on completeness in reporting, particularly for those items that may be relevant for implementation in practice.

Development and Validation of a Prediction Model for Early Diagnosis of SCN1A-Related Epilepsies

BACKGROUND AND OBJECTIVES

Pathogenic variants in the neuronal sodium channel α1 subunit gene (SCN1A) are the most frequent monogenic cause of epilepsy. Phenotypes comprise a wide clinical spectrum, including severe childhood epilepsy; Dravet syndrome, characterized by drug-resistant seizures, intellectual disability, and high mortality; and the milder genetic epilepsy with febrile seizures plus (GEFS+), characterized by normal cognition. Early recognition of a child's risk for developing Dravet syndrome vs GEFS+ is key for implementing disease-modifying therapies when available before cognitive impairment emerges. Our objective was to develop and validate a prediction model using clinical and genetic biomarkers for early diagnosis of SCN1A-related epilepsies.

METHODS

We performed a retrospective multicenter cohort study comprising data from patients with SCN1A-positive Dravet syndrome and patients with GEFS+ consecutively referred for genetic testing (March 2001-June 2020) including age at seizure onset and a newly developed SCN1A genetic score. A training cohort was used to develop multiple prediction models that were validated using 2 independent blinded cohorts. Primary outcome was the discriminative accuracy of the model predicting Dravet syndrome vs other GEFS+ phenotypes.

RESULTS

A total of 1,018 participants were included. The frequency of Dravet syndrome was 616/743 (83%) in the training cohort, 147/203 (72%) in validation cohort 1, and 60/72 (83%) in validation cohort 2. A high SCN1A genetic score (133.4 [SD 78.5] vs 52.0 [SD 57.5]; p < 0.001) and young age at onset (6.0 [SD 3.0] vs 14.8 [SD 11.8] months; p < 0.001) were each associated with Dravet syndrome vs GEFS+. A combined SCN1A genetic score and seizure onset model separated Dravet syndrome from GEFS+ more effectively (area under the curve [AUC] 0.89 [95% CI 0.86-0.92]) and outperformed all other models (AUC 0.79-0.85; p < 0.001). Model performance was replicated in both validation cohorts 1 (AUC 0.94 [95% CI 0.91-0.97]) and 2 (AUC 0.92 [95% CI 0.82-1.00]).

DISCUSSION

The prediction model allows objective estimation at disease onset whether a child will develop Dravet syndrome vs GEFS+, assisting clinicians with prognostic counseling and decisions on early institution of precision therapies (http://scn1a-prediction-model.broadinstitute.org/).

CLASSIFICATION OF EVIDENCE

This study provides Class II evidence that a combined SCN1A genetic score and seizure onset model distinguishes Dravet syndrome from other GEFS+ phenotypes.

Updated European Association of Urology (EAU) Prognostic Factor Risk Groups Overestimate the Risk of Progression in Patients with Non-muscle-invasive Bladder Cancer Treated with Bacillus Calmette-Guérin

BACKGROUND

The 2021 European Association of Urology (EAU) guidelines contain updated prognostic factor risk groups for non-muscle-invasive bladder cancer (NMIBC). These groups are based on the following predictors of progression: tumour stage, grade, number, and size; concomitant carcinoma in situ; and age. However, the groups were derived from datasets excluding patients treated with bacillus Calmette-Guérin (BCG).

OBJECTIVE

To determine the validity of the updated EAU prognostic factor risk groups in patients with NMIBC treated with BCG.

DESIGN, SETTING, AND PARTICIPANTS

We reviewed patients treated with BCG at our institution between 2000 and 2018. Patients were analysed according to the receipt of "at least induction" and "adequate" BCG (as defined by the US Food and Drug Administration). Risk groups were assigned according to the 2021 EAU NMIBC risk calculator (https://nmibc.net/).

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The Kaplan-Meier method was used to estimate the risks of progression at 1 and 5 yr. Probabilities of progression obtained with the updated prognostic factor risk groups in our series were compared with those reported by the EAU. Discrimination was assessed using the concordance index (c-index).

RESULTS AND LIMITATIONS

A total of 529 patients received at least induction BCG with a median follow-up of 47.3 mo (interquartile range 25.3-86.9). Of these patients, 494 received adequate BCG. We found lower progression rates at 1 yr in the very-high-risk group patients receiving at least induction (6.9%) and adequate BCG (4.0%) versus 16.0% for the EAU predicted rates. Additionally, progression rates were also lower at 5 yr in the high-risk group-7.4% for at least induction and 5.3% for adequate BCG versus 9.6% for EAU predicted rates; the rates in the very-high-risk group were as follows: 16.7% for at least induction and 14.9% for adequate BCG versus 40.0% for EAU predicted rates. The c-index in our series was lower than that reported by the EAU (0.63 vs 0.80). Of interest, our multivariable analysis identified grade, stage, and age (p < 0.02) to be the predictors of progression after BCG therapy.

CONCLUSIONS

While the 2021 EAU prognostic factor risk groups successfully stratified progression risks in our cohort, treatment with BCG reduced their discriminative ability. Furthermore, the groups overestimate progression risks in BCG-treated patients. These findings should be used in conjunction with the updated risk groups to counsel patients with higher-risk NMIBC about their risk of progression with and without BCG.

PATIENT SUMMARY

Although the updated European Association of Urology prognostic factor risk groups are able to stratify patients with non-muscle-invasive bladder cancer according to their risk of progression to muscle-invasive bladder cancer, this risk is overestimated in patients treated with bacillus Calmette-Guérin (BCG).

Liver resection is associated with good outcomes for hepatocellular carcinoma patients beyond the Barcelona Clinic Liver Cancer criteria: A multicenter study with the Hiroshima Surgical study group of Clinical Oncology

BACKGROUND

Liver resection for hepatocellular carcinoma beyond the Barcelona Clinic Liver Cancer criteria remains controversial. Strict candidate selection is crucial to achieve optimal results in this population. This study explored postoperative outcomes and developed a preoperative predictive formula to identify patients most likely to benefit from liver resection.

METHODS

In total, 382 patients who underwent liver resection for hepatocellular carcinoma beyond the Barcelona Clinic Liver Cancer resection criteria between 2000 and 2017 were identified from a multicenter database with the Hiroshima Surgical study group of Clinical Oncology. An overall survival prediction model was developed, and patients were classified by risk status.

RESULTS

The 5-year overall survival after curative resection was 50.0%. Overall survival multivariate analysis identified that a high a-fetoprotein level, macrovascular invasion, and high total tumor burden were independent prognostic risk factors; these factors were used to formulate risk scores. Patients were divided into low-, moderate-, and high-risk groups; the 5-year overall survival was 65.7%, 49.5%, and 17.0% (P < .001), and the 5-year recurrence-free survival was 31.3%, 26.2%, and 0%, respectively (P < .001). The model performance was good (C-index, 0.76). Both the early and extrahepatic recurrence increased with higher risk score.

CONCLUSION

The prognosis of patients with hepatocellular carcinoma beyond the Barcelona Clinic Liver Cancer resection criteria depended on a high a-fetoprotein level, macrovascular invasion, and high total tumor burden, and risk scores based on these factors stratified the prognoses. Liver resection should be considered in patients with hepatocellular carcinoma beyond the Barcelona Clinic Liver Cancer criteria with a low or moderate-risk score.

Composite Score: prognostic tool to predict survival in patients undergoing surgery for colorectal liver metastases

BACKGROUND

Several existing scoring systems predict survival of patients with colorectal liver metastases. Many lack validation, rely on old clinical data, and have been found to be less accurate since the introduction of chemotherapy. This study aimed to construct and validate a clinically relevant preoperative prognostic model for patients with colorectal liver metastases.

METHODS

A predictive model with data available before surgery was developed. Survival was analysed by Cox regression analysis, and the quality of the model was assessed using discrimination and calibration. The model was validated using multifold cross-validation.

RESULTS

The model included 1212 consecutive patients who underwent liver resection for colorectal liver metastases between 2005 and 2015. Prognostic factors for survival included advanced age, raised C-reactive protein level, hypoalbuminaemia, extended liver resection, larger number of metastases, and midgut origin of the primary tumour. A Composite Score was developed based on the prognostic variables. Patients were classified into those at low, medium, and high risk. Survival differences between the groups were significant; median overall survival was 87.4 months in the low-risk group, 50.1 months in the medium-risk group, and 22.6 months in the high-risk group. The discriminative performance, assessed by the concordance index, was 0.71, 0.67, and 0.67 respectively at 1, 3, and 5 years. Calibration, assessed graphically, was close to perfect. A multifold cross-validation of the model confirmed its internal validity (C-index 0.63 versus 0.62).

CONCLUSION

The Composite Score categorizes patients into risk strata, and may help identify patients who have a poor prognosis, for whom surgery is questionable.

Improving the investigative approach to polycythaemia vera: a critical assessment of current evidence and vision for the future

Polycythaemia vera is a challenging disease to study given its low prevalence and prolonged time-to-event for important clinical endpoints such as thrombosis, progression, and mortality. Although researchers in this space often rise to meet these challenges, there is considerable room for improvement in the analysis of retrospective data, the development of risk-stratification tools, and the design of randomised controlled trials. In this Viewpoint, we review the evidence behind the contemporary approach to risk stratification and treatment of polycythaemia vera. Frameworks for using data more efficiently, constructing more nuanced prognostic models, and overcoming challenges in clinical trial design are discussed.

Why did European Radiology reject my radiomic biomarker paper? How to correctly evaluate imaging biomarkers in a clinical setting

This review explains in simple terms, accessible to the non-statistician, general principles regarding the correct research methods to develop and then evaluate imaging biomarkers in a clinical setting, including radiomic biomarkers. The distinction between diagnostic and prognostic biomarkers is made and emphasis placed on the need to assess clinical utility within the context of a multivariable model. Such models should not be restricted to imaging biomarkers and must include relevant disease and patient characteristics likely to be clinically useful. Biomarker utility is based on whether its addition to the basic clinical model improves diagnosis or prediction. Approaches to both model development and evaluation are explained and the need for adequate amounts of representative data stressed so as to avoid underpowering and overfitting. Advice is provided regarding how to report the research correctly. KEY POINTS: • Imaging biomarker research is common but methodological errors are encountered frequently that may mean the research is not clinically useful. • The clinical utility of imaging biomarkers is best assessed by their additive effect on multivariable models based on clinical factors known to be important. • The data used to develop such models should be sufficient for the number of variables investigated and the model should be evaluated, preferably using data unrelated to development.

Improving patient selection for immuno-oncology phase 1 trials: External validation of six prognostic scores in a French Cancer Center

We compared the performance of six prognostic scores (Royal Marsden Hospital, MDACC: MD Anderson Clinical Center and MDACC + NLR: neutrophil-to-lymphocyte ratio, MD Anderson - immune checkpoint inhibitors (MDA-ICI), GRIm: Gustave Roussy Immune Score and LIPI: Lung Immune Prognostic Index) in predicting overall survival (OS) in phase I trial patients treated with immune checkpoint inhibitors (ICI). Medical records of patients with advanced solid tumors enrolled in ICI phase I trials between 2015 and 2018 at Institut Universitaire du Cancer de Toulouse-Oncopole were reviewed. The performance of prognostic scores on OS was compared using different criteria. A total of 259 patients were included. Median age was 63 years (range: 18-83). Main primary cancers were melanoma (19%), head and neck (16%), lung (13%) and bladder (10%). With a median follow-up of 15 months (95% confidence interval [CI] = [11.6;17.5]), median OS was 12.5 months (95% CI = [10.3;16.0]). All scores were associated with OS. The MDACC, LIPI and GRIm scores performed better than the others. Concordance of risk group assignment between the scoring systems was poor. According to our results, the MDACC, GRIm and LIPI scores better suited to ICI phase I settings. Adequate scoring would allow better patient selection in early ICI trials, especially during the critical period of dose escalation, and in proof-of-concept expansion cohorts.

Defining an Ultra-Low Risk Group in Asymptomatic IgM Monoclonal Gammopathy

Simple Summary

Patients with asymptomatic IgM monoclonal gammopathies include IgM monoclonal gammopathy of undetermined significance (IgM MGUS) and smoldering Waldenström macroglobulinemia (SWM), all with some risk of progression to symptomatic Waldenström macroglobulinemia, amyloidosis, or other lymphoproliferative disorder. Due to their low incidence, few studies have focused on the risk of progression, with SWM being the most studied. As both are recognized clinical-pathological entities that share similar clonal and phenotypical features, we focus on defining new biomarkers of progression in this population with long follow-up.

Abstract

We analyzed 171 patients with asymptomatic IgM monoclonal gammopathies (64 with IgM monoclonal gammopathy of undetermined significance—MGUS and 107 with smoldering Waldenström macroglobulinemia - SWM) who had a bone marrow (BM) evaluation performed at diagnosis. Abnormal free-light chain ratio (53% vs. 31%) and MYD88 mutation prevalence (66% vs. 30%) were higher in patients with SWM. No other differences were found among groups. With a median follow-up of 4.3 years, 14 patients progressed to Waldenström macroglobulinemia, 1 to amyloidosis, and 28 died without progression. The MYD88 mutation was found in 53% of patients (available in 160 patients). Multivariate analysis showed that immunoparesis (subhazard ratio—SHR 10.2, 95% confidence interval—CI: 4.2–24.8; p < 0.001) and BM lymphoplasmacytic infiltration ≥ 20% (SHR: 6, 95% CI: 1.6–22.1; p = 0.007) were associated with higher risk of progression. We developed a risk model based on these two risk factors. In the absence of both variables, an ultra-low risk group was identified (SHR 0.1, 95% CI 0.02–0.5; p = 0.004), with 3% and 6% of cumulative incidence of progression at 10 and 20 years, respectively. Bootstrap analysis confirmed the reproducibility of these results. This study finds immunoparesis and BM infiltration as biomarkers of progression as well as a low-risk group of progression in asymptomatic IgM monoclonal gammopathies.

Explainable machine learning can outperform Cox regression predictions and provide insights in breast cancer survival

Cox Proportional Hazards (CPH) analysis is the standard for survival analysis in oncology. Recently, several machine learning (ML) techniques have been adapted for this task. Although they have shown to yield results at least as good as classical methods, they are often disregarded because of their lack of transparency and little to no explainability, which are key for their adoption in clinical settings. In this paper, we used data from the Netherlands Cancer Registry of 36,658 non-metastatic breast cancer patients to compare the performance of CPH with ML techniques (Random Survival Forests, Survival Support Vector Machines, and Extreme Gradient Boosting [XGB]) in predicting survival using the [Formula: see text]-index. We demonstrated that in our dataset, ML-based models can perform at least as good as the classical CPH regression ([Formula: see text]-index [Formula: see text]), and in the case of XGB even better ([Formula: see text]-index [Formula: see text]). Furthermore, we used Shapley Additive Explanation (SHAP) values to explain the models' predictions. We concluded that the difference in performance can be attributed to XGB's ability to model nonlinearities and complex interactions. We also investigated the impact of specific features on the models' predictions as well as their corresponding insights. Lastly, we showed that explainable ML can generate explicit knowledge of how models make their predictions, which is crucial in increasing the trust and adoption of innovative ML techniques in oncology and healthcare overall.

Doug Altman: Driving critical appraisal and improvements in the quality of methodological and medical research

Doug Altman was a visionary leader and one of the most influential medical statisticians of the last 40 years. Based on a presentation in the "Invited session in memory of Doug Altman" at the 40th Annual Conference of the International Society for Clinical Biostatistics (ISCB) in Leuven, Belgium and our long-standing collaborations with Doug, we discuss his contributions to regression modeling, reporting, prognosis research, as well as some more general issues while acknowledging that we cannot cover the whole spectrum of Doug's considerable methodological output. His statement "To maximize the benefit to society, you need to not just do research but do it well" should be a driver for all researchers. To improve current and future research, we aim to summarize Doug's messages for these three topics.

Design, methods, and reporting of impact studies of cardiovascular clinical prediction rules are suboptimal: a systematic review

OBJECTIVES

To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs).

STUDY DESIGN AND SETTING

We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and nonrandomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we assessed the quality of methods and reporting. We compared the quality of methods and reporting between impact and matched control studies.

RESULTS

We found 110 impact studies of cardiovascular CPRs. Of these, 65 (59.1%) used inappropriate designs. Of 45 impact studies with appropriate design, 31 (68.9%) had substantial risk of bias. Mean number of reporting domains that impact studies with appropriate study design adhered to was 10.2 of 21 domains (95% confidence interval, 9.3 and 11.1). The quality of methods and reporting was not clearly different between impact and matched control studies.

CONCLUSION

We found most impact studies either used inappropriate study design, had substantial risk of bias, or poorly complied with reporting guidelines. This appears to be a common feature of complex interventions. Users of CPRs should critically evaluate evidence showing the effectiveness of CPRs.

RNA-Sequencing Analysis Pipeline for Prognostic Marker Identification in Cancer

Sequencing analysis finds many applications in various fields of biology from comparative genomics to clinical research. Recent studies, using high-throughput sequencing method, has generated terabytes of data. It is challenging to interpret and draw a meaningful conclusion without the proper understanding of various steps involved in the analysis of such data. This chapter deals with the pipeline to be followed to process the raw RNA sequencing (RNA-Seq) reads, align, assemble, and quantify them in order to draw significant clinical conclusions from them.

Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

OBJECTIVE

Developing clinical prediction models (CPMs) on data of sufficient sample size is critical to help minimize overfitting. Using prostate cancer as a clinical exemplar, we aimed to investigate to what extent existing CPMs adhere to recent formal sample size criteria, or historic rules of thumb of events per predictor parameter (EPP)≥10.

STUDY DESIGN AND SETTING

A systematic review to identify CPMs related to prostate cancer, which provided enough information to calculate minimum sample size. We compared the reported sample size of each CPM against the traditional 10 EPP rule of thumb and formal sample size criteria.

RESULTS

About 211 CPMs were included. Three of the studies justified the sample size used, mostly using EPP rules of thumb. Overall, 69% of the CPMs were derived on sample sizes that surpassed the traditional EPP≥10 rule of thumb, but only 48% surpassed recent formal sample size criteria. For most CPMs, the required sample size based on formal criteria was higher than the sample sizes to surpass 10 EPP.

CONCLUSION

Few of the CPMs included in this study justified their sample size, with most justifications being based on EPP. This study shows that, in real-world data sets, adhering to the classic EPP rules of thumb is insufficient to adhere to recent formal sample size criteria.

Survival prediction of glioblastoma patients-are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential

Glioblastoma is associated with a poor prognosis. Even though survival statistics are well-described at the population level, it remains challenging to predict the prognosis of an individual patient despite the increasing number of prognostic models. The aim of this study is to systematically review the literature on prognostic modeling in glioblastoma patients. A systematic literature search was performed to identify all relevant studies that developed a prognostic model for predicting overall survival in glioblastoma patients following the PRISMA guidelines. Participants, type of input, algorithm type, validation, and testing procedures were reviewed per prognostic model. Among 595 citations, 27 studies were included for qualitative review. The included studies developed and evaluated a total of 59 models, of which only seven were externally validated in a different patient cohort. The predictive performance among these studies varied widely according to the AUC (0.58-0.98), accuracy (0.69-0.98), and C-index (0.66-0.70). Three studies deployed their model as an online prediction tool, all of which were based on a statistical algorithm. The increasing performance of survival prediction models will aid personalized clinical decision-making in glioblastoma patients. The scientific realm is gravitating towards the use of machine learning models developed on high-dimensional data, often with promising results. However, none of these models has been implemented into clinical care. To facilitate the clinical implementation of high-performing survival prediction models, future efforts should focus on harmonizing data acquisition methods, improving model interpretability, and externally validating these models in multicentered, prospective fashion.

Risk Prediction Models for Melanoma: A Systematic Review on the Heterogeneity in Model Development and Validation

The rising incidence of cutaneous melanoma over the past few decades has prompted substantial efforts to develop risk prediction models identifying people at high risk of developing melanoma to facilitate targeted screening programs. We review these models, regarding study characteristics, differences in risk factor selection and assessment, evaluation, and validation methods. Our systematic literature search revealed 40 studies comprising 46 different risk prediction models eligible for the review. Altogether, 35 different risk factors were part of the models with nevi being the most common one (n = 35, 78%); little consistency in other risk factors was observed. Results of an internal validation were reported for less than half of the studies (n = 18, 45%), and only 6 performed external validation. In terms of model performance, 29 studies assessed the discriminative ability of their models; other performance measures, e.g., regarding calibration or clinical usefulness, were rarely reported. Due to the substantial heterogeneity in risk factor selection and assessment as well as methodologic aspects of model development, direct comparisons between models are hardly possible. Uniform methodologic standards for the development and validation of risk prediction models for melanoma and reporting standards for the accompanying publications are necessary and need to be obligatory for that reason.

TRIPOD statement: a preliminary pre-post analysis of reporting and methods of prediction models

OBJECTIVES

To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

METHODS

In the seven general medicine journals with the highest impact factor, we compared the completeness of the reporting and the quality of the methodology of prediction model studies published between 2012 and 2014 (pre-TRIPOD) with studies published between 2016 and 2017 (post-TRIPOD). For articles published in the post-TRIPOD period, we examined whether there was improved reporting for articles (1) citing the TRIPOD statement, and (2) published in journals that published the TRIPOD statement.

RESULTS

A total of 70 articles was included (pre-TRIPOD: 32, post-TRIPOD: 38). No improvement was seen for the overall percentage of reported items after the publication of the TRIPOD statement (pre-TRIPOD 74%, post-TRIPOD 76%, 95% CI of absolute difference: -4% to 7%). For the individual TRIPOD items, an improvement was seen for 16 (44%) items, while 3 (8%) items showed no improvement and 17 (47%) items showed a deterioration. Post-TRIPOD, there was no improved reporting for articles citing the TRIPOD statement, nor for articles published in journals that published the TRIPOD statement. The methodological quality improved in the post-TRIPOD period. More models were externally validated in the same article (absolute difference 8%, post-TRIPOD: 39%), used measures of calibration (21%, post-TRIPOD: 87%) and discrimination (9%, post-TRIPOD: 100%), and used multiple imputation for handling missing data (12%, post-TRIPOD: 50%).

CONCLUSIONS

Since the publication of the TRIPOD statement, some reporting and methodological aspects have improved. Prediction models are still often poorly developed and validated and many aspects remain poorly reported, hindering optimal clinical application of these models. Long-term effects of the TRIPOD statement publication should be evaluated in future studies.

How Can Gene-Expression Information Improve Prognostic Prediction in TCGA Cancers: An Empirical Comparison Study on Regularization and Mixed Cox Models

Background

Previous cancer prognostic prediction models often consider only the most important transcriptomic expressions, and their power is limited. It is unknown whether prediction power can be further improved when additional transcriptomic information is incorporated.

Methods

To integrate transcriptomes, four models are compared based on 32 types of cancer in the Cancer Genome Atlas, including the general Cox model with only clinical covariates, the Cox model with a lasso penalty (coxlasso), the Cox model with an elastic net penalty (coxenet), and the mixed-effects Cox model (coxlmm). Furthermore, we partition the survival variance into the relative contribution of clinical and transcriptomic components within the framework of coxlmm. Finally, the influence of different numbers of genes was evaluated in the context of coxlmm.

Results

Compared with the clinical covariates–only Cox model, the average prediction gain was 2.4% for coxlasso, 4.2% for coxenet, and 7.2% for coxlmm across 16 low-censored cancers; a significant elevation of prediction power was observed for SARC, SKCM, LGG, PAAD, and HNSC. Similar findings were observed for all 32 cancers with the average prediction gain of 2.7, 3.8, and 5.8% for coxlasso, coxenet, and coxlmm. Coxlmm always had comparable or better prediction performance relative to coxlasso and coxenet with an average of 2.8% prediction improvement across the 16 low-censored cancers. In addition, it is shown that the predictive accuracy of coxlmm generally increases with the number of genes included. The survival variance partition analysis demonstrates that the transcriptomic contribution was higher for some cancers (e.g., LGG, CESC, PAAD, SKCM, and SARC) and lower for others (e.g., BRCA, COAD, KIRC, and STAD).

Conclusion

This study demonstrates that the integration of transcriptomic information can substantially improve prognostic prediction accuracy, but the prediction performance is cancer-specific and varies across cancer types. It further reveals that gene expression exhibits distinct contributions to survival variation across cancers.

Temporal recalibration for improving prognostic model development and risk predictions in settings where survival is improving over time

BACKGROUND

Prognostic models are typically developed in studies covering long time periods. However, if more recent years have seen improvements in survival, then using the full dataset may lead to out-of-date survival predictions. Period analysis addresses this by developing the model in a subset of the data from a recent time window, but results in a reduction of sample size.

METHODS

We propose a new approach, called temporal recalibration, to combine the advantages of period analysis and full cohort analysis. This approach develops a model in the entire dataset and then recalibrates the baseline survival using a period analysis sample. The approaches are demonstrated utilizing a prognostic model in colon cancer built using both Cox proportional hazards and flexible parametric survival models with data from 1996-2005 from the Surveillance, Epidemiology, and End Results (SEER) Program database. Comparison of model predictions with observed survival estimates were made for new patients subsequently diagnosed in 2006 and followed-up until 2015.

RESULTS

Period analysis and temporal recalibration provided more up-to-date survival predictions that more closely matched observed survival in subsequent data than the standard full cohort models. In addition, temporal recalibration provided more precise estimates of predictor effects.

CONCLUSION

Prognostic models are typically developed using a full cohort analysis that can result in out-of-date long-term survival estimates when survival has improved in recent years. Temporal recalibration is a simple method to address this, which can be used when developing and updating prognostic models to ensure survival predictions are more closely calibrated with the observed survival of individuals diagnosed subsequently.

Establishment and evaluation of a multicenter collaborative prediction model construction framework supporting model generalization and continuous improvement: A pilot study

BACKGROUND AND OBJECTIVE

In recent years, an increasing number of clinical prediction models have been developed to serve clinical care. Establishing a data-driven prediction model based on large-scale electronic health record (EHR) data can provide a more empirical basis for clinical decision making. However, research on model generalization and continuous improvement is insufficiently focused, which also hinders the application and evaluation of prediction models in real clinical environments. Therefore, this study proposes a multicenter collaborative prediction model construction framework to build a prediction model with greater generalizability and continuous improvement capabilities while preserving patient data security and privacy.

MATERIALS AND METHODS

Based on a multicenter collaborative research network, such as the Observational Health Data Sciences and Informatics (OHDSI), a multicenter collaborative prediction model construction framework is proposed. Based on the idea of multi-source transfer learning, in each source hospital, a base classifier was trained according to the model research setting. Then, in the target hospital with missing calibration data, a prediction model was established through weighted integration of base classifiers from source hospitals based on the smoothness assumption. Moreover, a passive-aggressive online learning algorithm was used for continuous improvement of the prediction model, which can help to maintain a high predictive performance to provide reliable clinical decision-making abilities. To evaluate the proposed prediction model construction framework, a prototype system for colorectal cancer prognosis prediction was developed. To evaluate the performance of models, 70,906 patients were screened, including 70,090 from 5 US hospital-specific datasets and 816 from a Chinese hospital-specific dataset. The area under the receiver operating characteristic curve (AUC) and the estimated calibration index (ECI) were used to evaluate the discrimination and calibration of models.

RESULTS

Regarding the colorectal cancer prognosis prediction in our prototype system, compared with the reference models, our model achieved a better performance in model calibration (ECI = 9.294 [9.146, 9.441]) and a similar ability in model discrimination (AUC = 0.783 [0.780, 0.786]). Furthermore, the online learning process provided in this study can continuously improve the performance of the prediction model when patient data with specified labels arrive (the AUC value increased from 0.709 to 0.715 and the ECI value decreased from 13.013 to 9.634 after 650 patient instances with specified labels from the Chinese hospital arrived), enabling the prediction model to maintain a good predictive performance during clinical application.

CONCLUSIONS

This study proposes and evaluates a multicenter collaborative prediction model construction framework that can support the construction of prediction models with better generalizability and continuous improvement capabilities without the need to aggregate multicenter patient-level data.

Prognostic Models for Predicting Overall Survival in Patients with Primary Gastric Cancer: A Systematic Review

Background

This study was designed to review the methodology and reporting of gastric cancer prognostic models and identify potential problems in model development.

Methods

This systematic review was conducted following the CHARMS checklist. MEDLINE and EMBASE were searched. Information on patient characteristics, methodological details, and models' performance was extracted. Descriptive statistics was used to summarize the methodological and reporting quality.

Results

In total, 101 model developments and 32 external validations were included. The median (range) of training sample size, number of death, and number of final predictors were 360 (29 to 15320), 193 (14 to 9560), and 5 (2 to 53), respectively. Ninety-one models were developed from routine clinical data. Statistical assumptions were reported to be checked in only nine models. Most model developments (94/101) used complete-case analysis. Discrimination and calibration were not reported in 33 and 55 models, respectively. The majority of models (81/101) have never been externally validated. None of the models have been evaluated regarding clinical impact.

Conclusions

Many prognostic models have been developed, but their usefulness in clinical practice remains uncertain due to methodological shortcomings, insufficient reporting, and lack of external validation and impact studies.

Impact

Future research should improve methodological and reporting quality and emphasize more on external validation and impact assessment.

Predicting survival in cancer patients with and without 30-day readmission of an unplanned hospitalization using a deficit accumulation approach

BACKGROUND

For cancer patients with an unplanned hospitalization, estimating survival has been limited. We examined factors predicting survival and investigated the concept of using a deficit-accumulation survival index (DASI) in this population.

METHODS

Data were abstracted from medical records of 145 patients who had an unplanned 30-day readmission between 01/01/16 and 09/30/16. Comparison data were obtained for patients who were admitted as close in time to the date of index admission of a study patient, but who did not experience a readmission within 30 days of their discharge date. Our survival analysis compared those readmitted within 30 days versus those who were not. Scores from 23 medical record elements used in our DASI system categorized patients into low-, moderate-, and high-score groups.

RESULTS

Thirty-day readmission was strongly associated with the survival (adjusted hazard ratio [HR] 2.39; 95% confidence interval [CI], 1.46-3.92). Patients readmitted within 30 days of discharge from index admission had a median survival of 147 days (95% CI, 85-207) versus patients not readmitted who had not reached median survival by the end of the study (P < .0001). DASI was useful in predicting the survival; median survival time was 78 days (95% CI, 61-131) for the high score, 318 days (95% CI, 207-426) for the moderate score, and not reached as of 426 days (95% CI, 251 to undetermined) for the low-score DASI group (P < .0001).

CONCLUSIONS

Patients readmitted within 30 days of an unplanned hospitalization are at higher risk of mortality than those not readmitted. A novel DASI developed from clinical documentation may help to predict survival in this population.

A systematic review of predictive models for recurrence and mortality in patients with tongue cancer

INTRODUCTION

Predictive models must meet clinical/methodological standards to be used in clinical practice. However, no critique of those models relating to mortality/recurrence in tongue cancer has been done bearing in mind the accepted standards.

METHODS

We conducted a systematic review evaluating the methodology and clinical applicability of predictive models for mortality/recurrence in tongue cancer published in MEDLINE and Scopus. For each model, we analysed (domains of CHARMS, Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) the following: source of data, participants, outcome to be predicted, candidate predictors, sample size, missing data, model development, model performance, model evaluation, results and interpretation and discussion.

RESULTS

We found two papers that included eight prediction models, neither of which adhered to the CHARMS recommendations.

CONCLUSION

Given the quality of tongue cancer models, new studies following current consensus are needed to develop predictive tools applicable in clinical practice.

External validation of a prognostic model for mortality among patients with non-small-cell lung cancer using the Veterans Precision Oncology Data Commons

BACKGROUND

There is wide interest in developing prognostic models in non-small-cell lung cancer (NSCLC) due to the heterogeneity of the disease. Models developed at other healthcare institutions may not be directly applicable for patients treated at the Department of Veterans Affairs (VA). External validation of a candidate prognostic model among VA patients would be crucial before it can be implemented to aid clinical decision-making.

METHODS

A prognostic model for mortality developed in the Military Health System (MHS) was applied to data from the VA Precision Oncology Data Repository (VA-PODR), which is available to researchers inside and outside the VA at the Veterans Precision Oncology Data Commons (VPODC). Measures of discrimination and calibration were calculated for the MHS model. The MHS model was also refitted in VA-PODR data using the same risk factors to compare the effect of specific factors and predictive performance when the model is developed using VA data.

RESULTS

Time-dependent AUC of the MHS prognostic model was 0.788, 0.806, 0.780, and 0.779 for predicting survival at 1, 2, 3, and 5 years following diagnosis, respectively. Significant discrepancies were found between predicted and observed rates of survival, particularly for later years. When the model is refit in VA-PODR data, it achieved cross-validated AUCs of 0.739, 0.773, 0.769, and 0.807 at the same time points, and discrepancies between predicted and observed survival were reduced.

CONCLUSIONS

Validation of the MHS prognostic model in VA-PODR demonstrates that its discrimination remains strong when applied to VA patients. Nevertheless, further calibration to VA data may be needed to improve its risk estimation performance. This study highlights the utility of VA-PODR and the VPODC as a national resource for developing analytic tools that are well adapted to the Veteran population.

Predicting Renal Cancer Recurrence: Defining Limitations of Existing Prognostic Models With Prospective Trial-Based Validation

PURPOSE

To validate currently used recurrence prediction models for renal cell carcinoma (RCC) by using prospective data from the ASSURE (ECOG-ACRIN E2805; Adjuvant Sorafenib or Sunitinib for Unfavorable Renal Carcinoma) adjuvant trial.

PATIENTS AND METHODS

Eight RCC recurrence models (University of California at Los Angeles Integrated Staging System [UISS]; Stage, Size, Grade, and Necrosis [SSIGN]; Leibovich; Kattan; Memorial Sloan Kettering Cancer Center [MSKCC]; Yaycioglu; Karakiewicz; and Cindolo) were selected on the basis of their use in clinical practice and clinical trial designs. These models along with the TNM staging system were validated using 1,647 patients with resected localized high-grade or locally advanced disease (≥ pT1b grade 3 and 4/pTanyN1Mo) from the ASSURE cohort. The predictive performance of the model was quantified by assessing its discriminatory and calibration abilities.

RESULTS

Prospective validation of predictive and prognostic models for localized RCC showed a substantial decrease in each of the predictive abilities of the model compared with their original and externally validated discriminatory estimates. Among the models, the SSIGN score performed best (0.688; 95% CI, 0.686 to 0.689), and the UISS model performed worst (0.556; 95% CI, 0.555 to 0.557). Compared with the 2002 TNM staging system (C-index, 0.60), most models only marginally outperformed standard staging. Importantly, all models, including TNM, demonstrated statistically significant variability in their predictive ability over time and were most useful within the first 2 years after diagnosis.

CONCLUSION

In RCC, as in many other solid malignancies, clinicians rely on retrospective prediction tools to guide patient care and clinical trial selection and largely overestimate their predictive abilities. We used prospective collected adjuvant trial data to validate existing RCC prediction models and demonstrate a sharp decrease in the predictive ability of all models compared with their previous retrospective validations. Accordingly, we recommend prospective validation of any predictive model before implementing it into clinical practice and clinical trial design.

External validation of a modified trauma and injury severity score model in major trauma injury

BACKGROUND

The establishment of an accurate prognostic model in major trauma patients is important mainly because this group of patients will benefit the most. Clinical prediction models must be validated internally and externally on a regular basis to ensure the prediction is accurate and current. This study aims to externally validate two prediction models, the Trauma and Injury Severity Score model developed using the Major Trauma Outcome Study in North America (MTOS-TRISS model), and the NTrD-TRISS model, which is a refined MTOS-TRISS model with coefficients derived from the Malaysian National Trauma Database (NTrD), by regarding mortality as the outcome measurement.

METHOD

This retrospective study included patients with major trauma injuries reported to a trauma centre of Hospital Sultanah Aminah over a 6-year period from 2011 and 2017. Model validation was examined using the measures of discrimination and calibration. Discrimination was assessed using the area under the receiver operating characteristic curve (AUC) and 95% confidence interval (CI). The Hosmer-Lemeshow (H-L) goodness-of-fit test was used to examine calibration capabilities. The predictive validity of both MTOS-TRISS and NTrD-TRISS models were further evaluated by incorporating parameters such as the New Injury Severity Scale and the Injury Severity Score.

RESULTS

Total patients of 3788 (3434 blunt and 354 penetrating injuries) with average age of 37 years (standard deviation of 16 years) were included in this study. All MTOS-TRISS and NTrD-TRISS models examined in this study showed adequate discriminative ability with AUCs ranged from 0.86 to 0.89 for patients with blunt trauma mechanism and 0.89 to 0.99 for patients with penetrating trauma mechanism. The H-L goodness-of-fit test indicated the NTrD-TRISS model calibrated as good as the MTOS-TRISS model for patients with blunt trauma mechanism.

CONCLUSION

For patients with blunt trauma mechanism, both the MTOS-TRISS and NTrD-TRISS models showed good discrimination and calibration performances. Discrimination performance for the NTrD-TRISS model was revealed to be as good as the MTOS-TRISS model specifically for patients with penetrating trauma mechanism. Overall, this validation study has ascertained the discrimination and calibration performances of the NTrD-TRISS model to be as good as the MTOS-TRISS model particularly for patients with blunt trauma mechanism.

MRI Radiomic Features Are Independently Associated With Overall Survival in Soft Tissue Sarcoma

PURPOSE

Soft tissue sarcomas (STS) represent a heterogeneous group of diseases, and selection of individualized treatments remains a challenge. The goal of this study was to determine whether radiomic features extracted from magnetic resonance (MR) images are independently associated with overall survival (OS) in STS.

METHODS AND MATERIALS

This study analyzed 2 independent cohorts of adult patients with stage II-III STS treated at center 1 (N = 165) and center 2 (N = 61). Thirty radiomic features were extracted from pretreatment T1-weighted contrast-enhanced MR images. Prognostic models for OS were derived on the center 1 cohort and validated on the center 2 cohort. Clinical-only (C), radiomics-only (R), and clinical and radiomics (C+R) penalized Cox models were constructed. Model performance was assessed using Harrell's concordance index.

RESULTS

In the R model, tumor volume (hazard ratio [HR], 1.5) and 4 texture features (HR, 1.1-1.5) were selected. In the C+R model, both age (HR, 1.4) and grade (HR, 1.7) were selected along with 5 radiomic features. The adjusted c-indices of the 3 models ranged from 0.68 (C) to 0.74 (C+R) in the derivation cohort and 0.68 (R) to 0.78 (C+R) in the validation cohort. The radiomic features were independently associated with OS in the validation cohort after accounting for age and grade (HR, 2.4; P = .009).

CONCLUSIONS

This study found that radiomic features extracted from MR images are independently associated with OS when accounting for age and tumor grade. The overall predictive performance of 3-year OS using a model based on clinical and radiomic features was replicated in an independent cohort. Optimal models using clinical and radiomic features could improve personalized selection of therapy in patients with STS.

Prognostic models for breast cancer: a systematic review

BACKGROUND

Breast cancer is the most common cancer in women worldwide, with a great diversity in outcomes among individual patients. The ability to accurately predict a breast cancer outcome is important to patients, physicians, researchers, and policy makers. Many models have been developed and tested in different settings. We systematically reviewed the prognostic models developed and/or validated for patients with breast cancer.

METHODS

We conducted a systematic search in four electronic databases and some oncology websites, and a manual search in the bibliographies of the included studies. We identified original studies that were published prior to 1st January 2017, and presented the development and/or validation of models based mainly on clinico-pathological factors to predict mortality and/or recurrence in female breast cancer patients.

RESULTS

From the 96 articles selected from 4095 citations found, we identified 58 models, which predicted mortality (n = 28), recurrence (n = 23), or both (n = 7). The most frequently used predictors were nodal status (n = 49), tumour size (n = 42), tumour grade (n = 29), age at diagnosis (n = 24), and oestrogen receptor status (n = 21). Models were developed in Europe (n = 25), Asia (n = 13), North America (n = 12), and Australia (n = 1) between 1982 and 2016. Models were validated in the development cohorts (n = 43) and/or independent populations (n = 17), by comparing the predicted outcomes with the observed outcomes (n = 55) and/or with the outcomes estimated by other models (n = 32), or the outcomes estimated by individual prognostic factors (n = 8). The most commonly used methods were: Cox proportional hazards regression for model development (n = 32); the absolute differences between the predicted and observed outcomes (n = 30) for calibration; and C-index/AUC (n = 44) for discrimination. Overall, the models performed well in the development cohorts but less accurately in some independent populations, particularly in patients with high risk and young and elderly patients. An exception is the Nottingham Prognostic Index, which retains its predicting ability in most independent populations.

CONCLUSIONS

Many prognostic models have been developed for breast cancer, but only a few have been validated widely in different settings. Importantly, their performance was suboptimal in independent populations, particularly in patients with high risk and in young and elderly patients.