Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data

GEMA-Na and MELD 3.0 severity scores to address sex disparities for accessing liver transplantation: a nationwide retrospective cohort study

Background

The Gender-Equity Model for liver Allocation corrected by serum sodium (GEMA-Na) and the Model for End-stage Liver Disease 3.0 (MELD 3.0) could amend sex disparities for accessing liver transplantation (LT). We aimed to assess these inequities in Spain and to compare the performance of GEMA-Na and MELD 3.0.

Methods

Nationwide cohort study including adult patients listed for a first elective LT (January 2016-December 2021). The primary outcome was mortality or delisting for sickness within the first 90 days. Independent predictors of the primary outcome were evaluated using multivariate Cox's regression with adjusted relative risks (RR) and 95% confidence intervals (95% CI). The discrimination of GEMA-Na and MELD 3.0was assessed using Harrell c-statistics (Hc).

Findings

The study included 6071 patients (4697 men and 1374 women). Mortality or delisting for clinical deterioration occurred in 286 patients at 90 days (4.7%). Women had reduced access to LT (83.7% vs. 85.9%; p = 0.037) and increased risk of mortality or delisting for sickness at 90 days (adjusted RR = 1.57 [95% CI 1.09-2.28]; p = 0.017). Female sex remained as an independent risk factor when using MELD or MELD-Na but lost its significance in the presence of GEMA-Na or MELD 3.0. Among patients included for reasons other than tumours (n = 3606; 59.4%), GEMA-Na had Hc = 0.753 (95% CI 0.715-0.792), which was higher than MELD 3.0 (Hc = 0.726 [95% CI 0.686-0.767; p = 0.001), showing both models adequate calibration.

Interpretation

GEMA-Na and MELD 3.0 might correct sex disparities for accessing LT, but GEMA-Na provides more accurate predictions of waiting list outcomes and could be considered the standard of care for waiting list prioritization.

Funding

Instituto de Salud Carlos III, Agencia Estatal de Investigación (Spain), and European Union.

Cohort size required for prognostic genes analysis of stage II/III esophageal squamous cell carcinoma

Background: Few overlaps between prognostic biomarkers are observed among different independently performed genomic studies of esophageal squamous cell carcinoma (ESCC). One of the reasons for this is the insufficient cohort size. How many cases are needed to prognostic genes analysis in ESCC? Methods: Here, based on 387 stage II/III ESCC cases analyzed by whole-genome sequencing from one single center, effects of cohort size on prognostic genes analysis were investigated. Prognostic genes analysis was performed in 100 replicates at each cohort size level using a random resampling method. Results: The number of prognostic genes followed a power-law increase with cohort size in ESCC patients with stage II and stage III, with exponents of 2.27 and 2.25, respectively. Power-law curves with increasing events number were also observed in stage II and III ESCC, respectively, and they almost overlapped. The probability of obtaining statistically significant prognostic genes shows a logistic cumulative distribution function with respect to cohort size. To achieve a 100% probability of obtaining statistically significant prognostic genes, the minimum cohort sizes required in stage II and III ESCC were approximately 95 and 60, respectively, corresponding to a number of outcome events of 33 and 36, respectively. Conclusion: In summary, the number of prognostic genes follows a power-law growth with the cohort size or events number in ESCC. The minimum events number required to achieve a 100% probability of obtaining a statistically significant prognostic gene is approximately 35.

Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome

Previous articles in Statistics in Medicine describe how to calculate the sample size required for external validation of prediction models with continuous and binary outcomes. The minimum sample size criteria aim to ensure precise estimation of key measures of a model's predictive performance, including measures of calibration, discrimination, and net benefit. Here, we extend the sample size guidance to prediction models with a time-to-event (survival) outcome, to cover external validation in datasets containing censoring. A simulation-based framework is proposed, which calculates the sample size required to target a particular confidence interval width for the calibration slope measuring the agreement between predicted risks (from the model) and observed risks (derived using pseudo-observations to account for censoring) on the log cumulative hazard scale. Precise estimation of calibration curves, discrimination, and net-benefit can also be checked in this framework. The process requires assumptions about the validation population in terms of the (i) distribution of the model's linear predictor and (ii) event and censoring distributions. Existing information can inform this; in particular, the linear predictor distribution can be approximated using the C-index or Royston's D statistic from the model development article, together with the overall event risk. We demonstrate how the approach can be used to calculate the sample size required to validate a prediction model for recurrent venous thromboembolism. Ideally the sample size should ensure precise calibration across the entire range of predicted risks, but must at least ensure adequate precision in regions important for clinical decision-making. Stata and R code are provided.

S-GRAS score for prognostic classification of adrenocortical carcinoma: an international, multicenter ENSAT study

OBJECTIVE

Adrenocortical carcinoma (ACC) has an aggressive but variable clinical course. Prognostic stratification based on the European Network for the Study of Adrenal Tumours stage and Ki67 index is limited. We aimed to demonstrate the prognostic role of a points-based score (S-GRAS) in a large cohort of patients with ACC.

DESIGN

This is a multicentre, retrospective study on ACC patients who underwent adrenalectomy.

METHODS

The S-GRAS score was calculated as a sum of the following points: tumour stage (1-2 = 0; 3 = 1; 4 = 2), grade (Ki67 index 0-9% = 0; 10-19% = 1; ≥20% = 2 points), resection status (R0 = 0; RX = 1; R1 = 2; R2 = 3), age (<50 years = 0; ≥50 years = 1), symptoms (no = 0; yes = 1), and categorised, generating four groups (0-1, 2-3, 4-5, and 6-9). Endpoints were progression-free survival (PFS) and disease-specific survival (DSS). The discriminative performance of S-GRAS and its components was tested by Harrell's Concordance index (C-index) and Royston-Sauerbrei's R2D statistic.

RESULTS

We included 942 ACC patients. The S-GRAS score showed superior prognostic performance for both PFS and DSS, with best discrimination obtained using the individual scores (0-9) (C-index = 0.73, R2D = 0.30, and C-index = 0.79, R2D = 0.45, respectively, all P < 0.01vs each component). The superiority of S-GRAS score remained when comparing patients treated or not with adjuvant mitotane (n = 481 vs 314). In particular, the risk of recurrence was significantly reduced as a result of adjuvant mitotane only in patients with S-GRAS 4-5.

CONCLUSION

The prognostic performance of S-GRAS is superior to tumour stage and Ki67 in operated ACC patients, independently from adjuvant mitotane. S-GRAS score provides a new important guide for personalised management of ACC (i.e. radiological surveillance and adjuvant treatment).

Nomogram to predict risk of incident chronic kidney disease in high-risk population of cardiovascular disease in China: community-based cohort study

AIMS

To develop a nomogram for incident chronic kidney disease (CKD) risk evaluation among community residents with high cardiovascular disease (CVD) risk.

METHODS

In this retrospective cohort study, 5730 non-CKD residents with high CVD risk participating the National Basic Public Health Service between January 2015 and December 2020 in Guangzhou were included. Endpoint was incident CKD defined as an estimated glomerular filtration rate (eGFR) less than 60 mL/min/1.73 m² during the follow-up period. The entire cohorts were randomly (2:1) assigned to a development cohort and a validation cohort. Predictors of incident CKD were selected by multivariable Cox regression and stepwise approach. A nomogram based on these predictors was developed and evaluated with concordance index (C-index) and area under curve (AUC).

RESULTS

During the median follow-up period of 4.22 years, the incidence of CKD was 19.09% (n=1094) in the entire cohort, 19.03% (727 patients) in the development cohort and 19.21% (367 patients) in the validation cohort. Age, body mass index, eGFR 60-89 mL/min/1.73 m², diabetes and hypertension were selected as predictors. The nomogram demonstrated a good discriminative power with C-index of 0.778 and 0.785 in the development and validation cohort. The 3-year, 4-year and 5-year AUCs were 0.817, 0.814 and 0.834 in the development cohort, and 0.830, 0.847 and 0.839 in the validation cohort.

CONCLUSION

Our nomogram based on five readily available predictors is a reliable tool to identify high-CVD risk patients at risk of incident CKD. This prediction model may help improving the healthcare strategies in primary care.

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

INTRODUCTION

Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.

METHODS

Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.

RESULTS

Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.

CONCLUSION

Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Clinical prediction tools to identify patients at highest risk of myeloma in primary care: a retrospective open cohort study

BACKGROUND

Patients with myeloma experience substantial delays in their diagnosis, which can adversely affect their prognosis.

AIM

To generate a clinical prediction rule to identify primary care patients who are at highest risk of myeloma.

DESIGN AND SETTING

Retrospective open cohort study using electronic health records data from the UK's Clinical Practice Research Datalink (CPRD) between 1 January 2000 and 1 January 2014.

METHOD

Patients from the CPRD were included in the study if they were aged ≥40 years, had two full blood counts within a year, and had no previous diagnosis of myeloma. Cases of myeloma were identified in the following 2 years. Derivation and external validation datasets were created based on geographical region. Prediction equations were estimated using Cox proportional hazards models including patient characteristics, symptoms, and blood test results. Calibration, discrimination, and clinical utility were evaluated in the validation set.

RESULTS

Of 1 281 926 eligible patients, 737 (0.06%) were diagnosed with myeloma within 2 years. Independent predictors of myeloma included: older age; male sex; back, chest and rib pain; nosebleeds; low haemoglobin, platelets, and white cell count; and raised mean corpuscular volume, calcium, and erythrocyte sedimentation rate. A model including symptoms and full blood count had an area under the curve of 0.84 (95% CI = 0.81 to 0.87) and sensitivity of 62% (95% CI = 55% to 68%) at the highest risk decile. The corresponding statistics for a second model, which also included calcium and inflammatory markers, were an area under the curve of 0.87 (95% CI = 0.84 to 0.90) and sensitivity of 72% (95% CI = 66% to 78%).

CONCLUSION

The implementation of these prediction rules would highlight the possibility of myeloma in patients where GPs do not suspect myeloma. Future research should focus on the prospective evaluation of further external validity and the impact on clinical practice.

A note on estimating the Cox-Snell R2 from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome

In 2019 we published a pair of articles in Statistics in Medicine that describe how to calculate the minimum sample size for developing a multivariable prediction model with a continuous outcome, or with a binary or time-to-event outcome. As for any sample size calculation, the approach requires the user to specify anticipated values for key parameters. In particular, for a prediction model with a binary outcome, the outcome proportion and a conservative estimate for the overall fit of the developed model as measured by the Cox-Snell R2 (proportion of variance explained) must be specified. This proposal raises the question of how to identify a plausible value for R2 in advance of model development. Our articles suggest researchers should identify R2 from closely related models already published in their field. In this letter, we present details on how to derive R2 using the reported C statistic (AUROC) for such existing prediction models with a binary outcome. The C statistic is commonly reported, and so our approach allows researchers to obtain R2 for subsequent sample size calculations for new models. Stata and R code is provided, and a small simulation study.

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.

Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

OBJECTIVE

Developing clinical prediction models (CPMs) on data of sufficient sample size is critical to help minimize overfitting. Using prostate cancer as a clinical exemplar, we aimed to investigate to what extent existing CPMs adhere to recent formal sample size criteria, or historic rules of thumb of events per predictor parameter (EPP)≥10.

STUDY DESIGN AND SETTING

A systematic review to identify CPMs related to prostate cancer, which provided enough information to calculate minimum sample size. We compared the reported sample size of each CPM against the traditional 10 EPP rule of thumb and formal sample size criteria.

RESULTS

About 211 CPMs were included. Three of the studies justified the sample size used, mostly using EPP rules of thumb. Overall, 69% of the CPMs were derived on sample sizes that surpassed the traditional EPP≥10 rule of thumb, but only 48% surpassed recent formal sample size criteria. For most CPMs, the required sample size based on formal criteria was higher than the sample sizes to surpass 10 EPP.

CONCLUSION

Few of the CPMs included in this study justified their sample size, with most justifications being based on EPP. This study shows that, in real-world data sets, adhering to the classic EPP rules of thumb is insufficient to adhere to recent formal sample size criteria.

Risk assessment for hospital admission in patients with COPD; a multi-centre UK prospective observational study

In chronic obstructive pulmonary disease (COPD), acute exacerbation of COPD requiring hospital admission is associated with mortality and healthcare costs. The ERICA study assessed multiple clinical measures in people with COPD, including the short physical performance battery (SPPB), a simple test of physical function with 3 components (gait speed, balance and sit-to-stand). We tested the hypothesis that SPPB score would relate to risk of hospital admissions and length of hospital stay. Data were analysed from 714 of the total 729 participants (434 men and 280 women) with COPD. Data from this prospective observational longitudinal study were obtained from 4 secondary and 1 tertiary centres from England, Scotland, and Wales. The main outcome measures were to estimate the risk of hospitalisation with acute exacerbation of COPD (AECOPD and length of hospital stay derived from hospital episode statistics (HES). In total, 291 of 714 individuals experienced 762 hospitalised AECOPD during five-year follow up. Poorer performance of SPPB was associated with both higher rate (IRR 1.08 per 1 point decrease, 95% CI 1.01 to 1.14) and increased length of stay (IRR 1.18 per 1 point decrease, 95% CI 1.10 to 1.27) for hospitalised AECOPD. For the individual sit-to-stand component of the SPPB, the association was even stronger (IRR 1.14, 95% CI 1.02 to 1.26 for rate and IRR 1.32, 95% CI 1.16 to 1.49 for length of stay for hospitalised AECOPD). The SPPB, and in particular the sit-to-stand component can both evaluate the risk of H-AECOPD and length of hospital stay in COPD. The SPPB can aid in clinical decision making and when prioritising healthcare resources.

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

It is widely recommended that any developed-diagnostic or prognostic-prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc".

Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes

When designing a study to develop a new prediction model with binary or time-to-event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R2 , and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox-Snell R2 , which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

Development and validation of a new predictive model for breast cancer survival in New Zealand and comparison to the Nottingham prognostic index

Background

The only available predictive models for the outcome of breast cancer patients in New Zealand (NZ) are based on data in other countries. We aimed to develop and validate a predictive model using NZ data for this population, and compare its performance to a widely used overseas model, the Nottingham Prognostic Index (NPI).

Methods

We developed a model to predict 10-year breast cancer-specific survival, using data collected prospectively in the largest population-based regional breast cancer registry in NZ (Auckland, 9182 patients), and assessed its performance in this data set (internal validation) and in an independent NZ population-based series of 2625 patients in Waikato (external validation). The data included all women with primary invasive breast cancer diagnosed from 1 June 2000 to 30 June 2014, with follow up to death or Dec 31, 2014. We used multivariate Cox proportional hazards regression to assess predictors and to calculate predicted 10-year breast cancer mortality, and therefore survival, probability for each patient. We assessed observed survival by the Kaplan Meier method. We assessed discrimination by the C statistic, and calibration by comparing predicted and observed survival rates for patients in 10 groups ordered by predicted 10-year survival. We compared this NZ model with the Nottingham Prognostic Index (NPI) in this validation data set.

Results

Discrimination was good: C statistics were 0.84 for internal validity and 0.83 for an independent external validity. For calibration, for both internal and external validity the predicted 10-year survival probabilities in all groups of patients, ordered by predicted survival, were within the 95% confidence intervals (CI) of the observed Kaplan-Meier survival probabilities. The NZ model showed good discrimination even within the prognostic groups defined by the NPI.

Conclusions

These results for the New Zealand model show good internal and external validity, transportability, and potential clinical value of the model, and its clear superiority over the NPI. Further research is needed to assess other potential predictors, to assess the model’s performance in specific subgroups of patients, and to compare it to other models, which have been developed in other countries and have not yet been tested in NZ.

Electronic supplementary material

The online version of this article (10.1186/s12885-018-4791-x) contains supplementary material, which is available to authorized users.

Simulation of complex data structures for planning of studies with focus on biomarker comparison

BACKGROUND

There are a growing number of observational studies that do not only focus on single biomarkers for predicting an outcome event, but address questions in a multivariable setting. For example, when quantifying the added value of new biomarkers in addition to established risk factors, the aim might be to rank several new markers with respect to their prediction performance. This makes it important to consider the marker correlation structure for planning such a study. Because of the complexity, a simulation approach may be required to adequately assess sample size or other aspects, such as the choice of a performance measure.

METHODS

In a simulation study based on real data, we investigated how to generate covariates with realistic distributions and what generating model should be used for the outcome, aiming to determine the least amount of information and complexity needed to obtain realistic results. As a basis for the simulation a large epidemiological cohort study, the Gutenberg Health Study was used. The added value of markers was quantified and ranked in subsampling data sets of this population data, and simulation approaches were judged by the quality of the ranking. One of the evaluated approaches, the random forest, requires original data at the individual level. Therefore, also the effect of the size of a pilot study for random forest based simulation was investigated.

RESULTS

We found that simple logistic regression models failed to adequately generate realistic data, even with extensions such as interaction terms or non-linear effects. The random forest approach was seen to be more appropriate for simulation of complex data structures. Pilot studies starting at about 250 observations were seen to provide a reasonable level of information for this approach.

CONCLUSIONS

We advise to avoid oversimplified regression models for simulation, in particular when focusing on multivariable research questions. More generally, a simulation should be based on real data for adequately reflecting complex observational data structures, such as found in epidemiological cohort studies.