Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

Epidemiological description and trajectories of patients with prostate cancer in Denmark: an observational study of 7448 patients

OBJECTIVE

Identification of patients at high risk of aggressive prostate cancer is a major clinical challenge. With the view of developing artificial intelligence-based methods for identification of these patients, we are constructing a comprehensive clinical database including 7448 prostate cancer (PCa) Danish patients. In this paper we provide an epidemiological description and patients' trajectories of this retrospective observational population, to contribute to the understanding of the characteristics and pathways of PCa patients in Denmark.

RESULTS

Individuals receiving a PCa diagnosis during 2008-2014 in Region Southern Denmark were identified, and all diagnoses, operations, investigations, and biochemistry analyses, from 4 years prior, to 5 years after PCa diagnosis were obtained. About 85.1% were not diagnosed with metastatic PCa during the study period (unaggressive PCa); 9.2% were simultaneously diagnosed with PCa and metastasis (aggressive-advanced PCa), while 5.7% were not diagnosed with metastatic PCa at first, but they were diagnosed with metastasis at some point during the 5 years follow-up (aggressive-not advanced PCa). Patients with unaggressive PCa had more clinical investigations directly related to PCa detection (prostate ultrasounds and biopsies) during the 4 years prior to PCa diagnosis, compared to patients with aggressive PCa, which may have contributed to the early detection of PCa.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

BACKGROUND

Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome.

METHODS

We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size.

RESULTS

A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84).

CONCLUSIONS

Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review

OBJECTIVE

The purpose of this systematic review was to identify and appraise externally validated prognostic models to predict a patient's health outcomes relevant to physical rehabilitation of musculoskeletal (MSK) conditions.

METHODS

We systematically reviewed 8 databases and reported our findings according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020. An information specialist designed a search strategy to identify externally validated prognostic models for MSK conditions. Paired reviewers independently screened the title, abstract, and full text and conducted data extraction. We extracted characteristics of included studies (eg, country and study design), prognostic models (eg, performance measures and type of model) and predicted clinical outcomes (eg, pain and disability). We assessed the risk of bias and concerns of applicability using the prediction model risk of bias assessment tool. We proposed and used a 5-step method to determine which prognostic models were clinically valuable.

RESULTS

We found 4896 citations, read 300 full-text articles, and included 46 papers (37 distinct models). Prognostic models were externally validated for the spine, upper limb, lower limb conditions, and MSK trauma, injuries, and pain. All studies presented a high risk of bias. Half of the models showed low concerns for applicability. Reporting of calibration and discrimination performance measures was often lacking. We found 6 externally validated models with adequate measures, which could be deemed clinically valuable [ie, (1) STart Back Screening Tool, (2) Wallis Occupational Rehabilitation RisK model, (3) Da Silva model, (4) PICKUP model, (5) Schellingerhout rule, and (6) Keene model]. Despite having a high risk of bias, which is mostly explained by the very conservative properties of the PROBAST tool, the 6 models remain clinically relevant.

CONCLUSION

We found 6 externally validated prognostic models developed to predict patients' health outcomes that were clinically relevant to the physical rehabilitation of MSK conditions.

IMPACT

Our results provide clinicians with externally validated prognostic models to help them better predict patients' clinical outcomes and facilitate personalized treatment plans. Incorporating clinically valuable prognostic models could inherently improve the value of care provided by physical therapists.

Predictive index for adverse perinatal outcome in pregnancies complicated by fetal growth restriction

OBJECTIVES

To develop and validate an index predictive of adverse perinatal outcome (APO) in pregnancies meeting the consensus-based criteria for fetal growth restriction (FGR) endorsed by the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG).

METHODS

This was a retrospective analysis of consecutive singleton non-anomalous gestations meeting the ISUOG-endorsed criteria for FGR at a single tertiary care center from November 2010 to August 2020. The dataset was divided randomly into a development set (two-thirds) and a validation set (one-third). The primary composite APO comprised one or more of: perinatal demise, Grade III-IV intraventricular hemorrhage (IVH), periventricular leukomalacia (PVL), seizures, hypoxic ischemic encephalopathy (HIE), necrotizing enterocolitis (NEC), sepsis, bronchopulmonary dysplasia (BPD) and length of stay in the neonatal intensive care unit (NICU) > 7 days. Regression analysis incorporated clinical factors readily available at the time of FGR diagnosis. The sum of β coefficient-based weights yielded an index score, the performance of which was assessed in the validation set. Score cut-offs were selected to identify 'high-risk' and 'low-risk' ranges for which positive (PPV) and negative (NPV) predictive values and positive (LR+) and negative (LR-) likelihood ratios were calculated.

RESULTS

Of the 875 consecutive pregnancies that met the criteria for FGR and were included in the study cohort, 405 (46%) were complicated by one or more components of the composite APO, including 54 (6%) perinatal deaths, 22 (3%) neonates with Grade III-IV IVH and/or PVL, nine (1%) with seizures and/or HIE, 91 (10%) with BPD, 57 (7%) with sepsis, 21 (2%) with NEC, and 361 (41%) who remained in the NICU > 7 days. In addition, 270 (31%) pregnancies were delivered by Cesarean section for non-reassuring fetal status, 43 (5%) were admitted to the NICU for < 7 days, 79 (9%) had 5-min Apgar score < 7, 125/631 (20%) had a cord gas pH ≤ 7.1 and 35/631 (6%) had a base excess ≥ 12 mmol/L. The predictive index we developed included seven factors available at the time of FGR diagnosis: hypertensive disorder of pregnancy (HDP) (+8 points), chronic hypertension without HDP (+4 points), gestational age ≤ 32 weeks (+5 points), absent or reversed end-diastolic flow in the umbilical artery (+8 points), prepregnancy body mass index ≥ 35 kg/m2 (+3 points), isolated abdominal circumference < 3rd  percentile (-4 points) and non-Hispanic black race (-2 points). The bias-corrected bootstrapped (1000 replicates) area under the receiver-operating-characteristics curve (AUC) of the predictive index for composite APO in the validation group was 0.88 (95% CI, 0.84-0.92), which was similar to that in the development group (AUC, 0.86 (95% CI, 0.82-0.89); P = 0.34). In the total cohort, 40% of pregnancies had a low-risk index score (≤ 2), associated with a NPV of 85% (95% CI, 81-88%) and a LR- of 0.21 (95% CI, 0.16-0.27), and 23% had a high-risk index score (≥ 10), associated with a PPV of 96% (95% CI, 93-98%) and a LR+ of 27.36 (95% CI, 14.33-52.23). Of the remaining pregnancies that had an intermediate-risk score, 50% were complicated by composite APO.

CONCLUSION

An easy-to-use index incorporating seven clinical factors readily available at the time of FGR diagnosis is predictive of APO and may prove useful in counseling and management of pregnancies meeting the ISUOG-endorsed criteria for FGR. © 2022 International Society of Ultrasound in Obstetrics and Gynecology.

Joint models for dynamic prediction in localised prostate cancer: a literature review

BACKGROUND

Prostate cancer is a very prevalent disease in men. Patients are monitored regularly during and after treatment with repeated assessment of prostate-specific antigen (PSA) levels. Prognosis of localised prostate cancer is generally good after treatment, and the risk of having a recurrence is usually estimated based on factors measured at diagnosis. Incorporating PSA measurements over time in a dynamic prediction joint model enables updates of patients' risk as new information becomes available. We review joint model strategies that have been applied to model time-dependent PSA trajectories to predict time-to-event outcomes in localised prostate cancer.

METHODS

We identify articles that developed joint models for prediction of localised prostate cancer recurrence over the last two decades. We report, compare, and summarise the methodological approaches and applications that use joint modelling accounting for two processes: the longitudinal model (PSA), and the time-to-event process (clinical failure). The methods explored differ in how they specify the association between these two processes.

RESULTS

Twelve relevant articles were identified. A range of methodological frameworks were found, and we describe in detail shared-parameter joint models (9 of 12, 75%) and joint latent class models (3 of 12, 25%). Within each framework, these articles presented model development, estimation of dynamic predictions and model validations.

CONCLUSIONS

Each framework has its unique principles with corresponding advantages and differing interpretations. Regardless of the framework used, dynamic prediction models enable real-time prediction of individual patient prognosis. They utilise all available longitudinal information, in addition to baseline prognostic risk factors, and are superior to traditional baseline-only prediction models.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

BACKGROUND

Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology.

METHODS

We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models.

RESULTS

Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available.

CONCLUSIONS

The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Sample size justifications in Gait & Posture

BACKGROUND

Context regarding how researchers determine the sample size of their experiments is important for interpreting the results and determining their value and meaning. Between 2018 and 2019, the journal Gait & Posture introduced a requirement for sample size justification in their author guidelines.

RESEARCH QUESTION

How frequently and in what ways are sample sizes justified in Gait & Posture research articles and was the inclusion of a guideline requiring sample size justification associated with a change in practice?

METHODS

The guideline was not in place prior to May 2018 and was in place from 25th July 2019. All articles in the three most recent volumes of the journal (84-86) and the three most recent, pre-guideline volumes (60-62) at time of preregistration were included in this analysis. This provided an initial sample of 324 articles (176 pre-guideline and 148 post-guideline). Articles were screened by two authors to extract author data, article metadata and sample size justification data. Specifically, screeners identified if (yes or no) and how sample sizes were justified. Six potential justification types (Measure Entire Population, Resource Constraints, Accuracy, A priori Power Analysis, Heuristics, No Justification) and an additional option of Other/Unsure/Unclear were used.

RESULTS

In most cases, authors of Gait & Posture articles did not provide a justification for their study's sample size. The inclusion of the guideline was associated with a modest increase in the percentage of articles providing a justification (16.6-28.1%). A priori power calculations were the dominant type of justification, but many were not reported in enough detail to allow replication.

SIGNIFICANCE

Gait & Posture researchers should be more transparent in how they determine their sample sizes and carefully consider if they are suitable. Editors and journals may consider adding a similar guideline as a low-resource way to improve sample size justification reporting.