Minimum sample size for external validation of a clinical prediction model with a binary outcome

Preoperative risk prediction models for acute kidney injury after noncardiac surgery: an independent external validation cohort study

BACKGROUND

Numerous models have been developed to predict acute kidney injury (AKI) after noncardiac surgery, yet there is a lack of independent validation and comparison among them.

METHODS

We conducted a systematic literature search to review published risk prediction models for AKI after noncardiac surgery. An independent external validation was performed using a retrospective surgical cohort at a large Chinese hospital from January 2019 to October 2022. The cohort included patients undergoing a wide range of noncardiac surgeries with perioperative creatinine measurements. Postoperative AKI was defined according to the Kidney Disease Improving Global Outcomes creatinine criteria. Model performance was assessed in terms of discrimination (area under the receiver operating characteristic curve, AUROC), calibration (calibration plot), and clinical utility (net benefit), before and after model recalibration through intercept and slope updates. A sensitivity analysis was conducted by including patients without postoperative creatinine measurements in the validation cohort and categorising them as non-AKI cases.

RESULTS

Nine prediction models were evaluated, each with varying clinical and methodological characteristics, including the types of surgical cohorts used for model development, AKI definitions, and predictors. In the validation cohort involving 13,186 patients, 650 (4.9%) developed AKI. Three models demonstrated fair discrimination (AUROC between 0.71 and 0.75); other models had poor or failed discrimination. All models exhibited some miscalibration; five of the nine models were well-calibrated after intercept and slope updates. Decision curve analysis indicated that the three models with fair discrimination consistently provided a positive net benefit after recalibration. The results were confirmed in the sensitivity analysis.

CONCLUSIONS

We identified three models with fair discrimination and potential clinical utility after recalibration for assessing the risk of acute kidney injury after noncardiac surgery.

External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions

BACKGROUND

Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model.

METHODS

In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots).

RESULTS

Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 [95 % CI 0.86-0.88]) and good calibration to the TRFT dataset (slope = 1.03 [95 % CI 0.98-1.08] intercept = 0 [95 % CI -0.06-0.07]) after re-calibrating for differences in baseline mortality (intercept = 0.96 [95 % CI 0.90-1.03] before re-calibration).

CONCLUSION

In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.

Development and Validation of a Supplementary Grading Scale for Outcomes of Brainstem Cavernous Malformations

BACKGROUND

Surgical risk assessment is intriguing for clinical decision-making of brainstem cavernous malformation (BSCM) treatment. While the BSCM grading scale, encompassing size, developmental venous anomaly, crossing axial midpoint, age, and timing of intervention, is increasingly utilized, the clinical relevance of neurological fluctuation and recurrent hemorrhage has not been incorporated. This study aimed to propose a supplementary grading scale with enhanced predictive efficacy.

METHODS

Using a retrospective nationwide registry of consecutive patients with BSCMs undergoing surgery in China from March 2011 to May 2023, a new supplementary BSCM grading scale was developed from a derivative cohort of 260 patients and validated in an independent concurrent cohort of 67 patients. The primary outcome was unfavorable neurological function (modified Rankin Scale score >2) at the latest follow-up. The performance of the supplementary grading system was evaluated for discrimination, calibration, and clinical utility and further compared with its original counterpart.

RESULTS

Over a follow-up of at least 6 months after surgery, the unfavorable outcomes were 31% in the overall cohort (101/327 patients). A preoperative motor deficit (odds ratio, 3.13; P=0.001), recurrent hemorrhage (odds ratio, 3.05; P<0.001), timing of intervention (odds ratio, 7.08; P<0.001), and crossing the axial midpoint (odds ratio, 2.57; P=0.006) were associated with the unfavorable outcomes and composed the initial Huashan grading variables. A supplementary BSCM grading system was subsequently developed by incorporating the Huashan grading variables into the original BSCM grading scale. The predictive capability of the supplementary scale was consistently superior to the original counterpart in either the derivative cohort (area under the receiver operating characteristic curve, 0.74 [95% CI, 0.68-0.80] for the supplementary versus 0.68 [95% CI, 0.61-0.74] for the original) or the validation cohort (0.75 [95% CI, 0.62-0.87] versus 0.64 [95% CI, 0.48-0.81]).

CONCLUSIONS

This study highlights the neurological relevance of BSCM hemorrhage in surgical risk assessment. Via compositing preoperative motor function and recurrent hemorrhages, a supplementary grading scale may improve a dynamic risk assessment for clinical decisions in the management of BSCMs.

Deep learning-based platform performs high detection sensitivity of intracranial aneurysms in 3D brain TOF-MRA: An external clinical validation study

PURPOSE

To evaluate the diagnostic efficacy of a developed artificial intelligence (AI) platform incorporating deep learning algorithms for the automated detection of intracranial aneurysms in time-of-flight (TOF) magnetic resonance angiography (MRA).

METHOD

This retrospective study encompassed 3D TOF MRA images acquired between January 2023 and June 2023, aiming to validate the presence of intracranial aneurysms via our developed AI platform. The manual segmentation results by experienced neuroradiologists served as the "gold standard". Following annotation of MRA images by neuroradiologists using InferScholar software, the AI platform conducted automatic segmentation of intracranial aneurysms. Various metrics including accuracy (ACC), balanced ACC, area under the curve (AUC), sensitivity (SE), specificity (SP), F1 score, Brier Score, and Net Benefit were utilized to evaluate the generalization of AI platform. Comparison of intracranial aneurysm identification performance was conducted between the AI platform and six radiologists with experience ranging from 3 to 12 years in interpreting MR images. Additionally, a comparative analysis was carried out between radiologists' detection performance based on independent visual diagnosis and AI-assisted diagnosis. Subgroup analyses were also performed based on the size and location of the aneurysms to explore factors impacting aneurysm detectability.

RESULTS

510 patients were enrolled including 215 patients (42.16 %) with intracranial aneurysms and 295 patients (57.84 %) without aneurysms. Compared with six radiologists, the AI platform showed competitive discrimination power (AUC, 0.96), acceptable calibration (Brier Score loss, 0.08), and clinical utility (Net Benefit, 86.96 %). The AI platform demonstrated superior performance in detecting aneurysms with an overall SE, SP, ACC, balanced ACC, and F1 score of 91.63 %, 92.20 %, 91.96 %, 91.92 %, and 90.57 % respectively, outperforming the detectability of the two resident radiologists. For subgroup analysis based on aneurysm size and location, we observed that the SE of the AI platform for identifying tiny (diameter＜3mm), small (3 mm ≤ diameter＜5mm), medium (5 mm ≤ diameter＜7mm) and large aneurysms (diameter ≥ 7 mm) was 87.80 %, 93.14 %, 95.45 %, and 100 %, respectively. Furthermore, the SE for detecting aneurysms in the anterior circulation was higher than that in the posterior circulation. Utilizing the AI assistance, six radiologists (i.e., two residents, two attendings and two professors) achieved statistically significant improvements in mean SE (residents: 71.40 % vs. 88.37 %; attendings: 82.79 % vs. 93.26 %; professors: 90.07 % vs. 97.44 %; P < 0.05) and ACC (residents: 85.29 % vs. 94.12 %; attendings: 91.76 % vs. 97.06 %; professors: 95.29 % vs. 98.82 %; P < 0.05) while no statistically significant change was observed in SP. Overall, radiologists' mean SE increased by 11.40 %, mean SP increased by 1.86 %, and mean ACC increased by 5.88 %, mean balanced ACC promoted by 6.63 %, mean F1 score grew by 7.89 %, and Net Benefit rose by 12.52 %, with a concurrent decrease in mean Brier score declined by 0.06.

CONCLUSIONS

The deep learning algorithms implemented in the AI platform effectively detected intracranial aneurysms on TOF-MRA and notably enhanced the diagnostic capabilities of radiologists. This indicates that the AI-based auxiliary diagnosis model can provide dependable and precise prediction to improve the diagnostic capacity of radiologists.

Risk of serious adverse events after primary shoulder replacement: development and external validation of a prediction model using linked national data from England and Denmark

BACKGROUND

Despite a rising rate of serious medical complications after shoulder replacement surgery, there are no prediction models in widespread use to guide surgeons in identifying patients at high risk and to provide patients with personalised risk estimates to support shared decision making. Our aim was to develop and externally validate a prediction model for serious adverse events within 90 days of primary shoulder replacement surgery.

METHODS

Linked data from the National Joint Registry, National Health Service Hospital Episode Statistics Admitted Patient Care of England, and Civil Registration Mortality databases and Danish Shoulder Arthroplasty Registry and National Patient Register were used for our modelling study. Patients aged 18-100 years who had a primary shoulder replacement between April 1, 2012, and Oct 2, 2020, in England, and April 1, 2012, and Oct 2, 2018, in Denmark, were included. We developed a multivariable logistic regression model using the English dataset to predict the risk of 90-day serious adverse events, which were defined as medical complications requiring admission to hospital and all-cause death. We undertook internal validation using bootstrapping, and internal-external cross-validation across different geographical regions of England. The English model was externally validated on the Danish dataset.

FINDINGS

Data for 40 631 patients undergoing primary shoulder replacement (mean age 72·5 years [SD 9·9]; 28 709 [70·7%] women and 11 922 [29·3%] men) were used for model development, of whom 2270 (5·6%) had a 90-day serious adverse event. On internal validation, the model had a C-statistic of 0·717 (95% CI 0·707-0·728) and was well calibrated. Internal-external cross-validation showed consistent model performance across all regions in England. Upon external validation on the Danish dataset (n=6653; mean age 70·5 years [SD 10·3]; 4503 [67·7%] women and 2150 [32·3%] men), the model had a C-statistic of 0·750 (95% CI 0·723-0·776). Decision curve analysis showed clinical utility, with net benefit across all risk thresholds.

INTERPRETATION

This externally validated prediction model uses commonly available clinical variables to accurately predict the risk of serious medical complications after primary shoulder replacement surgery. The model is generalisable and applicable to most patients in need of a shoulder replacement. Its use offers support to clinicians and could inform and empower patients in the shared decision-making process.

FUNDING

National Institute for Health and Care Research and the Department of Orthopaedic Surgery, Herlev and Gentofte Hospital, Denmark.

How to develop, validate, and update clinical prediction models using multinomial logistic regression

OBJECTIVES

Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression.

STUDY DESIGN AND SETTING

We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research.

RESULTS

This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided.

CONCLUSION

We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation.

Second opinion machine learning for fast-track pathway assignment in hip and knee replacement surgery: the use of patient-reported outcome measures

BACKGROUND

The frequency of hip and knee arthroplasty surgeries has been rising steadily in recent decades. This trend is attributed to an aging population, leading to increased demands on healthcare systems. Fast Track (FT) surgical protocols, perioperative procedures designed to expedite patient recovery and early mobilization, have demonstrated efficacy in reducing hospital stays, convalescence periods, and associated costs. However, the criteria for selecting patients for FT procedures have not fully capitalized on the available patient data, including patient-reported outcome measures (PROMs).

METHODS

Our study focused on developing machine learning (ML) models to support decision making in assigning patients to FT procedures, utilizing data from patients' self-reported health status. These models are specifically designed to predict the potential health status improvement in patients initially selected for FT. Our approach focused on techniques inspired by the concept of controllable AI. This includes eXplainable AI (XAI), which aims to make the model's recommendations comprehensible to clinicians, and cautious prediction, a method used to alert clinicians about potential control losses, thereby enhancing the models' trustworthiness and reliability.

RESULTS

Our models were trained and tested using a dataset comprising 899 records from individual patients admitted to the FT program at IRCCS Ospedale Galeazzi-Sant'Ambrogio. After training and selecting hyper-parameters, the models were assessed using a separate internal test set. The interpretable models demonstrated performance on par or even better than the most effective 'black-box' model (Random Forest). These models achieved sensitivity, specificity, and positive predictive value (PPV) exceeding 70%, with an area under the curve (AUC) greater than 80%. The cautious prediction models exhibited enhanced performance while maintaining satisfactory coverage (over 50%). Further, when externally validated on a separate cohort from the same hospital-comprising patients from a subsequent time period-the models showed no pragmatically notable decline in performance.

CONCLUSIONS

Our results demonstrate the effectiveness of utilizing PROMs as basis to develop ML models for planning assignments to FT procedures. Notably, the application of controllable AI techniques, particularly those based on XAI and cautious prediction, emerges as a promising approach. These techniques provide reliable and interpretable support, essential for informed decision-making in clinical processes.

Prediction models for diagnosis and prognosis of the colonization or infection of multidrug-resistant organisms in adults: a systematic review, critical appraisal, and meta-analysis

BACKGROUND

Prediction models help to target patients at risk of multidrug-resistant organism (MDRO) colonization or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, there is limited evidence to identify which among the available models are of low risk of bias and suitable for clinical application.

OBJECTIVES

To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonization or infection.

DATA SOURCES

Six electronic literature databases and clinical registration databases were searched until April 2022.

STUDY ELIGIBILITY CRITERIA

Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonization or infection in adults.

PARTICIPANTS

Adults (≥ 18 years old) without MDRO colonization or infection (in prognostic models) or with unknown or suspected MDRO colonization or infection (in diagnostic models).

ASSESSMENT OF RISK OF BIAS

The Prediction Model Risk of Bias Assessment Tool was used to assess the risk of bias. Evidence certainty was assessed using the Grading of Recommendations Assessment, Development, and Evaluation approach.

METHODS OF DATA SYNTHESIS

Meta-analyses were conducted to summarize the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets.

RESULTS

We included 162 models (108 studies) developed for diagnosing (n = 135) and predicting (n = 27) MDRO colonization or infection. Models exhibited a high-risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalization. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very low to low certainty of evidence.

CONCLUSIONS

The review comprehensively described the models for identifying patients at risk of MDRO colonization or infection. We cannot recommend which models are ready for application because of the high-risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.

External validation of a 2-year all-cause mortality prediction tool developed using machine learning in patients with stage 4-5 chronic kidney disease

BACKGROUND

Chronic kidney disease (CKD) is associated with increased mortality. Individual mortality prediction could be of interest to improve individual clinical outcomes. Using an independent regional dataset, the aim of the present study was to externally validate the recently published 2-year all-cause mortality prediction tool developed using machine learning.

METHODS

A validation dataset of stage 4 or 5 CKD outpatients was used. External validation performance of the prediction tool at the optimal cutoff-point was assessed by the area under the receiver operating characteristic curve (AUC-ROC), accuracy, sensitivity, and specificity. A survival analysis was then performed using the Kaplan-Meier method.

RESULTS

Data of 527 outpatients with stage 4 or 5 CKD were analyzed. During the 2 years of follow-up, 91 patients died and 436 survived. Compared to the learning dataset, patients in the validation dataset were significantly younger, and the ratio of deceased patients in the validation dataset was significantly lower. The performance of the prediction tool at the optimal cutoff-point was: AUC-ROC = 0.72, accuracy = 63.6%, sensitivity = 72.5%, and specificity = 61.7%. The survival curves of the predicted survived and the predicted deceased groups were significantly different (p < 0.001).

CONCLUSION

The 2-year all-cause mortality prediction tool for patients with stage 4 or 5 CKD showed satisfactory discriminatory capacity with emphasis on sensitivity. The proposed prediction tool appears to be of clinical interest for further development.

Validity of nomograms for predicting cancer risk in oral leukoplakia and oral lichen planus

OBJECTIVES

This study assessed the validity of nomograms for predicting malignant transformation (MT) among patients with oral leukoplakia (OL) and oral lichen planus (OLP).

MATERIALS AND METHODS

Two nomograms were identified following a systematic search. Variables to interrogate both nomograms were obtained for a retrospective OL/OLP cohort. Then, the nomograms were applied to estimate MT probabilities twice and their average was used to calculate the discriminatory performance, calibration, and potential net benefit of the models. Subgroup analyses were performed for patients with OL, OLP, and oral epithelial dysplasia.

RESULTS

Predicted probabilities were mostly significantly higher among OL/OLP patients who developed MT compared to those who did not (p = <0.001-0.034). AUC values and Brier scores of the nomograms were 0.644-0.844 and 0.040-0.088 among OL patients and 0.580-0.743 and 0.008-0.072 among OLP patients. Decision curve analysis suggested that the nomograms had some net benefit for risk stratification. However, the models did not best binary dysplasia grading in discriminatory validity and net benefit among patients with OL and oral epithelial dysplasia.

CONCLUSION

Nomograms for predicting MT may have satisfactory validity among patients with OL than OLP, but they do not outperform binary dysplasia grading in risk stratification of OL.

Developing and validating clinical prediction models in hepatology - An overview for clinicians

Prediction models are everywhere in clinical medicine. We use them to assign a diagnosis or a prognosis, and there have been continuous efforts to develop better prediction models. It is important to understand the fundamentals of prediction modelling, thus, we herein describe nine steps to develop and validate a clinical prediction model with the intention of implementing it in clinical practice: Determine if there is a need for a new prediction model; define the purpose and intended use of the model; assess the quality and quantity of the data you wish to develop the model on; develop the model using sound statistical methods; generate risk predictions on the probability scale (0-100%); evaluate the performance of the model in terms of discrimination, calibration, and clinical utility; validate the model using bootstrapping to correct for the apparent optimism in performance; validate the model on external datasets to assess the generalisability and transportability of the model; and finally publish the model so that it can be implemented or validated by others.

Global Validation of a Model to Predict Reduced Estimated GFR in People With Type 2 Diabetes Without Diagnosis of CKD

Introduction

A minimal-resource model for predicting reduced kidney function among people with type 2 diabetes and no diagnosis of chronic kidney disease (CKD) stages 3 to 5 was previously developed in a UK population to pre-screen for undiagnosed CKD. This study aims to evaluate the performance of the model on a global population and assess its adequacy with and without regional adjustment.

Methods

A retrospective observational study was performed using data collected from the iCaReMe global registry (NCT03549754) and the DISCOVER study (NCT02322762 and NCT02226822). Patients were grouped by their World Health Organization classified region. An estimated glomerular filtration rate (eGFR) <60 ml/min per 1.73 m2 was the marker of reduced kidney function. A regional-intercept recalibration was applied to adjust for regional variation. Discrimination and calibration were evaluated for the UK-developed and recalibrated models.

Results

A total of 14,180 patients (46 countries, 6 regions) were identified with type 2 diabetes, no previous diagnosis of CKD stages 3 to 5, and had a serum creatinine measurement or eGFR recorded. The UK model underestimated risk when applied globally and was deemed inadequate. The model with regional adjustment achieved the target sensitivity (80.5%; 95% confidence interval [CI]: 78.8%-82.3%) and demonstrated a relative improvement of 51.5% (95% CI: 48.1%-55.1%) in the positive predictive value (PPV), compared to a screen-all approach.

Conclusion

The regional-adjusted model demonstrated adequate performance globally. Incorporating the model within practice could help clinicians to risk-stratify and prioritize patients at high risk. This could enable improved efficiency via risk-tailored screening, particularly in lower-middle-income countries (LMICs).

Calibration plots for multistate risk predictions models

INTRODUCTION

There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.

METHODS

We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.

RESULTS

The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability.

CONCLUSIONS

We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer

BACKGROUND

The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups.

OBJECTIVE

To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms.

DESIGN, SETTING, AND PARTICIPANTS

Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined.

RESULTS AND LIMITATIONS

There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups.

CONCLUSIONS

The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer.

PATIENT SUMMARY

We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly.

External validation and comparison of magnetic resonance imaging-based risk prediction models for prostate biopsy stratification

PURPOSE

Magnetic resonance imaging (MRI) is a promising tool for risk assessment, potentially reducing the burden of unnecessary prostate biopsies. Risk prediction models that incorporate MRI data have gained attention, but their external validation and comparison are essential for guiding clinical practice. The aim is to externally validate and compare risk prediction models for the diagnosis of clinically significant prostate cancer (csPCa).

METHODS

A cohort of 4606 patients across fifteen European tertiary referral centers were identified from a prospective maintained database between January 2016 and April 2023. Transrectal or transperineal image-fusion MRI-targeted and systematic biopsies for PI-RADS score of ≥ 3 or ≥ 2 depending on patient characteristics and physician preferences. Probabilities for csPCa, defined as International Society of Urological Pathology (ISUP) grade ≥ 2, were calculated for each patients using eight models. Performance was characterized by area under the receiver operating characteristic curve (AUC), calibration, and net benefit. Subgroup analyses were performed across various clinically relevant subgroups.

RESULTS

Overall, csPCa was detected in 2154 (47%) patients. The models exhibited satisfactory performance, demonstrating good discrimination (AUC ranging from 0.75 to 0.78, p < 0.001), adequate calibration, and high net benefit. The model described by Alberts showed the highest clinical utility for threshold probabilities between 10 and 20%. Subgroup analyses highlighted variations in models' performance, particularly when stratified according to PSA level, biopsy technique and PI-RADS version.

CONCLUSIONS

We report a comprehensive external validation of risk prediction models for csPCa diagnosis in patients who underwent MRI-targeted and systematic biopsies. The model by Alberts demonstrated superior clinical utility and should be favored when determining the need for a prostate biopsy.

External validation of the CholeS conversion from laparoscopic to open cholecystectomy (CLOC) risk score in Aotearoa New Zealand: a validation study

BACKGROUND

Conversion of laparoscopic cholecystectomy to open is uncommon, but is associated with longer hospital stay and recovery. Prognosticating conversion may aid service planning and provision. We therefore aimed to assess the external validity of the largest risk score for operative conversion.

METHODS

CHOLENZ was a multicentre, prospective, national cohort study of cholecystectomy for benign biliary disease conducted by STRATA, a trainee-led collaborative network. Data were collected from patients undergoing cholecystectomy in New Zealand hospitals between 1 August and 30 October 2021 with 30-day follow-up. The Conversion from Laparoscopic to Open Cholecystectomy (CLOC) score from the CholeS study was assessed for external validity by interrogating its accuracy and calibration in the CHOLENZ dataset.

RESULTS

Of 1162 cholecystectomies started laparoscopically, 20 (1.7%) were converted to open in the CHOLENZ dataset. The CLOC score predicted 2.9% (IQR 1.3%-8.1%) would be converted. Area under the curve was 0.65 (95% 0.51-0.79) and calibration was acceptable with a Hosmer-Lemeshow p value of 0.45; with evidence of tendency to overestimate with interrogation of calibration across a continuous risk profile (intercept 1.27, slope 0.4). Sensitivity analysis with imputed data improved accuracy. Recalibration with the addition of body mass index, and preoperative bilirubin also improved accuracy to 0.86 (95% CI 0.78-0.95).

CONCLUSIONS

The CLOC score in its original form is not generalisable to the Aotearoa New Zealand setting and is therefore not suitable for clinical use in our local setting.

Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty

OBJECTIVE

This study aimed to externally validate a reported model for identifying patients requiring extended stay following lower limb arthroplasty in a new setting.

DESIGN

External validation of a previously reported prognostic model, using retrospective data.

SETTING

Medium-sized hospital orthopaedic department, Australia.

PARTICIPANTS

Electronic medical records were accessed for data collection between Sep-2019 and Feb-2020 and retrospective data extracted from 200 randomly selected total hip or knee arthroplasty patients.

INTERVENTION

Participants received total hip or knee replacement between 2-Feb-16 and 4-Apr-19. This study was a non-interventional retrospective study.

MAIN MEASURES

Model validation was assessed with discrimination, calibration on both original and adjusted forms of the candidate model. Decision curve analysis was conducted on the outputs of the adjusted model to determine net benefit at a predetermined decision threshold (0.5).

RESULTS

The original model performed poorly, grossly overestimating length of stay with mean calibration of -3.6 (95% confidence interval -3.9 to -3.2) and calibration slope of 0.52. Performance improved following adjustment of the model intercept and model coefficients (mean calibration 0.48, 95% confidence interval 0.16 to 0.80 and slope of 1.0), but remained poorly calibrated at low and medium risk threshold and net benefit was modest (three additional patients per hundred identified as at-risk) at the a-priori risk threshold.

CONCLUSIONS

External validation demonstrated poor performance when applied to a new patient population and would provide limited benefit for our institution. Implementation of predictive models for arthroplasty should include practical assessment of discrimination, calibration and net benefit at a clinically acceptable threshold.

Performance of the first-trimester Fetal Medicine Foundation competing risks model for preeclampsia prediction: an external validation study in Brazil

BACKGROUND

The current version of the Fetal Medicine Foundation competing risks model for preeclampsia prediction has not been previously validated in Brazil.

OBJECTIVE

This study aimed (1) to validate the Fetal Medicine Foundation combined algorithm for the prediction of preterm preeclampsia in the Brazilian population and (2) to describe the accuracy and calibration of the Fetal Medicine Foundation algorithm when considering the prophylactic use of aspirin by clinical criteria.

STUDY DESIGN

This was a cohort study, including consecutive singleton pregnancies undergoing preeclampsia screening at 11 to 14 weeks of gestation, examining maternal characteristics, medical history, and biophysical markers between October 2010 and December 2018 in a university hospital in Brazil. Risks were calculated using the 2018 version of the algorithm available on the Fetal Medicine Foundation website, and cases were classified as low or high risk using a cutoff of 1/100 to evaluate predictive performance. Expected and observed cases with preeclampsia according to the Fetal Medicine Foundation-estimated risk range (≥1 in 10; 1 in 11 to 1 in 50; 1 in 51 to 1 in 100; 1 in 101 to 1 in 150; and <1 in 150) were compared. After identifying high-risk pregnant women who used aspirin, the treatment effect of 62% reduction in preterm preeclampsia identified in the Combined Multimarker Screening and Randomized Patient Treatment with Aspirin for Evidence-Based Preeclampsia Prevention trial was used to evaluate the predictive performance adjusted for the effect of aspirin. The number of potentially unpreventable cases in the group without aspirin use was estimated.

RESULTS

Among 2749 pregnancies, preterm preeclampsia occurred in 84 (3.1%). With a risk cutoff of 1/100, the screen-positive rate was 25.8%. The detection rate was 71.4%, with a false positive rate of 24.4%. The area under the curve was 0.818 (95% confidence interval, 0.773-0.863). In the risk range ≥1/10, there is an agreement between the number of expected cases and the number of observed cases, and in the other ranges, the predicted risk was lower than the observed rates. Accounting for the effect of aspirin resulted in an increase in detection rate and positive predictive values and a slight decrease in the false positive rate. With 27 cases of preterm preeclampsia in the high-risk group without aspirin use, we estimated that 16 of these cases of preterm preeclampsia would have been avoided if this group had received prophylaxis.

CONCLUSION

In a high-prevalence setting, the Fetal Medicine Foundation algorithm can identify women who are more likely to develop preterm preeclampsia. Not accounting for the effect of aspirin underestimates the screening performance.

Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism

Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.

Cell-free DNA assay for malignancy classification of high-risk lung nodules

OBJECTIVE

Although low-dose computed tomography has been proven effective to reduce lung cancer-specific mortality, a considerable proportion of surgically resected high-risk lung nodules were still confirmed pathologically benign. There is an unmet need of a novel method for malignancy classification in lung nodules.

METHODS

We recruited 307 patients with high-risk lung nodules who underwent curative surgery, and 247 and 60 cases were pathologically confirmed malignant and benign lung lesions, respectively. Plasma samples from each patient were collected before surgery and performed low-depth (5×) whole-genome sequencing. We extracted cell-free DNA characteristics and determined radiomic features. We built models to classify the malignancy using our data and further validated models with 2 independent lung nodule cohorts.

RESULTS

Our models using one type of profile were able to distinguish lung cancer and benign lung nodules at an area under the curve metrics of 0.69 to 0.91 in the study cohort. Integrating all the 5 base models using cell-free DNA profiles, the cell-free DNA-based ensemble model achieved an area under the curve of 0.95 (95% CI, 0.92-0.97) in the study cohort and 0.98 (95% CI, 0.96-1.00) in the validation cohort. At a specificity of 95.0%, the sensitivity reached 80.0% in the study cohort. With the same threshold, the specificity and sensitivity had similar performances in both validation cohorts. Furthermore, the performance of area under the curve reached 0.97 in both the study and validation cohorts when considering the radiomic profile.

CONCLUSIONS

The cell-free DNA profiles-based method is an efficient noninvasive tool to distinguish malignancies and high-risk but pathologically benign lung nodules.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Development and external validation of the eFalls tool: a multivariable prediction model for the risk of ED attendance or hospitalisation with a fall or fracture in older adults

BACKGROUND

Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year.

METHODS

Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components, supplemented with variables informed by literature reviews and clinical expertise. Fall/fracture risk was modelled using multivariable logistic regression with a Least Absolute Shrinkage and Selection Operator penalty. Predictive performance was assessed through calibration, discrimination and clinical utility. Apparent, internal-external cross-validation and external validation performance were assessed across general practices and in clinically relevant subgroups.

RESULTS

The model's discrimination performance (c-statistic) was 0.72 (95% confidence interval, CI: 0.68 to 0.76) on internal-external cross-validation and 0.82 (95% CI: 0.80 to 0.83) on external validation. Calibration was variable across practices, with some over-prediction in the validation population (calibration-in-the-large, -0.87; 95% CI: -0.96 to -0.78). Clinical utility on external validation was improved after recalibration.

CONCLUSION

The eFalls prediction model shows good performance and could support proactive stratification for falls prevention services if appropriately embedded into primary care EHR systems.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies

Objectives

To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.

Design

Systematic review and meta-analysis of external validation studies.

Data sources

Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023.

Eligibility criteria for selecting studies

All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed.

Results

47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125).

Conclusions

The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed.

Systematic review registration

PROSPERO CRD42022373182.

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration

OBJECTIVES

In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.

STUDY DESIGN AND SETTING

A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).

RESULTS

The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.

CONCLUSION

We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

Illusory generalizability of clinical prediction models

It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model's success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.

Peer review of clinical and translational research manuscripts: Perspectives from statistical collaborators

Research articles in the clinical and translational science literature commonly use quantitative data to inform evaluation of interventions, learn about the etiology of disease, or develop methods for diagnostic testing or risk prediction of future events. The peer review process must evaluate the methodology used therein, including use of quantitative statistical methods. In this manuscript, we provide guidance for peer reviewers tasked with assessing quantitative methodology, intended to complement guidelines and recommendations that exist for manuscript authors. We describe components of clinical and translational science research manuscripts that require assessment including study design and hypothesis evaluation, sampling and data acquisition, interventions (for studies that include an intervention), measurement of data, statistical analysis methods, presentation of the study results, and interpretation of the study results. For each component, we describe what reviewers should look for and assess; how reviewers should provide helpful comments for fixable errors or omissions; and how reviewers should communicate uncorrectable and irreparable errors. We then discuss the critical concepts of transparency and acceptance/revision guidelines when communicating with responsible journal editors.

Artificial intelligence in detecting left atrial appendage thrombus by transthoracic echocardiography and clinical features: the Left Atrial Thrombus on Transoesophageal Echocardiography (LATTEE) registry

AIMS

Transoesophageal echocardiography (TOE) is often performed before catheter ablation or cardioversion to rule out the presence of left atrial appendage thrombus (LAT) in patients on chronic oral anticoagulation (OAC), despite associated discomfort. A machine learning model [LAT-artificial intelligence (AI)] was developed to predict the presence of LAT based on clinical and transthoracic echocardiography (TTE) features.

METHODS AND RESULTS

Data from a 13-site prospective registry of patients who underwent TOE before cardioversion or catheter ablation were used. LAT-AI was trained to predict LAT using data from 12 sites (n = 2827) and tested externally in patients on chronic OAC from two sites (n = 1284). Areas under the receiver operating characteristic curve (AUC) of LAT-AI were compared with that of left ventricular ejection fraction (LVEF) and CHA2DS2-VASc score. A decision threshold allowing for a 99% negative predictive value was defined in the development cohort. A protocol where TOE in patients on chronic OAC is performed depending on the LAT-AI score was validated in the external cohort. In the external testing cohort, LAT was found in 5.5% of patients. LAT-AI achieved an AUC of 0.85 [95% confidence interval (CI): 0.82-0.89], outperforming LVEF (0.81, 95% CI 0.76-0.86, P < .0001) and CHA2DS2-VASc score (0.69, 95% CI: 0.63-0.7, P < .0001) in the entire external cohort. Based on the proposed protocol, 40% of patients on chronic OAC from the external cohort would safely avoid TOE.

CONCLUSION

LAT-AI allows accurate prediction of LAT. A LAT-AI-based protocol could be used to guide the decision to perform TOE despite chronic OAC.

Derivation and External Validation of the Ottawa Bloodstream Infection Model for Acutely Ill Adults

BACKGROUND

Knowing the probability that patients have a bloodstream infection (BSI) could influence the ordering of blood cultures and interpretation of their preliminary results. Many previous BSI probability models have limited applicability and accuracy. This study used currently recommended modeling techniques and a large sample to derive and validate the Ottawa BSI Model.

METHODS

At a tertiary care teaching hospital, we retrieved a random sample of 4180 adults having blood cultures in our emergency department or during the initial 48 h of the encounter. Variable selection was based on clinical experience and a systematic review of previous model performance. Model performance was measured in a temporal external validation group of 4680 patients.

RESULTS

A total of 327 derivation patients had a BSI (8.0%). BSI risk increased with increased number of culture sets (2 sets: adjusted odds ratio [aOR] 1.52 [1.10-2.11]; 3 sets: 1.99 [0.86-4.58]); with indwelling catheter (aOR 2.07 [1.34-3.20); with increasing temperature, heart rate, and neutrophil-lymphocyte ratio; and with decreasing systolic blood pressure, platelet count, urea-creatinine ratio, and estimated glomerular filtration rate. In the temporal external validation group, model discrimination was good (c-statistic 0.71 [0.69-0.74]) and calibration was very good (integrated calibration index .016 [.010-.024]). Exclusion of validation patients with acute SARS-CoV-2 infection improved discrimination slightly (c-statistic 0.73 [0.69-0.76]).

CONCLUSIONS

The Ottawa BSI Model uses commonly available data to return an expected BSI probability for acutely ill patients. However, it cannot exclude BSI and its complexity requires computational assistance to use.

Development and validation of a model to predict acute kidney injury following high-dose methotrexate in patients with lymphoma

STUDY OBJECTIVE

To develop and validate a model for predicting acute kidney injury (AKI) after high-dose methotrexate (HDMTX) exposure.

DESIGN

Retrospective analysis.

SETTING

Multisite integrated health system throughout Minnesota and Wisconsin.

PATIENTS

Adult patients with lymphoma who received HDMTX as a 4-h infusion.

MEASUREMENTS AND MAIN RESULTS

LASSO methodology was used to identify factors available at the outset of therapy that predicted incident AKI within 7 days following HDMTX. The model was then validated in an independent cohort. The incidence of AKI within 7 days following HDMTX was 21.6% (95% confidence interval (CI) 18.4%-24.8%) in the derivation cohort (435 unique patients who received a total of 1642 doses of HDMTX) and 15.6% (95% CI 5.3%-24.8%) in the validation cohort (55 unique patients who received a total of 247 doses of HDMTX). Factors significantly associated with AKI after HDMTX in the multivariable model included age ≥ 55 years, male sex, and lower HDMTX dose number. Other factors that were not found to be significantly associated with AKI on multivariable analysis, but were included in the final model, were body surface area, Charlson Comorbidity Index, and estimated glomerular filtration rate. The c-statistic of the model was 0.72 (95% CI 0.69-0.75) in the derivation cohort and 0.72 (95% CI 0.60-0.84) in the validation cohort.

CONCLUSION

This model utilizing identified sociodemographic and clinical factors is predictive of AKI following HDMTX administration in adult patients with lymphoma.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Validation of the ADFICE_IT Models for Predicting Falls and Recurrent Falls in Geriatric Outpatients

OBJECTIVES

Before being used in clinical practice, a prediction model should be tested in patients whose data were not used in model development. Previously, we developed the ADFICE_IT models for predicting any fall and recurrent falls, referred as Any_fall and Recur_fall. In this study, we externally validated the models and compared their clinical value to a practical screening strategy where patients are screened for falls history alone.

DESIGN

Retrospective, combined analysis of 2 prospective cohorts.

SETTING AND PARTICIPANTS

Data were included of 1125 patients (aged ≥65 years) who visited the geriatrics department or the emergency department.

METHODS

We evaluated the models' discrimination using the C-statistic. Models were updated using logistic regression if calibration intercept or slope values deviated significantly from their ideal values. Decision curve analysis was applied to compare the models' clinical value (ie, net benefit) against that of falls history for different decision thresholds.

RESULTS

During the 1-year follow-up, 428 participants (42.7%) endured 1 or more falls, and 224 participants (23.1%) endured a recurrent fall (≥2 falls). C-statistic values were 0.66 (95% CI 0.63-0.69) and 0.69 (95% CI 0.65-0.72) for the Any_fall and Recur_fall models, respectively. Any_fall overestimated the fall risk and we therefore updated only its intercept whereas Recur_fall showed good calibration and required no update. Compared with falls history, Any_fall and Recur_fall showed greater net benefit for decision thresholds of 35% to 60% and 15% to 45%, respectively.

CONCLUSIONS AND IMPLICATIONS

The models performed similarly in this data set of geriatric outpatients as in the development sample. This suggests that fall-risk assessment tools that were developed in community-dwelling older adults may perform well in geriatric outpatients. We found that in geriatric outpatients the models have greater clinical value across a wide range of decision thresholds compared with screening for falls history alone.

External validation and updating of prognostic prediction models for nonrecovery among older adults seeking primary care for back pain

ABSTRACT

Prognostic prediction models for 3 different definitions of nonrecovery were developed in the Back Complaints in the Elders study in the Netherlands. The models' performance was good (optimism-adjusted area under receiver operating characteristics [AUC] curve ≥0.77, R2 ≥0.3). This study aimed to assess the external validity of the 3 prognostic prediction models in the Norwegian Back Complaints in the Elders study. We conducted a prospective cohort study, including 452 patients aged ≥55 years, seeking primary care for a new episode of back pain. Nonrecovery was defined for 2 outcomes, combining 6- and 12-month follow-up data: Persistent back pain (≥3/10 on numeric rating scale) and persistent disability (≥4/24 on Roland-Morris Disability Questionnaire). We could not assess the third model (self-reported nonrecovery) because of substantial missing data (>50%). The models consisted of biopsychosocial prognostic factors. First, we assessed Nagelkerke R2 , discrimination (AUC) and calibration (calibration-in-the-large [CITL], slope, and calibration plot). Step 2 was to recalibrate the models based on CITL and slope. Step 3 was to reestimate the model coefficients and assess if this improved performance. The back pain model demonstrated acceptable discrimination (AUC 0.74, 95% confidence interval: 0.69-0.79), and R2 was 0.23. The disability model demonstrated excellent discrimination (AUC 0.81, 95% confidence interval: 0.76-0.85), and R2 was 0.35. Both models had poor calibration (CITL <0, slope <1). Recalibration yielded acceptable calibration for both models, according to the calibration plots. Step 3 did not improve performance substantially. The recalibrated models may need further external validation, and the models' clinical impact should be assessed.

Derivation and external validation of machine-learning models for risk stratification in chest pain with normal troponin

AIMS

Risk stratification of patients with chest pain and a high-sensitivity cardiac troponin T (hs-cTnT) concentration

METHODS AND RESULTS

Four machine-learning-based models and one logistic regression (LR) model were trained on 4075 patients (single-centre Spanish cohort) and externally validated on 3609 patients (international prospective Advantageous Predictors of Acute Coronary syndromes Evaluation cohort). Models were compared with GRACE and HEART scores and a single undetectable hs-cTnT-based strategy (u-cTn; hs-cTnT < 5 ng/L and time from symptoms onset >180 min). Probability thresholds for safe discharge were derived in the derivation cohort. The endpoint occurred in 105 (2.6%) patients in the training set and 98 (2.7%) in the external validation set. Gradient boosting full (GBf) showed the best discrimination (area under the curve = 0.808). Calibration was good for the reduced neural network and LR models. Gradient boosting full identified the highest proportion of patients for safe discharge (36.7 vs. 23.4 vs. 27.2%; GBf vs. LR vs. u-cTn, respectively) with similar safety (missed endpoint per 1000 patients: 2.2 vs. 3.5 vs. 3.1, respectively). All derived models were superior to the HEART and GRACE scores (P < 0.001).

CONCLUSION

Machine-learning and LR prediction models were superior to the HEART, GRACE, and u-cTn for risk stratification of patients with chest pain and a baseline hs-cTnT

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov number, NCT00470587, https://clinicaltrials.gov/ct2/show/NCT00470587.

Collapse

Key Words

• Machine learning
• Myocardial infarction
• Prediction
• Troponin

Collapse

MESH Headings

• Humans
• Troponin
• Prospective Studies
• Biomarkers
• Troponin T
• Chest Pain/diagnosis
• Chest Pain/etiology
• Risk Assessment

Collapse

Grants

• Swiss National Science Foundation

Collapse

Affiliation(s)

Agustín Fernández-Cisnal Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
Pedro Lopez-Ayala Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
Ernesto Valero Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
Luca Koechlin Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
Arturo Catarralá Clinical Biochemistry Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), València 46010, Spain
Jasper Boeddinghaus Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
José Noceda Emergency Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), València 46010, Spain
Thomas Nestelberger Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
Òscar Miró Emergency Department, Hospital Clinic, Barcelona, Catalonia, Spain
Núñez Julio Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
Christian Mueller Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
Juan Sanchis Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain

Collapse

Development and External Validation of Individualized Prediction Models for Pain Intensity Outcomes in Patients With Neck Pain, Low Back Pain, or Both in Primary Care Settings

OBJECTIVE

The purpose of this study was to develop and externally validate multivariable prediction models for future pain intensity outcomes to inform targeted interventions for patients with neck or low back pain in primary care settings.

METHODS

Model development data were obtained from a group of 679 adults with neck or low back pain who consulted a participating United Kingdom general practice. Predictors included self-report items regarding pain severity and impact from the STarT MSK Tool. Pain intensity at 2 and 6 months was modeled separately for continuous and dichotomized outcomes using linear and logistic regression, respectively. External validation of all models was conducted in a separate group of 586 patients recruited from a similar population with patients' predictor information collected both at point of consultation and 2 to 4 weeks later using self-report questionnaires. Calibration and discrimination of the models were assessed separately using STarT MSK Tool data from both time points to assess differences in predictive performance.

RESULTS

Pain intensity and patients reporting their condition would last a long time contributed most to predictions of future pain intensity conditional on other variables. On external validation, models were reasonably well calibrated on average when using tool measurements taken 2 to 4 weeks after consultation (calibration slope = 0.848 [95% CI = 0.767 to 0.928] for 2-month pain intensity score), but performance was poor using point-of-consultation tool data (calibration slope for 2-month pain intensity score of 0.650 [95% CI = 0.549 to 0.750]).

CONCLUSION

Model predictive accuracy was good when predictors were measured 2 to 4 weeks after primary care consultation, but poor when measured at the point of consultation. Future research will explore whether additional, nonmodifiable predictors improve point-of-consultation predictive performance.

IMPACT

External validation demonstrated that these individualized prediction models were not sufficiently accurate to recommend their use in clinical practice. Further research is required to improve performance through inclusion of additional nonmodifiable risk factors.

Prognostic models predicting transition to psychotic disorder using blood-based biomarkers: a systematic review and critical appraisal

Accumulating evidence suggests individuals with psychotic disorder show abnormalities in metabolic and inflammatory processes. Recently, several studies have employed blood-based predictors in models predicting transition to psychotic disorder in risk-enriched populations. A systematic review of the performance and methodology of prognostic models using blood-based biomarkers in the prediction of psychotic disorder from risk-enriched populations is warranted. Databases (PubMed, EMBASE and PsycINFO) were searched for eligible texts from 1998 to 15/05/2023, which detailed model development or validation studies. The checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was used to guide data extraction from eligible texts and the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias and applicability of the studies. A narrative synthesis of the included studies was performed. Seventeen eligible studies were identified: 16 eligible model development studies and one eligible model validation study. A wide range of biomarkers were assessed, including nucleic acids, proteins, metabolites, and lipids. The range of C-index (area under the curve) estimates reported for the models was 0.67-1.00. No studies assessed model calibration. According to PROBAST criteria, all studies were at high risk of bias in the analysis domain. While a wide range of potentially predictive biomarkers were identified in the included studies, most studies did not account for overfitting in model performance estimates, no studies assessed calibration, and all models were at high risk of bias according to PROBAST criteria. External validation of the models is needed to provide more accurate estimates of their performance. Future studies which follow the latest available methodological and reporting guidelines and adopt strategies to accommodate required sample sizes for model development or validation will clarify the value of including blood-based biomarkers in models predicting psychosis.

Self-Perceived Clinical Competence of Nurses in Different Working Experiences: A Cross-Sectional Study

BACKGROUND

Competence is an essential concept for measuring nurses' performance in terms of effectiveness and quality. To this end, our analysis highlighted the process of acquiring competencies, their self-evaluation into clinical practice, and how their proficiency levels change throughout the nursing career. In detail, this research explored nurses' perceived level of competence and the factors that influence it in different contexts.

METHODS

A cross-sectional survey using a structured questionnaire to assess the nursing participants' perception of their competencies in different clinical settings was accomplished.

RESULTS

A descriptive and bivariate analysis was performed on 431 nurses. Most respondents assessed their level of competence to be higher than their roles required. The Kruskal-Wallis test confirmed that nursing experience was a relevant factor influencing nursing competencies.

CONCLUSIONS

We suggest improving the competence of practicing nurses, using experience as a measurable effect of their development.

Internal and External Validation of the Generalizability of Machine Learning Algorithms in Predicting Non-home Discharge Disposition Following Primary Total Knee Joint Arthroplasty

BACKGROUND

Nonhome discharge disposition following primary total knee arthroplasty (TKA) is associated with a higher rate of complications and constitutes a socioeconomic burden on the health care system. While existing algorithms predicting nonhome discharge disposition varied in degrees of mathematical complexity and prediction power, their capacity to generalize predictions beyond the development dataset remains limited. Therefore, this study aimed to establish the machine learning model generalizability by performing internal and external validations using nation-scale and institutional cohorts, respectively.

METHODS

Four machine learning models were trained using the national cohort. Recursive feature elimination and hyper-parameter tuning were applied. Internal validation was achieved through five-fold cross-validation during model training. The trained models' performance was externally validated using the institutional cohort and assessed by discrimination, calibration, and clinical utility.

RESULTS

The national (424,354 patients) and institutional (10,196 patients) cohorts had non-home discharge rates of 19.4 and 36.4%, respectively. The areas under the receiver operating curve of the model predictions were 0.83 to 0.84 during internal validation and increased to 0.88 to 0.89 during external validation. Artificial neural network and histogram-based gradient boosting elicited the best performance with a mean area under the receiver operating curve of 0.89, calibration slope of 1.39, and Brier score of 0.14, which indicated that the two models were robust in distinguishing non-home discharge and well-calibrated with accurate predictions of the probabilities. The low inter-dataset similarity indicated reliable external validation. Length of stay, age, body mass index, and sex were the strongest predictors of discharge destination after primary TKA.

CONCLUSION

The machine learning models demonstrated excellent predictive performance during both internal and external validations, supporting their generalizability across different patient cohorts and potential applicability in the clinical workflow.

Predicting Death or Disability after Surgery in the Older Adult

BACKGROUND

Older patients are vulnerable to developing new or worsening disability after surgery. Despite this, patient or surgical characteristics predisposing to postoperative disability are poorly defined. The aim of the study was to develop and validate a model, subsequently transformed to point-score form, to predict 6-month death or disability in older patients after surgery.

METHODS

The authors built a prospective, single-center registry to develop and validate the prediction model. The registry included patients 70 yr of age or older undergoing elective and nonelective, cardiac and noncardiac surgery between May 25, 2017, and February 11, 2021, and combined clinical data from the electronic medical record, hospital administrative data (International Classification of Diseases, Tenth Revision, Australian Modification codes) and World Health Organization (Geneva, Switzerland) Disability Assessment Schedule data collected directly from the patients. Death or disability was defined as being dead or having a World Health Organization Disability Assessment Schedule score 16% or greater. Included patients were randomly divided into model development (70%) and internal validation (30%) cohorts. Once constructed, the logistic regression and point-score models were assessed using the internal validation cohort and an external validation cohort comprising data from a separate randomized trial.

RESULTS

Of 2,176 patients who completed the World Health Organization Disability Assessment Schedule immediately before surgery, 927 (43%) patients were disabled, and 413 (19%) had significant disability. By 6 months after surgery, 1,640 patients (75%) had data available for the primary outcome analysis. Of these patients, 195 (12%) patients had died, and 691 (42%) were dead or disabled. The developed point-score model included the preoperative World Health Organization Disability Assessment Schedule score, patient age, dementia, and chronic kidney disease. The point score model retained good discrimination in the internal (area under the curve, 0.74; 95% CI, 0.69 to 0.79) and external (area under the curve, 0.77; 95% CI, 0.74 to 0.80) validation data sets.

CONCLUSIONS

The authors developed and validated a point score model to predict death or disability in older patients after surgery.

EDITOR’S PERSPECTIVE

Prediction of postoperative hypokalemia in patients with oral cancer undergoing en bloc cancer resection: a retrospective cohort study

BACKGROUND

The factors associated with postoperative hypokalemia in patients with oral cancer remain unclear. We determined the preoperative factors associated with postoperative hypokalemia in patients with oral cancer following en bloc cancer resection and established a nomogram for postoperative hypokalemia prediction.

METHODS

Data from 381 patients with oral cancer who underwent en bloc cancer resection were retrospectively analyzed. Univariate and multivariate analyses were performed to identify the risk factors for postoperative hypokalemia. We used receiver operating characteristic (ROC) curves to quantify the factors' effectiveness. A nomogram was created to show each predictor's relative weight and the likelihood of postoperative hypokalemia development. The multinomial regression model's effectiveness was also evaluated.

RESULTS

Preoperative factors, including sex, preoperative serum potassium level, and preoperative platelet-to-lymphocyte ratio (PLR), were significantly associated with postoperative hypokalemia. Based on the ROC curve, the preoperative serum potassium and PLR cut-off levels were 3.98 mmol/L and 117, respectively. Further multivariate analysis indicated that female sex, preoperative serum potassium level < 3.98 mmol/L, and preoperative PLR ≥ 117 were independently associated with postoperative hypokalemia. We constructed a predictive nomogram with all these factors for the risk of postoperative hypokalemia with good discrimination and internal validation.

CONCLUSIONS

The predictive nomogram for postoperative hypokalemia risk constructed with these factors had good discrimination and internal validation. The developed nomogram will add value to these independent risk factors that can be identified at admission in order to predict postoperative hypokalemia.

Training and testing of a gradient boosted machine learning model to predict adverse outcome in patients presenting to emergency departments with suspected covid-19 infection in a middle-income setting

COVID-19 infection rates remain high in South Africa. Clinical prediction models may be helpful for rapid triage, and supporting clinical decision making, for patients with suspected COVID-19 infection. The Western Cape, South Africa, has integrated electronic health care data facilitating large-scale linked routine datasets. The aim of this study was to develop a machine learning model to predict adverse outcome in patients presenting with suspected COVID-19 suitable for use in a middle-income setting. A retrospective cohort study was conducted using linked, routine data, from patients presenting with suspected COVID-19 infection to public-sector emergency departments (EDs) in the Western Cape, South Africa between 27th August 2020 and 31st October 2021. The primary outcome was death or critical care admission at 30 days. An XGBoost machine learning model was trained and internally tested using split-sample validation. External validation was performed in 3 test cohorts: Western Cape patients presenting during the Omicron COVID-19 wave, a UK cohort during the ancestral COVID-19 wave, and a Sudanese cohort during ancestral and Eta waves. A total of 282,051 cases were included in a complete case training dataset. The prevalence of 30-day adverse outcome was 4.0%. The most important features for predicting adverse outcome were the requirement for supplemental oxygen, peripheral oxygen saturations, level of consciousness and age. Internal validation using split-sample test data revealed excellent discrimination (C-statistic 0.91, 95% CI 0.90 to 0.91) and calibration (CITL of 1.05). The model achieved C-statistics of 0.84 (95% CI 0.84 to 0.85), 0.72 (95% CI 0.71 to 0.73), and 0.62, (95% CI 0.59 to 0.65) in the Omicron, UK, and Sudanese test cohorts. Results were materially unchanged in sensitivity analyses examining missing data. An XGBoost machine learning model achieved good discrimination and calibration in prediction of adverse outcome in patients presenting with suspected COVID19 to Western Cape EDs. Performance was reduced in temporal and geographical external validation.

Systematic review finds risk of bias and applicability concerns for models predicting central line-associated bloodstream infection

OBJECTIVES

To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients.

STUDY DESIGN AND SETTING

Systematic review of literature in PubMed, Embase, Web of Science Core Collection, and Scopus up to July 10, 2023. Two authors independently appraised risk models using CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and assessed their risk of bias and applicability using Prediction model Risk Of Bias ASsessment Tool (PROBAST).

RESULTS

Sixteen studies were included, describing 37 models. When studies presented multiple algorithms, we focused on the model that was selected as the best by the study authors. Eventually we appraised 19 models, among which 15 were regression models and four machine learning models. All models were at a high risk of bias, primarily due to inappropriate proxy outcomes, predictors that are unavailable at prediction time in clinical practice, inadequate sample size, negligence of missing data, lack of model validation, and absence of calibration assessment. 18 out of 19 models had a high concern for applicability, one model had unclear concern for applicability due to incomplete reporting.

CONCLUSION

We did not identify a prediction model of potential clinical use. There is a pressing need to develop an applicable model for CLA-BSI.

Development and Validation of Multivariable Prediction Models for In-Hospital Death, 30-Day Death, and Change in Residence After Hip Fracture Surgery and the "Stratify-Hip" Algorithm

BACKGROUND

To develop and validate the stratify-hip algorithm (multivariable prediction models to predict those at low, medium, and high risk across in-hospital death, 30-day death, and residence change after hip fracture).

METHODS

Multivariable Fine-Gray and logistic regression of audit data linked to hospital records for older adults surgically treated for hip fracture in England/Wales 2011-14 (development n = 170 411) and 2015-16 (external validation, n = 90 102). Outcomes included time to in-hospital death, death at 30 days, and time to residence change. Predictors included age, sex, pre-fracture mobility, dementia, and pre-fracture residence (not for residence change). Model assumptions, performance, and sensitivity to missingness were assessed. Models were incorporated into the stratify-hip algorithm assigning patients to overall low (low risk across outcomes), medium (low death risk, medium/high risk of residence change), or high (high risk of in-hospital death, high/medium risk of 30-day death) risk.

RESULTS

For complete-case analysis, 6 780 of 141 158 patients (4.8%) died in-hospital, 8 693 of 149 258 patients (5.8%) died by 30 days, and 4 461 of 119 420 patients (3.7%) had residence change. Models demonstrated acceptable calibration (observed:expected ratio 0.90, 0.99, and 0.94), and discrimination (area under curve 73.1, 71.1, and 71.5; Brier score 5.7, 5.3, and 5.6) for in-hospital death, 30-day death, and residence change, respectively. Overall, 31%, 28%, and 41% of patients were assigned to overall low, medium, and high risk. External validation and missing data analyses elicited similar findings. The algorithm is available at https://stratifyhip.co.uk.

CONCLUSIONS

The current study developed and validated the stratify-hip algorithm as a new tool to risk stratify patients after hip fracture.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

BACKGROUND

Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome.

METHODS

We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size.

RESULTS

A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84).

CONCLUSIONS

Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.