Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

Development of a multicentre cohort study to understand the role of MRI and ultrasound in the diagnosis of acute haematogenous bone and joint infection in children (the PIC Bone study) : a study protocol

Aims

Bone and joint infections (BJI) in children are rare but can be serious. Differentiating BJI from other conditions with similar symptoms is critical. Advanced imaging (ultrasound scans (USS) and MRI) is often required to confirm the diagnosis. The differing merits of imaging type and regional variation in access to advanced imaging can lead to diagnostic uncertainty and treatment variation. The aim of this study is to evaluate the diagnostic accuracy of MRI and USS for the investigation of BJI in children, and develop and validate prediction models to aid the diagnosis of BJI in children. A nested qualitative sub-study will explore acceptability of the imaging to children, parents, and health practitioners.

Methods

A multicentre retrospective cohort of children (aged < 16 years) with suspected diagnosis of BJI will be used to estimate the diagnostic accuracy of the two imaging methods and develop the prediction models. The models will be evaluated in a second cohort of prospectively recruited children. Diagnostic test accuracy will be estimated overall, and separately for children aged under and over five years. The prediction models will be fit using logistic regression, with candidate predictors chosen based on clinical plausibility and from a review of the literature. Continuous predictors will be examined for non-linearity with confirmed BJI using fractional polynomials. Multiple imputation will be used to replace missing values. Internal validation will be carried out using bootstrapping. Model performance will be assessed with discrimination and calibration.

Discussion

Ethical approval for this study (registration: ISRCTN15471635) was granted (REC reference 23/WM/0027). Informed consent is being obtained from participants in the prospective cohort and the qualitative sub-study. Study findings will be published in an open access journal and presented at relevant national and international conferences. Relevant charities and associations are being engaged to promote awareness of the project.

Early Immune Checkpoint Inhibitor Administration Increases the Risk of Radiation-Induced Pneumonitis in Patients with Stage III Unresectable NSCLC Undergoing Chemoradiotherapy

BACKGROUND/OBJECTIVES

The PACIFIC trial showed that immune checkpoint inhibitors (ICI) administered after concurrent chemoradiotherapy (cCRT) significantly improve survival in stage III unresectable non-small cell lung cancer (NSCLC). However, the optimal timing of ICI administration with cCRT is still debated, with concerns about increased risks of adverse effects, particularly radiation-induced pneumonitis (RP), from combining radiotherapy and immunotherapy.

METHODS

A search of multiple databases identified studies on stage III unresectable NSCLC patients receiving cCRT and ICI. A meta-analysis was performed utilizing the meta package in R software. Furthermore, data from 170 patients treated at Shandong Cancer Hospital and Institute between 2019 and 2023 were analyzed to assess RP following cCRT and ICI treatment.

RESULTS

The meta-analysis revealed that the incidences of ≥grade 2 RP were 25.3%, 24.3%, and 45.3% in the ICI following cCRT group, the ICI concurrent with cCRT group, and the ICI prior to cCRT group, respectively. The ICI prior to cCRT group exhibited significantly elevated rates. In the clinical retrospective study, ≥grade 2 RP was more prevalent in the ICI concurrent with cCRT group (HR: 2.258, 95% CI: 1.135-4.492, p = 0.020) and the ICI prior to cCRT group (HR: 2.843, 95% CI: 1.453-5.561, p = 0.002) compared with the ICI following cCRT group. Furthermore, a shorter interval between treatments correlates with an increased incidence of RP.

CONCLUSIONS

Advancing the timing of ICI administration is associated with an increased incidence of ≥grade 2 RP following cCRT in patients with stage III unresectable NSCLC.

Clinical prediction models to guide treatment of periprosthetic joint infections: A systematic review and meta-analysis

BACKGROUND

Several clinical prediction models that aim to guide decisions about the management of periprosthetic joint infections (PJI) have been developed. While some models have been recommended for use in clinical settings, their suitability remains uncertain.

METHODS

We systematically reviewed and critically appraised all multivariable prediction models for the treatment of PJI. We searched MEDLINE, EMBASE, Web of Science, and Google Scholar from inception until March 1st, 2024 and included studies that developed or validated models that predict the outcome of PJI. We used PROBAST (Prediction model Risk Of Bias ASsessment Tool) to assess the risk of bias and applicability. Model performance estimates were pooled via random effect meta-analysis.

RESULTS

Thirteen predictive models and seven external validations were identified. Methodological issues were identified in all studies. Pooled estimates indicated that the KLIC (Kidney, Liver, Index surgery, Cemented prosthesis, C-reactive protein) score had fair discriminative performance (pooled c-statistic 0.62, 95% CI 0.55 to 0.69). Both the τ2 (0.02) and I2 (33.4) estimates indicated that between study heterogeneity was minimal. Meta-analysis indicated Shohat et al's model had good discriminative performance (pooled c-statistic 0.74, 95% CI 0.57 to 0.85). Both the τ2 (0.0) and I2 (0.0) indicated that between study heterogeneity was minimal.

CONCLUSIONS

Clinicians should be aware of limitations in the methods used to develop available models to predict outcomes of PJI. As no models have consistently demonstrated adequate performance across external validation studies, it remains unclear if any available models would provide reliable information if used to guide clinical decision-making.

External validation and comparison of six cardiovascular risk prediction models in the Prospective Urban Rural Epidemiology (PURE)-Colombia study

AIMS

To externally validate the SCORE2, AHA/ACC pooled cohort equation (PCE), Framingham Risk Score (FRS), Non-Laboratory INTERHEART Risk Score (NL-IHRS), Globorisk-LAC, and WHO prediction models and compare their discrimination and calibration capacity.

METHODS AND RESULTS

Validation in individuals aged 40-69 years with at least 10 years of follow-up and without baseline use of statins or cardiovascular diseases from the Prospective Urban Rural Epidemiology (PURE)-Colombia prospective cohort study. For discrimination, the C-statistic, and receiver operating characteristic curves with the integrated area under the curve (AUCi) were used and compared. For calibration, the smoothed time-to-event method was used, choosing a recalibration factor based on the integrated calibration index (ICI). In the NL-IHRS, linear regressions were used. In 3802 participants (59.1% women), baseline risk ranged from 4.8% (SCORE2 women) to 55.7% (NL-IHRS). After a mean follow-up of 13.2 years, 234 events were reported (4.8 cases per 1000 person-years). The C-statistic ranged between 0.637 (0.601-0.672) in NL-IHRS and 0.767 (0.657-0.877) in AHA/ACC PCE. Discrimination was similar between AUCi. In women, higher over-prediction was observed in the Globorisk-LAC (61%) and WHO (59%). In men, higher over-prediction was observed in FRS (72%) and AHA/ACC PCE (71%). Overestimations were corrected after multiplying by a factor derived from the ICI.

CONCLUSION

Six prediction models had a similar discrimination capacity, supporting their use after multiplying by a correction factor. If blood tests are unavailable, NL-IHRS is a reasonable option. Our results suggest that these models could be used in other countries of Latin America after correcting the overestimations with a multiplying factor.

External validation of the eCura system and comparison with the W-eCura score for predicting lymph node metastasis after non-curative endoscopic submucosal dissection for early gastric cancer: a multicenter retrospective cohort study

BACKGROUND

The eCura system is a widely used risk-scoring model for predicting lymph node metastasis (LNM) after non-curative endoscopic submucosal dissection (ESD) for early gastric cancer (EGC), but its external validation is limited. Recently, the W-eCura score, a modified version, was proposed. We aimed to validate the eCura system and compare its discriminatory performance with the W-eCura score.

METHODS

A multicenter retrospective study was conducted using data from 19 Japanese institutions. The patients who underwent ESD for EGC followed by gastrectomy with lymph node dissection were included. The predictive performance of the eCura system, including calibration and discrimination, was evaluated and its discrimination was compared with the W-eCura score.

RESULTS

Among 901 eligible patients, 65 cases (7.2%) showed LNM. The eCura system demonstrated good calibration, with a calibration-in-the-large of -0.008 (95% confidence interval [CI] -0.024-0.010), an observed-to-expected ratio of 0.905 (95% CI 0.707-1.121), and a calibration slope of 0.975 (95% CI 0.692-1.257). Discrimination was also good, with a C-statistic of 0.741 (95% CI 0.676-0.806). In patients evaluable for both systems, the C-statistics for the eCura system and W-eCura score were 0.745 (95% CI 0.675-0.816) and 0.750 (95% CI 0.684-0.817), respectively, showing no significant difference (P = 0.547).

CONCLUSIONS

The eCura system was validated as a reliable tool for predicting LNM following ESD in real-world clinical settings.

Statistical primer: sample size considerations for developing and validating clinical prediction models

Clinical prediction models are statistical models or machine learning algorithms that combine information on a set of predictor variables about an individual to estimate their risk of a given clinical outcome. It is crucial to ensure that the sample size of the data used to develop or validate a clinical prediction model is large enough. If the data are inadequate, developed models can be unstable and estimates of predictive performance imprecise. This can lead to models that are unfit or even harmful for clinical practice. Recently, there have been a series of sample size formulae developed to estimate the minimum required sample size for prediction model development or external validation. The aim of this statistical primer is to provide an overview of these criteria, describe what information is required to make the calculations and illustrate their implementation through worked examples. The software that is available to implement the sample size criteria is reviewed, and code is provided for all the worked examples.

External Validation of CRASH Prognostic Model in an Urban Tertiary Care Public University Hospital

INTRODUCTION

Trauma represents 9% of global mortality, where traumatic brain injuries are the leading cause in low-middle income countries, most commonly due to road traffic injuries. The multicenter randomized controlled trial CRASH (corticosteroid randomization after significant head injury) published a prediction model to estimate prognosis in traumatic brain injury patients. This prediction model was derived based on data from high-, low-, and middle-income countries. The external validity of this prediction model was not assessed in low and middle-income countries. To fill this gap, we aim to external validate the CRASH prediction model in traumatic brain injury (TBI) patients in India, a lower-middle-income country.

METHODS

We conducted a prospective observational study at the General Surgery department of an urban tertiary care hospital in India. We collected data on the 14-d mortality and 6-mo unfavorable outcomes in patients with TBI. Calibration and discrimination of the CRASH models (basic and computed tomography [CT] model) comparing the observed and predicted outcomes using logistic regression, and area under the curve was analyzed to validate the model.

RESULTS

In this study, 417 patients with the median age of 40 y and age range of 18-95 y were evaluated. There was no significant difference between the calibration of the models in prediction of a 14-d mortality (basic P = 0.082, CT P = 0.067) and 6-mo unfavorable outcome (basic P = 0.688, CT P = 0.204). The area under the receiver operating characteristic curve in basic and CT models in prediction of 14-d mortality were 0.885 and 0.885 respectively. In addition, the area under the receiver operating characteristic curve in basic and CT models in prediction of 6-mo unfavorable outcome were 0.901 and 0.896, respectively.

CONCLUSIONS

The results of this study showed that the CRASH basic and CT model both accurately predict 14 d mortality and 6 mo unfavorable outcomes of TBI patients in an urban tertiary care public university hospital in India.

Biparietal diameter for first-trimester pregnancy dating: multicenter cohort study

OBJECTIVE

To evaluate the accuracy of fetal biparietal diameter (BPD) measurement in comparison with crown-rump length (CRL) measurement for pregnancy dating at 11-13 weeks' gestation.

METHODS

This was a retrospective multicenter cohort study performed in five maternity units in Spain, the UK, Belgium and Bulgaria between January 2011 and December 2019. We included all women who attended a routine ultrasound examination at 11 + 0 to 13 + 6 weeks who had a singleton pregnancy with a viable non-malformed fetus/neonate and ultrasound-derived measurements for both CRL and BPD, along with a comprehensive record of pregnancy outcomes. We developed a formula for pregnancy dating based on BPD using data from pregnancies conceived via in-vitro fertilization (IVF) by applying a simple linear regression. We validated this formula both internally and externally and compared it with the most commonly used formulae (Robinson's CRL-based and Kustermann's BPD-based formulae) through utilization of the Euclidean distance, relative absolute error and mean squared error. We also examined the rate of induction of labor for post-term pregnancy based on dating using each of the formulae.

RESULTS

A total of 49 492 women were included in the study, comprising 47 223 (95.4%) who conceived spontaneously and 2269 (4.6%) who conceived via IVF. In the internal validation performed using data from IVF pregnancies, our newly developed formula showed no significant difference when compared with the true gestational age calculated using conception date, with a mean difference of 0.0006 (95% CI, -0.09 to 0.09) days. In contrast, the mean difference of Kustermann's BPD-based formula was -0.31 (95% CI, -0.46 to -0.17) days and the mean difference of Robinson's CRL-based formula was -1.78 (95% CI, -1.88 to -1.68) days. In the external validation using data from spontaneously conceived pregnancies, with dating using Robinson's formula as the reference for 'true' gestational age, both our formula and Kustermann's formula resulted in underestimation of gestational age, with significant mean differences of -1.25 (95% CI, -1.28 to -1.22) days and -0.96 (95% CI, -0.98 to -0.93) days, respectively. The largest differences compared with Robinson's formula-based dating results were observed between 11 + 0 and 12 + 0 weeks. Dating the pregnancy using Robinson's formula led to 8.1% of pregnancies identified as requiring induction after 41 + 3 weeks, compared with 6.8% (P < 0.001) and 7.0% (P < 0.001) when applying our formula and Kustermann's formula, respectively.

CONCLUSION

Pregnancy dating based on ultrasound measurement of fetal BPD between 11 + 0 and 13 + 6 weeks' gestation is a reliable alternative to dating based on fetal CRL. © 2025 International Society of Ultrasound in Obstetrics and Gynecology.

Risk Prediction Models of Subsyndromal Delirium in Critically Ill Patients: A Systematic Review and Meta-Analysis

BACKGROUND

The number of predictive models for assessing the risk of subsyndromal delirium (SSD) in critically ill patients is increasing, yet the quality and applicability of these models in clinical practice remain unclear.

AIM

To systematically review and critically evaluate the existing risk prediction models.

STUDY DESIGN

Eleven Chinese and English databases, including PubMed, Web of Science and Embase, were searched from their inception to August 16, 2024. Two researchers independently screened the literature, extracted data and assessed the risk of bias and applicability using the prediction model risk of bias assessment tool. Meta-analysis was conducted using Stata 17.0.

RESULTS

Eight studies were included. The SSD incidence in ICU patients ranged from 8.97% to 34.5%. The most commonly used predictors were the APACHE II score and age. The reported area under the curve (AUC) ranged from 0.788 to 0.923, with the pooled AUC value for the five validated models being 0.87 (95% CI: 0.82-0.92). Six studies had a high risk of bias, while two had an unclear risk.

CONCLUSIONS

The eight included models demonstrated good performance in early identification and screening of high-risk critically ill patients for SSD, but they all exhibited a high risk of bias regarding model quality.

RELEVANCE TO CLINICAL PRACTICE

ICU professionals should carefully select and validate existing models based on their specific clinical settings before applying them. Alternatively, they can conduct new models incorporating multimodal data and artificial intelligence algorithms, utilizing large sample sizes, robust research designs and multi-center external validation.

The state of prediction models in hematologic disease: a worrisome assessment

PURPOSE OF REVIEW

The lack of optimal treatments for haematological disorders has led to the need for prediction models for diagnosis, therapeutic decision-making and life planning. In this review, the worrying current state of predictive models in the field is discussed.

RECENT FINDINGS

Here, we reviewed 100 studies on prediction models in this field. Our analysis revealed a concerning state of affairs, with a prevalence of suboptimal research methodologies and questionable statistical practices. This includes insufficient sample sizes, inadequate model evaluations, lack of necessary reports of model results, etc. In this regard, we present statistical considerations in the development and validation process of numerous models. This will provide the reader with the statistical knowledge related to prediction model necessary to assess bias in studies, compare other published models and determine the clinical utility of models.

SUMMARY

Awareness among authors, reviewers and editors of the required statistical considerations is crucial. Reinforcing these in all studies involving prediction models is needed. We all should encourage their use in evaluating existing studies and taking them fully into account in future studies.

Developing prediction models for electrolyte abnormalities in patients indicated for antihypertensive therapy: evidence-based treatment and monitoring recommendations

OBJECTIVES

Evidence from clinical trials suggests that antihypertensive treatment is associated with an increased risk of common electrolyte abnormalities. We aimed to develop and validate two clinical prediction models to estimate the risk of hyperkalaemia and hyponatraemia, respectively, to facilitate targeted treatment and monitoring strategies for individuals indicated for antihypertensive therapy.

DESIGN AND METHODS

Participants aged at least 40 years, registered to an English primary care practice within the Clinical Practice Research Datalink (CPRD), with a systolic blood pressure reading between 130 and 179 mmHg were included the study. The primary outcomes were first hyperkalaemia or hyponatraemia event recorded in primary or secondary care. Model development used a Fine-Gray approach with death from other causes as competing event. Model performance was assessed using C-statistic, D-statistic, and Observed/Expected (O/E) ratio upon external validation.

RESULTS

The development cohort included 1 773 224 patients (mean age 59 years, median follow-up 6 years). The hyperkalaemia model contained 23 predictors and the hyponatraemia model contained 29 predictors, with all antihypertensive medications associated with the outcomes. Upon external validation in a cohort of 3 805 366 patients, both models calibrated well (O/E ratio: hyperkalaemia 1.16, 95% CI 1.13-1.19; hyponatraemia 1.00, 95% CI 0.98-1.02) and showed good discrimination at 10 years (C-statistic: 0.69, 95% CI 0.69-0.69; 0.80, 95% CI 0.80-0.80, respectively).

CONCLUSION

Current clinical guidelines recommend monitoring serum electrolytes after initiating antihypertensive treatment. These clinical prediction models predicted individuals' risk of electrolyte abnormalities associated with antihypertensive treatment and could be used to target closer monitoring for individuals at a higher risk, where resources are limited.

Prospective multicenter external validation of the rib fracture frailty index

BACKGROUND

The Rib Fracture Frailty (RFF) Index is an internally validated machine learning-based risk assessment tool for adult patients with rib fractures that requires minimal provider entry. Existing frailty risk scores have yet to undergo head-to-head performance comparison with age, a widely used proxy for frailty in clinical practice. Our aim was to externally validate the RFF Index in a small-scale implementation feasibility study.

METHODS

Prospective observational cohort study conducted across five ACS COT-verified trauma centers. Participants included ≥18-year-old adults presenting January 1, 2021, to December 31, 2021, with traumatic rib fractures. The primary outcome was a composite outcome score comprised of three clinical factors: hospitalization ≥5 days, discharge disposition, and inpatient mortality. Proportional odds logistic regression evaluated associations of age model or RFF Index score model with composite outcome scores. Models were compared using standard discrimination and calibration metrics. Secondary analysis delineated predictive performance among patients with lower (Injury Severity Score < 15) and higher Injury Severity Score ≥ 15) injury burden.

RESULTS

Of 849 participants, 546 (64%) were male and median age was 62 years (interquartile range, 46-76 years). A one-point increase in RFF score was associated with 6% increased odds of higher composite outcome score (odds ratio [OR], 1.06; 95% confidence interval [95% CI], 1.04-1.08), while a 1-year increase in age did not show statistically significant association (OR, 1.10; 95% CI, 0.75-1.61). The RFF score had higher discrimination (OR, 0.09; 95% CI, 0.08-0.11 vs. OR, 0.06; 95% CI, 0.04-0.08; p = 0.04) and calibration performance compared with age, but on secondary analysis, higher predictive performance was limited to patients with lower injury burden. Both RFF Index and age had poor calibration for predicting patients discharged to home after hospitalization ≥5 days.

CONCLUSION

This prospective external validation study found RFF Index may be a better alternative to age for predicting adverse outcomes among patients with traumatic rib fractures and lower overall injury burden. Staged implementation studies in accordance with clinical prediction model implementation guidelines are required to evaluate the RFF Index's clinical efficacy and guide potential adoption.

LEVEL OF EVIDENCE

Prognostic; Level II.

Predicting prolonged work absence due to musculoskeletal disorders: development, validation, and clinical usefulness of prognostic prediction models

PURPOSE

Given the lack of robust prognostic models for early identification of individuals at risk of work disability, this study aimed to develop and externally validate three models for prolonged work absence among individuals on sick leave due to musculoskeletal disorders.

METHODS

We developed three multivariable logistic regression models using data from 934 individuals on sick leave for 4-12 weeks due to musculoskeletal disorders, recruited through the Norwegian Labour and Welfare Administration. The models predicted three outcomes: (1) > 90 consecutive sick days, (2) > 180 consecutive sick days, and (3) any new or increased work assessment allowance or disability pension within 12 months. Each model was externally validated in a separate cohort of participants (8-12 weeks of sick leave) from a different geographical region in Norway. We evaluated model performance using discrimination (c-statistic), calibration, and assessed clinical usefulness using decision curve analysis (net benefit). Bootstrapping was used to adjust for overoptimism.

RESULTS

All three models showed good predictive performance in the external validation sample, with c-statistics exceeding 0.76. The model predicting > 180 days performed best, demonstrating good calibration and discrimination (c-statistic 0.79 (95% CI 0.73-0.85), and providing net benefit across a range of decision thresholds from 0.10 to 0.80.

CONCLUSIONS

These models, particularly the one predicting > 180 days, may facilitate secondary prevention strategies and guide future clinical trials. Further validation and refinement are necessary to optimise the models and to test their performance in larger samples.

Clinical prognostic models for sarcomas: a systematic review and critical appraisal of development and validation studies

BACKGROUND

Current clinical guidelines recommend the use of clinical prognostic models (CPMs) for therapeutic decision-making in sarcoma patients. However, the number and quality of developed and externally validated CPMs is unknown. Therefore, we aimed to describe and critically assess CPMs for sarcomas.

METHODS

We performed a systematic review including all studies describing the development and/or external validation of a CPM for sarcomas. We searched the databases MEDLINE, EMBASE, Cochrane Central, and Scopus from inception until June 7th, 2022. The risk of bias was assessed using the prediction model risk of bias assessment tool (PROBAST).

RESULTS

Seven thousand six hundred fifty-six records were screened, of which 145 studies were eventually included, developing 182 and externally validating 59 CPMs. The most frequently modeled type of sarcoma was osteosarcoma (43/182; 23.6%), and the most frequently predicted outcome was overall survival (81/182; 44.5%). The most used predictors were the patient's age (133/182; 73.1%) and tumor size (116/182; 63.7%). Univariable screening was used in 137 (75.3%) CPMs, and only 7 (3.9%) CPMs were developed using pre-specified predictors based on clinical knowledge or literature. The median c-statistic on the development dataset was 0.74 (interquartile range [IQR] 0.71, 0.78). Calibration was reported for 142 CPMs (142/182; 78.0%). The median c-statistic of external validations was 0.72 (IQR 0.68-0.75). Calibration was reported for 46 out of 59 (78.0%) externally validated CPMs. We found 169 out of 241 (70.1%) CPMs to be at high risk of bias, mostly due to the high risk of bias in the analysis domain.

DISCUSSION

While various CPMs for sarcomas have been developed, the clinical utility of most of them is hindered by a high risk of bias and limited external validation. Future research should prioritise validating and updating existing well-developed CPMs over developing new ones to ensure reliable prognostic tools.

TRIAL REGISTRATION

PROSPERO CRD42022335222.

Prospective evaluation of modified Cincinnati Prehospital Stroke Severity Scale for identifying large vessel occlusion

OBJECTIVE

To develop a novel, straightforward diagnostic scale for predicting large vessel occlusion (LVO) and anterior circulation LVO (ALVO) in the emergency setting, evaluating its validity against existing scales.

METHODS

We prospectively enrolled patients with suspected stroke presenting consecutively at the National Comprehensive Stroke Centre's emergency department between February 20, 2022, and November 11, 2022. Emergency physicians assessed each patient using the modified Cincinnati Prehospital Stroke Severity Scale (mCPSSS) and the National Institutes of Health Stroke Scale (NIHSS). The study analyzed the mCPSSS and other prevalent stroke scales to evaluate their efficacy in detecting LVO and ALVO, employing receiver operating characteristic curve (ROC) analysis and area under the curve (AUC) statistics to assess the scales' sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy.

RESULTS

A total of 383 patients with suspected stroke were included in this study. The performance in identifying LVO in the emergency setting was greatest for mCPSSS ≥ 2 with a sensitivity of 0.802 and specificity of 0.770, PPV of 0.644, NPV of 0.882, and accuracy of 0.781. mCPSSS ≥ 2 was 0.766 sensitive, 0.733 specific, PPV of 0.564, NPV of 0.886, and accuracy of 0.749 in predicting ALVO. The mCPSSS identified LVO and ALVO with an optimal cut-off value of 2, exhibiting AUC superior to those of other widely used stroke scales, with AUC values of 0.824 for LVO and 0.790 for ALVO.

CONCLUSION

The mCPSSS could serve as an effective and straightforward scale for identifying LVOs in emergency settings.

CLINICAL TRIAL REGISTRATION INFORMATION

https://www.chictr.org.cn/ (ChiCTR2200056776).

Evaluation of a comprehensive set of normal tissue complication probability models for patients with head and neck cancer in an international cohort

BACKGROUND/PURPOSE

Normal tissue complication probability (NTCP) models can be used to guide radiation therapy (RT) decisions by estimating side-effect risks pretreatment to minimize (late) side-effects. Recently, a comprehensive individual toxicity risk (CITOR) profile of NTCP models addressing common side-effects in head and neck cancer (HNC) patients was developed. This study investigates the generalizability of these models in an international setting, with different treatment approaches and side-effect assessments, promoting their integration into more widespread clinical practice.

MATERIALS/METHODS

From a prospective registry study, 407 HNC patients were included who were treated with definitive RT with or without systemic therapy between 2015 and 2022. NTCP models predicting dysphagia, aspiration, xerostomia, sticky saliva, taste loss, speech problems, oral pain, and fatigue at 6 and 12 months after RT were evaluated. All side-effects were patient-rated using the MDASI-HN, except dysphagia which was reported by clinicians using the PSS-HN diet normalcy score. Model performance was appraised by discrimination (area under the curve [AUC]) and calibration.

RESULTS

CITOR models showed moderate-to-high performance in this cohort (mean AUC = 0.67[range = 0.55-0.80], moderate-to-good calibration). NTCP models for dysphagia, xerostomia, sticky saliva, and fatigue were the top performing models. Models for aspiration, taste loss and speech problems performed moderately well, which was partly explained by lower incidences.

CONCLUSION

Despite differences between the CITOR development and this evaluation cohort, including use of different side-effect scoring systems, most models exhibited moderate-to-high performance. This demonstrated that the dose-effect relations were generalizable. Therefore, this study supports further integration of these NTCP models in clinical practice.

Validation of a Machine-Learning Clinical Decision Aid for the Differential Diagnosis of Transient Loss of Consciousness

Background and Objectives

The aim of this study was to develop and validate a machine-learning classifier based on patient and witness questionnaires to support differential diagnosis of transient loss of consciousness (TLOC) at first presentation.

Methods

We prospectively recruited patients newly presenting with TLOC to an emergency department, an acute medical unit, and a first seizure or syncope clinic. We invited participants to complete an online questionnaire, either at home or at time of initial assessment. Two expert raters determined the cause of participants' TLOC after 6-month follow-up. We used independent development and validation samples to train a random forest classifier to predict diagnosis from participants' questionnaire responses and validate classifier performance. We compared classifier performance against penalized linear regression and referrer diagnosis.

Results

We included 178 participants in the final analysis, of whom 46 identified a witness able to complete an additional witness questionnaire. Given low witness recruitment, we developed a classifier based on patient answers only. A classifier trained on 9 items correctly identified 63 of 78 diagnoses (80.8%) (95% CI 70.0-88.5), an increase over the accuracy of initial assessing clinicians who were only able to diagnose 70.5% correctly. Within this, 96% (87.0%-99.4%) of those expertly rated as having syncope were correctly classified by the classifier (classifier sensitivity); 40% (20%-63.6%) of those expertly rated after follow-up as having either epilepsy or functional/dissociative seizures were similarly classified as being nonsyncope (classifier specificity).

Discussion

A machine-learning classifier for differential diagnosis of TLOC has comparable performance in differentiating between 3 main causes of primary TLOC as the current standard of care but is insufficiently accurate in its current form to warrant incorporation into routine care. A system including information from witnesses might improve classification performance.

External validation of a machine learning prediction model for massive blood loss during surgery for spinal metastases: a multi-institutional study using 880 patients

BACKGROUND CONTEXT

A machine learning (ML) model was recently developed to predict massive intraoperative blood loss (>2,500 mL) during posterior decompressive surgery for spinal metastasis that performed well on external validation within the same region in China.

PURPOSE

We sought to externally validate this model across new geographic regions (North America and Europe) and patient cohorts.

STUDY DESIGN

Multi-institutional retrospective cohort study PATIENT SAMPLE: We retrospectively included patients 18 years or older who underwent decompressive surgery for spinal metastasis across three institutions in the United States, the United Kingdom and the Netherlands between 2016 and 2022. Inclusion and exclusion criteria were consistent with the development study with additional inclusion of (1) patients undergoing palliative decompression without stabilization, (2) patients with multiple myeloma and lymphoma, and (3) patients who continued anticoagulants perioperatively.

OUTCOME MEASURES

Model performance was assessed by comparing the incidence of massive intraoperative blood loss (>2,500 mL) in our cohort to the predicted risk generated by the ML model. Blood loss was quantified in 7 ways (including the formula from the development study) as no gold standard exists, and the method in the development paper was not clearly defined. We estimated blood loss using the anesthesia report, and calculated it using transfusion data, and preoperative and postoperative hematocrit levels.

METHODS

The following five input variables necessary for risk calculation by the ML model were manually collected: tumor type, smoking status, ECOG score, surgical process, and preoperative platelet count. Model performance was assessed on overall fit (Brier score), discriminatory ability (area under the curve (AUC)), calibration (intercept & slope), and clinical utility (decision curve analysis)) for the total validation cohort, and for the North American and European cohorts separately. A sub-analysis, excluding the additional included patient groups, assessed the predictive model's performance with the same inclusion and exclusion criteria as the development cohort.

RESULTS

A total of 880 patients were included with a massive blood loss incidence ranging from 5.3% to 18% depending on the quantification method used. Using the most favorable quantification method, the predictive model overestimated risk in our total validation cohort and scored poorly on overall fit (Brier score: 0.278), discrimination (AUC: 0.631 [95%CI: 0.583, 0.680]), calibration, (intercept: -2.082, [95%CI: -2.285, -1.879]), slope: 0.283 [95%CI: 0.173, 0.393]), and clinical utility, with net harm observed in decision curve analysis from 20%. Similar poor performance results were observed in the sub-analysis excluding the additional included patients (n=676) and when analyzing the North American (n=539) and European (n=341) cohorts separately.

CONCLUSIONS

To our knowledge, this is the first published external validation of a predictive ML model within orthopedic surgery to demonstrate poor performance. This poor performance might be attributed to overfitting and sampling bias as the development cohort had an insufficient sample size, and distributional shift as our cohort had key differences in predictive variables used by the model. These findings emphasize the importance of extensive validation in different geographical areas and addressing biases and pitfalls of ML model development before clinical implementation, as untested models may do more harm than good.

Validation of SAPS 3 for predicting in-hospital mortality in patients with haematological malignancy requiring ICU management

Prognostic systems predicting death risk may vary for patients with haematological malignancies needing ICU care. This study externally validated SAPS 3 using a retrospective cohort of adults with these conditions in the ICU. The score was calculated at admission using the general and South America-adjusted formulas. Mortality discrimination was assessed via AUC-ROC, and calibration by Hosmer-Lemeshow goodness-of-fit and graphical analysis with a calibration belt. The analysis included 273 admissions, with 119 deaths. Discriminative capacity was low (AUC-ROC 0.56, CI 95% 0.49-0.63). There was a poor correlation between expected and observed events across all risk deciles (Hosmer-Lemeshow 10.45, p = 0.0635). Similar results were found with the South America-adjusted formula. SAPS 3 does not effectively discriminate between survivors and non-survivors, underestimating risk in low-risk groups and overestimating it in high-risk groups. Mortality risk estimation in this scenario should rely on clinical judgment.

Evaluation of factors predicting transition from prediabetes to diabetes among patients residing in underserved communities in the United States - A machine learning approach

INTRODUCTION

Over one-third of the population in the United States (US) has prediabetes. Unfortunately, underserved population in the United States face a higher burden of prediabetes compared to urban areas, increasing the risk of stroke and heart disease. There is a gap in the literature in understanding early predictors of diabetes among patients with prediabetes living in underserved communities in the United States. Hence, this study's objective is to identify factors influencing the transition from prediabetes to diabetes in rural or underserved communities using a machine learning approach.

METHODS

We conducted a retrospective analysis of data from prediabetic patients between 2012 and 2022. Eligible participants were at least 18 years old with baseline HbA1c levels between 5.7 % and 6.4 %. Eleven machine learning algorithms were evaluated using ten-fold cross-validation, including Logistic Regression (LR), Support Vector Classifier (SVC), K-nearest Neighbor (KNN), Gaussian Naive Bayes (GaussianNB), Bernoulli Naive Bayes (BernoulliNB), Adaptive Boosting (AdaBoost), Decision Tree (DT), Random Forest (RF), Gradient Boosting (GB), Extreme Gradient Boosting (XGBoost), and Extra Trees (ET). Subsequently, the SHAP framework was used to assess predictor influence and interactions observed with the top model.

RESULTS

Out of 5816 patients, 1910 met the criteria, with 426 progressing to diabetes. The Random Forest model achieved the highest accuracy (90.0 %) and AUC (0.963), followed by Extra Trees (89.5 % accuracy, AUC 0.962) and XGBoost (88.6 % accuracy, AUC 0.952). Logistic Regression demonstrated lower performance but outperformed other models such as K-Nearest Neighbors and Gaussian Naive Bayes. SHAP analysis with the RF model identified key predictors and their interactions. A significant interaction showed that lower BMI values, combined with increasing age, were associated with a reduced risk of diabetes progression, while higher BMI at younger ages increased the likelihood of progression. Additionally, several social determinants of health were identified as significant predictors.

CONCLUSION

Among the 11 models, the Random Forest model showed the strongest reliability for predicting diabetes progression. The results of this study can be used to inform public policy implications for the development of early, targeted interventions focusing on social determinants of health, dietary counseling, and BMI management to prevent diabetes in underserved communities.

Clinical prediction model for transition to psychosis in individuals meeting At Risk Mental State criteria

BACKGROUND

The At Risk Mental State (ARMS) (also known as the Ultra or Clinical High Risk) criteria identify individuals at high risk for psychotic disorder. However, there is a need to improve prediction as only about 18% of individuals meeting these criteria develop a psychosis with 12-months. We have developed and internally validated a prediction model using characteristics that could be used in routine practice.

METHODS

We conducted a systematic review and individual participant data meta-analysis, followed by focus groups with clinicians and service users to ensure that identified factors were suitable for routine practice. The model was developed using logistic regression with backwards selection and an individual participant dataset. Model performance was evaluated via discrimination and calibration. Bootstrap resampling was used for internal validation.

RESULTS

We received data from 26 studies contributing 3739 individuals; 2909 from 20 of these studies, of whom 359 developed psychosis, were available for model building. Age, functioning, disorders of thought content, perceptual abnormalities, disorganised speech, antipsychotic medication, cognitive behavioural therapy, depression and negative symptoms were associated with transition to psychosis. The final prediction model included disorders of thought content, disorganised speech and functioning. Discrimination of 0.68 (0.5-1 scale; 1=perfect discrimination) and calibration of 0.91 (0-1 scale; 1=perfect calibration) showed the model had fairly good predictive ability.

DISCUSSION

The statistically robust prediction model, built using the largest dataset in the field to date, could be used to guide frequency of monitoring and enable rational use of health resources following assessment of external validity and clinical utility.

Prediction model for delirium in advanced cancer patients receiving palliative care: development and validation

BACKGROUND

Delirium is a common and distressing mental disorder in palliative care. To date, no delirium prediction model is available for thepalliative care population. The research aimed to develop and validate a nomogram model for predicting the occurrence of delirium in advanced cancer patients admitted to palliative care units.

METHODS

This was a prospective, multicenter, observational study. Logistic regression was used to identify the independent risk factors for incident delirium among advanced cancer patients in palliative care units. Advanced cancer patients admitted to palliative care units between February 2021 and January 2023 were recruited from four hospitals in Chengdu, Sichuan Province, China. Model performance was evaluated via the area under the receiver operating characteristic curve, calibration plots and decision curve analysis.

RESULTS

There were 592 advanced cancer patients receiving palliative care in the development cohort, 196 in the temporal validation cohort and 65 in the external validation cohort. The final nomogram model included 8 variables (age, the Charlson comorbidity index, cognitive function, the Barthel index, bilirubin, sodium, the opioid morphine equivalent dose and the use of anticholinergic drugs). The model revealed good performance in terms of discrimination, calibration, and clinical practicability, with an area under the receiver operating characteristic curve of 0.846 in the training set, 0.838 after bootstrapping, 0.829 in the temporal validation and 0.803 in the external validation set.

CONCLUSIONS

The model serves as a reliable tool to predict delirium onset for advanced cancer patients in palliative care units, which will facilitate early targeted preventive measures to reduce the burden of delirium.

The Barts Surgical Infection Risk (B-SIR) tool: external validation and comparison with existing tools to predict surgical site infection after cardiac surgery

OBJECTIVE

Further to previous development and internal validation of the Barts Surgical Infection Risk (B-SIR) tool, this study sought to explore the external validity of the B-SIR tool and compare it with the Australian Clinical Risk Index (ACRI), and the Brompton and Harefield Infection Score (BHIS).

STUDY DESIGN AND SETTING

This multi-centre retrospective analysis of prospectively collected local data included adult (age ≥18 years) patients undergoing cardiac surgery between January 2018 and December 2019. Pre-pandemic data were used as a reflection of standard practice. Area under the curve (AUC) was used to validate and compare the predictive power of the scores, and calibration was assessed using the Hosmer-Lemeshow test and calibration plots.

RESULTS

In total, 6022 patients from three centres were included in the complete case analysis. The mean age was 66 years, 75% were men and 3.19% developed a surgical site infection (SSI). The B-SIR tool had an area under the curve (AUC) of 0.686 [95% confidence interval (CI) 0.649-0.723], similar to the developmental study (AUC=0.682, 95% CI 0.652-0.713). This was significantly higher than the BHIS AUC of 0.610 (95% CI 0.045-0.109; P<0.001) and the ACRI AUC of 0.614 (95% CI 0.041-0.103; P<0.001). After recalibration using a correction factor, the B-SIR tool gave accurate risk predictions (Hosmer-Lemeshow test P=0.423). The multiple imputation result (AUC=0.676, 95% CI 0.639-0.712) was similar to development data, and higher than the ACRI and BHIS.

CONCLUSION

External validation indicated that the B-SIR tool predicted SSI after cardiac surgery better than the ACRI and BHIS. This suggests that the B-SIR tool could be useful for use in routine practice.

Development and validation of the Crohn's disease-intestinal failure-wean (CDIF-Wean) Score to predict outcomes of intestinal rehabilitation

BACKGROUND & AIMS

Enteral autonomy, a key outcome of intestinal rehabilitation in patients with intestinal failure (IF), is challenging to predict due to disease complexity and heterogeneity. The aim of this cohort study is to develop and validate a multivariate model to predict enteral autonomy in patients with IF caused by Crohn's disease (CDIF), and to derive an outcome-based severity classification for CDIF.

METHODS

The CDIF-Wean Score was constructed and internally validated in a cohort of 182 patients with CDIF from a tertiary IF unit. We performed stepwise backward selection to include relevant and significant clinical variables in a binomial regression with inverted probability of censoring weighting. The Score was externally validated in a separate cohort of 107 patients with CDIF from an independent tertiary IF unit. A severity classification, based on the CDIF-Wean Score, was evaluated with cumulative incidence curves for enteral autonomy and death during home parenteral support (HPS).

RESULTS

In the CDIF-Wean Score, age, HPS duration, chronicity of Crohn's disease, intestinal anatomy, and eligibility and type of reconstructive surgery was predictive of enteral autonomy. The Score performed well in discrimination and calibration, with 0.84 and 0.84 area under the receiver operating characteristic curve, and 0.13 and 0.16 Brier scores in internal and external validation, respectively. The CDIF severity classification was significantly associated with both short- and long-term prognosis, where mild patients had a 7.5, 5.8 and 5.2-fold higher probability of enteral autonomy than severe patients at 1, 5 and 10 years (p<0.0001).

CONCLUSION

The CDIF-Wean Score is the first validated prediction model for IF outcomes, and demonstrates accuracy, robustness and generalisability in the prognostication of CDIF patients.

On the estimation of inverse-probability-of-censoring weights for the evaluation of survival prediction error

Inverse probability weighting (IPW) is a popular method for making inferences regarding unobserved or unobservable data of a target population based on observed data. This paper considers IPW applied to right-censored time-to-event data. We investigate the behavior of the inverse-probability-of-censoring weighted (IPCW) Brier score, which is frequently used to assess the predictive performance of time-to-event models. A key requirement of the IPCW Brier score is the estimation of the censoring distribution, which is needed to compute the weights. The established paradigm of splitting a dataset into a training and a test set for model fitting and evaluation raises the question which of these datasets to use in order to fit the censoring model. There seems to be considerable disagreement between authors with regards to this issue, and no standard has been established so far. To shed light on this important question, we conducted a comprehensive experimental study exploring various data scenarios and estimation schemes. We found that it is generally of little importance which dataset is used to model the censoring distribution. However, in some circumstances, such as in the case of a covariate-dependent censoring process, a small sample size, or when dealing with noisy data, it may be advisable to use the test set instead of the training set to model the censoring distribution. A detailed set of practical recommendations concludes our paper.

External validation of twelve existing survival prediction models for patients with spinal metastases

BACKGROUND CONTEXT

Survival prediction models for patients with spinal metastases may inform patients and clinicians in shared decision-making.

PURPOSE

To externally validate all existing survival prediction models for patients with spinal metastases.

DESIGN

Prospective cohort study using retrospective data.

PATIENT SAMPLE

953 patients.

OUTCOME MEASURES

Survival in months, area under the curve (AUC), and calibration intercept and slope.

METHOD

This study included patients with spinal metastases referred to a single tertiary referral center between 2016 and 2021. Twelve models for predicting 3, 6, and 12-month survival were externally validated Bollen, Mizumoto, Modified Bauer, New England Spinal Metastasis Score, Original Bauer, Oswestry Spinal Risk Index (OSRI), PathFx, Revised Katagiri, Revised Tokuhashi, Skeletal Oncology Research Group Machine Learning Algorithm (SORG-MLA), Tomita, and Van der Linden. Discrimination was assessed using (AUC) and calibration using the intercept and slope. Calibration was considered appropriate if calibration measures were close to their ideal values with narrow confidence intervals.

RESULTS

In total, 953 patients were included. Survival was 76.4% at 3 months (728/953), 62.2% at 6 months (593/953), and 50.3% at 12 months (479/953). Revised Katagiri yielded AUCs of 0.79 (95% CI, 0.76-0.82) to 0.81 (95% CI, 0.79-0.84), Bollen yielded AUCs of 0.76 (95% CI, 0.73-0.80) to 0.77 (95% CI, 0.75-0.80), and OSRI yielded AUCs of 0.75 (95% CI, 0.72-0.78) to 0.77 (95% CI, 0.74-0.79). The other 9 prediction models yielded AUCs ranging from 0.59 (95% CI, 0.55-0.63) to 0.76 (95% CI, 0.74-0.79). None of the twelve models yielded appropriate calibration.

CONCLUSIONS

Twelve survival prediction models for patients with spinal metastases yielded poor to fair discrimination and poor calibration. Survival prediction models may inform decision-making in patients with spinal metastases, provided that recalibration using recent patient data is performed.

Prediction of atrial fibrillation after a stroke event: A systematic review with meta-analysis

BACKGROUND

Detecting atrial fibrillation (AF) after stroke is a key component of secondary prevention, but indiscriminate prolonged cardiac monitoring is costly and burdensome. Multivariable prediction models could be used to inform selection of patients.

OBJECTIVE

This study aimed to determine the performance of available models for predicting AF after a stroke.

METHODS

We searched for studies of multivariable models that were derived, validated, or augmented for prediction of AF in patients with a stroke, using MEDLINE and Embase from inception through September 20, 2024. Discrimination measures for tools with C statistic data from ≥3 cohorts were pooled by bayesian meta-analysis, with heterogeneity assessed through a 95% prediction interval. The risk of bias was assessed with the Prediction model Risk Of Bias Assessment tool (PROBAST).

RESULTS

We included 75 studies with 58 prediction models; 66% had a high risk of bias. Fifteen multivariable models were eligible for meta-analysis. Three models showed excellent discrimination: SAFE (C statistic, 0.856; 95% confidence interval [CI], 0.796-0.916), SURF (0.815; 95% CI, 0.728-0.893), and iPAB (0.888; 95% CI, 0.824-0.957). Excluding high-bias studies, only SAFE showed excellent discrimination (0.856; 95% CI 0.800-0.915). No model showed excellent discrimination when limited to external validation or studies with ≥100 AF events. No clinical impact studies were found.

CONCLUSION

Three of the 58 identified multivariable prediction models for AF after stroke demonstrated excellent statistical performance on meta-analysis. However, prospective validation is required to understand the effectiveness of these models in clinical practice before they can be recommended for inclusion in clinical guidelines.

The External Validation of a Multivariable Prediction Model for Recurrent Pelvic Organ Prolapse After Native Tissue Repair: A Prospective Cohort Study

Background/Objectives: A prediction model for anatomical cystocele recurrence after native tissue repair was developed and internally validated in 2016. This model estimates a patients' individual risk of recurrence and can be used for counseling. Before implementation in urogynecological clinical practice, external validation is needed. The aim of this study was to assess the external validity of this previously developed prediction model. The secondary aim was to test the performance of this model with a composite and subjective outcome of pelvic organ prolapse (POP) recurrence. Furthermore, the aim was to investigate whether risk factors for POP recurrence were in line with the population in which the original model was developed. Methods: In this prospective multicenter cohort study, 246 patients who underwent anterior colporrhaphy were included. Inclusion criteria were patients scheduled to undergo a primary anterior colporrhaphy (with a POP Quantification (POPQ) stage ≥ 2 cystocele). A combination of a primary anterior colporrhaphy with other POP or incontinence surgery (without the use of vaginal or abdominal mesh material) was permitted. Patients with prolapse or incontinence surgery prior to index surgery could not participate. All patients filled in questionnaires, pelvic floor ultrasound was performed preoperatively, and data from the medical file concerning POPQ stage and obstetric and general history were obtained. Results: Thirty women (12.2%) were lost at follow up. Anatomical cystocele recurrence was present in 107/216 (49.5%), subjective recurrence in 19/208 (9.1%), and 39/219 (17.8%) patients met the criteria for composite outcome. The area under the receiver operating characteristic curves for anatomical, composite, and subjective recurrence were 65.5% (95% CI: 58.7-72.4), 55.8% (95% CI 47.3-64.3%, NS), and 55.1% (95% CI 45.1-65.2%), respectively. In the multivariable analysis, preoperative cystocele stage 3 or 4 and a complete levator defect on ultrasound were independent risk factors for anatomical recurrence. For composite recurrence, younger age and an active employment status were only risk factors in univariable analysis. No significant risk factors for subjective recurrence could be identified. Conclusions: This external validation study showed a moderate performance for a prediction model for anatomical recurrence. The model cannot be used for a composite or subjective outcome prediction because of poor performance. For composite and subjective recurrence, new prediction models need to be developed.

External Validation of SHA2PE Score: A Score to Predict Low-Risk Lower Gastrointestinal Bleeding in the Emergency Department

Introduction: Lower gastrointestinal bleeding (LGIB) frequently leads to emergency department (ED) visits and hospitalizations, encompassing a spectrum of outcomes from spontaneous resolution to intrahospital mortality. Aim: The purpose of this study was to validate a scoring system designed to identify cases of low-risk LGIB, allowing for safe discharge from the ED. Methods: A retrospective analysis of all gastrointestinal bleeding cases presented at three EDs in 2020 was conducted, focusing specifically on patients with LGIB. The SHA2PE score incorporates factors such as systolic blood pressure, hemoglobin levels, use of antiplatelet or anticoagulant medications, pulse rate, and episodes of bright blood per rectum. Results: Out of 1112 patients presenting with LGIB to the ED, 55 were hospitalized, 20 required blood transfusions, 15 underwent colonoscopies, one underwent interventional radiology procedures, and two patients died. Employing a SHA2PE score with a cutoff value of 1 yielded a specificity of 78.5% (95% CI (confidence interval) [75.8-81.0]), sensitivity of 76.8% (95% CI [63.6-87.0]), positive predictive value (PPV) of 17.0% (95% CI [12.6-22.2]), and negative predictive value (NPV) of 98.3% (95% CI [97.2-99.1]) for predicting the need for hospitalization and intrahospital intervention. When considering return visits to the ED within 7 days with the same presentation, the score demonstrated a specificity of 78.8% (95% CI [76.0-81.3]), sensitivity of 68.6% (95% CI [56.4-79.1]), PPV of 19% (95% CI [14.3-24.4]), and NPV of 97.2% (95% CI [95.8-98.2]). Conclusions: The SHA2PE score demonstrates potential in predicting cases of low-risk LGIB, offering a high NPV for hospitalization, the need for intrahospital intervention, and return visits to the ED. However, these findings should be interpreted cautiously given the low prevalence of interventions and limitations in the study's population and design.

External validation and calibration of the HoLISTIC Consortium's advanced-stage Hodgkin lymphoma international prognostic index (A-HIPI) in the Brazilian Hodgkin lymphoma registry

The Hodgkin lymphoma International Study for Individual Care (HoLISTIC) Consortium's A-HIPI model, developed in 2022 for advanced-stage classical Hodgkin lymphoma (cHL), predicts survival within 5 years amongst newly diagnosed patients. This study validates its performance in the Brazilian Hodgkin lymphoma registry. By 2022, the Brazilian HL registry included 1357 cHL patients, with a median 5-year follow-up. Probabilities for 5-year progression-free survival (PFS) and overall survival (OS) were calculated using A-HIPI-model equations. Discrimination (Harrell C-statistic/Uno C-statistic) and calibration measures assessed external validation and calibration. Lab values beyond the allowed range were excluded, mirroring the initial A-HIPI analysis. A total of 694 advanced-stage cHL patients met the original inclusion criteria (age 18-65 years, Stage IIB-IV). Median age was 31 years; 46.3% were females. Stage distribution was IIB (33.1%), III (27.4%), IV (39.5%). Bulky disease in 32.6%. Five-year PFS and OS were 68.4% and 86.0%, respectively. Harrell C-statistics were 0.60 for PFS and 0.69 for OS, and Uno C-statistics were 0.63 for PFS and 0.72 for OS. Calibration plots demonstrated well-calibrated predictions with calibration slopes of 0.91 and 1.03 for 5-year OS and PFS, respectively. Despite differing patient, clinical characteristics, and socioeconomic factors, the baseline prediction tool performed well in the Brazilian cohort, demonstrating adequate discrimination and calibration. This supports its reliability in diverse settings.

External Validation of Two Clinical Prediction Models for Pediatric Pneumonia

OBJECTIVE

To externally validate two prediction models for pediatric radiographic pneumonia.

METHODS

We prospectively evaluated the performance of two prediction models (Pneumonia Risk Score [PRS] and Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine [CARPE DIEM] models) from a prospective convenience sample of children 90 days - 18 years of age from a pediatric emergency department undergoing chest radiography for suspected pneumonia between January 1, 2022, and December 31st, 2023. We evaluated model performance using the original intercepts and coefficients and evaluated for performance changes when performing recalibration and re-estimation procedures.

RESULTS

We included 202 patients (median age 3 years, IQR 1-6 years), of whom radiographic pneumonia was found in 92 (41.0%). The PRS model had an area under the receiver operator characteristic curve of 0.72 (95% confidence interval [CI] 0.64-0.79), which was higher than the CARPE DIEM (0.59; 95% CI 0.51-0.67) (P < 0.01). Using optimal cutpoints, the PRS model showed higher sensitivity (65.2%, 95% CI 54.6-74.9) and specificity (72.7%, 95% CI 63.4-80.8) compared to the CARPE DIEM model (sensitivity 56.5 [95% CI 45.8-66.8]; specificity 60.9 [95% CI 50.2-69.2]). Recalibration and re-estimation of models improved performance, particularly for the CARPE DIEM model, with gains in sensitivity and specificity, and improved calibration.

CONCLUSION

The PRS model demonstrated better performance than the CARPE DIEM model in predicting radiographic pneumonia. Among children with a high rate of pneumonia, these models did not reach a level of performance sufficient to be used independently of clinical judgment. These findings highlight the need for further validation and improvement of models to enhance their utility.

External Validation of Vision, Aphasia and Neglect, Ventura Emergent Large Vessel Occlusion and Large Artery Intracranial Occlusion Screening Tools for Emergent Large Vessel Occlusion Stroke: A Multicenter, Prospective, Cross-Sectional Study

BACKGROUND

Vision, Aphasia, and Neglect (VAN), Ventura Emergent Large Vessel Occlusion (VES), and Large Artery Intracranial Occlusion (LARIO) are promising stroke screening tools that were shown to have high diagnostic performance to detect Emergent Large Vessel Occlusion (ELVO) in their derivation studies.

OBJECTIVES

This study aimed to assess the validation of VAN, VES, and LARIO in predicting ELVO among patients presenting at emergency department (ED) triage with suspected acute ischemic stroke.

METHODS

This is a prospective multicenter study conducted in five EDs of tertiary stroke centers between June and October 2023. Patients with suspected stroke admitted to ED for triage were evaluated using the VAN, VES, and LARIO stroke screening tools. Diagnostic performances of these tools for predicting ELVO were determined and compared with the National Institute of Health Stroke Scale (NIHSS).

RESULTS

A total of 614 patients were included. The prevalence of ELVO was found to be 23.5% in the study population. VAN exhibited a sensitivity of 70.1% and specificity of 78.7%, VES showed a higher sensitivity (79.1%) with lower specificity (63.4%), while LARIO displayed high specificity (86%) with lower sensitivity (56.3%). Receiver operating characteristic curve analysis showed that LARIO and NIHSS had similar diagnostic performance (areas under the curve [AUC] 0.801 and 0.805, p = 0.7, respectively), while VES showed a modestly poorer performance (AUC 0.746, p < 0.001 and p = 0.003).

CONCLUSION

The comparable diagnostic performance of VAN, VES, and LARIO to the NIHSS, in addition to their straightforwardness and rapid evaluation time, can facilitate optimal care for patients with ELVO in prehospital or ED triage settings.

Fracture prediction in rheumatoid arthritis: validation of FRAX with bone mineral density for incident major osteoporotic fractures

OBJECTIVES

FRAX uses clinical risk factors, with or without BMD, to calculate 10-year fracture risk. RA is a risk factor for osteoporotic fracture and a FRAX input variable. FRAX predates the current era of RA treatment. We examined how well FRAX predicts fracture in contemporary RA patients.

METHODS

Administrative data from patients receiving BMD testing were linked to the Manitoba Population Health Research Data Repository. Observed cumulative 10-year major osteoporotic fracture (MOF) probability was compared with FRAX-predicted 10-year MOF probability with BMD for assessing calibration. MOF risk stratification was assessed using Cox regression.

RESULTS

RA patients (n = 2099, 208 with incident MOF) and non-RA patients (n = 2099, with 165 incident MOF) were identified. For RA patients, FRAX-predicted 10-year risk was 13.2% and observed 10-year MOF risk was 13.2% (95% CI 11.6, 15.1). The slope of the calibration plot was 0.67 (95% CI 0.53, 0.81) in those with RA vs 0.98 (95% CI 0.61, 1.34) in non-RA patients. Risk was overestimated in RA patients with high FRAX scores (>20%), but FRAX was well calibrated in other groups. FRAX stratified risk in those with and without RA [hazard ratio (HR) 1.52 (95% CI 1.25, 1.72) vs 2.00 (95% CI 1.73, 2.31)], with slightly better performance in the latter (P for interaction = 0.004).

CONCLUSIONS

FRAX predicts fracture risk in contemporary RA patients but may slightly overestimate risk in those already at high predicted risk. Thus the current FRAX tool continues to be appropriate for fracture risk assessment in RA patients.

Measuring personality in Libyan Arabs: validating the big five aspect scale with 10 factors domain

Research has developed the Big-Five Aspect Scale (BFAS), supporting a five-domain model that includes 10 related aspects. In Arabic societies, there is currently a lack of validation evidence for a scale with these 10 aspects. Thus, this study develops and examines the psychometric properties of the short version of the BFAS (BFAS-SV) within Libyan Arab adults. The sample (N = 1136; 74.6% women, Mage = 25.30, SDage = 8.44) completed the original BFAS and the Arabic version of the International Personality Item Pool (IPIP) to assess the BFAS-SV's convergent validity. Confirmatory Factor Analysis (CFA) was applied. The findings provide strong support for the presence of 10 distinct aspects within the Big Five personality domains. Additionally, a robust positive and negative correlation was found among the 10 BFAS-SV aspects, as well as between the BFAS-SV domains of Conscientiousness, Extraversion, Introversion, and Openness/Intellect and their corresponding dimensions in the IPIP, further confirming its concurrent and discriminant validity. Furthermore, the Cronbach's alpha coefficients for the five domains and their respective 10 aspects ranged from 0.61 to 0.85, indicating good internal consistency. Significant gender differences were observed in the Neuroticism domain, particularly in its two aspects (Volatility and Withdrawal), as well as in the Openness/Intellect domain and the Politeness aspect, with women scoring higher in all cases.In conclusion, this study establishes the reliability, validity, and applicability of the Arabic BFAS among the Libyan Arab population. The insights gained into the personality traits and behaviors of Libyan Arab individuals provide valuable implications for personal development and professional success.

Adding ethnicity to cardiovascular risk prediction: External validation and model updating of SCORE2 using data from the HELIUS population cohort

BACKGROUND

Current prediction models for mainland Europe do not include ethnicity, despite ethnic disparities in cardiovascular disease (CVD) risk. SCORE2 performance was evaluated across the largest ethnic groups in the Netherlands and ethnic backgrounds were added to the model.

METHODS

11,614 participants, aged between 40 and 70 years without CVD, from the population-based multi-ethnic HELIUS study were included. Fine and Gray models were used to calculate sub-distribution hazard ratios (SHR) for South-Asian Surinamese, African Surinamese, Ghanaian, Turkish and Moroccan origin groups, representing their CVD risk relative to the Dutch group, on top of individual SCORE2 risk predictions. Model performance was evaluated by discrimination, calibration and net reclassification index (NRI).

RESULTS

Overall, 274 fatal and non-fatal CVD events, and 146 non-cardiovascular deaths were observed during a median of 7.8 years follow-up (IQR 6.8-8.8). SHRs for CVD events were 1.86 (95 % CI 1.31-2.65) for the South-Asian Surinamese, 1.09 (95 % CI 0.76-1.56) for the African-Surinamese, 1.48 (95 % CI 0.94-2.31) for the Ghanaian, 1.63 (95 % CI 1.09-2.44) for the Turkish, and 0.67 (95 % CI 0.39-1.18) for the Moroccan origin groups. Adding ethnicity to SCORE2 yielded comparable calibration and discrimination [0.764 (95 % CI 0.735-0.792) vs. 0.769 (95 % CI 0.740-0.797)]. The NRI for adding ethnicity to SCORE2 was 0.24 (95 % CI 0.18-0.31) for events and - 0.12 (95 % CI -0.13-0.12) for non-events.

CONCLUSIONS

Adding ethnicity to the SCORE2 risk prediction model in a middle-aged, multi-ethnic Dutch population did not improve overall discrimination but improved risk classification, potentially helping to address CVD disparities through timely treatment.

External validation of a prognostic model to improve prediction of psychosis: a retrospective cohort study in primary care

BACKGROUND

Early detection could reduce the duration of untreated psychosis. GPs are a vital part of the psychosis care pathway, but find it difficult to detect the early features. An accurate risk prediction tool, P Risk, was developed to detect these.

AIM

To externally validate P Risk.

DESIGN AND SETTING

This retrospective cohort study used a validation dataset of 1 647 934 UK Clinical Practice Research Datalink (CPRD) primary care records linked to secondary care records.

METHOD

The same predictors (age; sex; ethnicity; social deprivation; consultations for suicidal behaviour, depression/anxiety, and substance misuse; history of consultations for suicidal behaviour; smoking history; substance misuse; prescribed medications for depression/anxiety/post-traumatic stress disorder/obsessive compulsive disorder; and total number of consultations) were used as for the development of P Risk. Predictive risk, sensitivity, specificity, and likelihood ratios were calculated for various risk thresholds. Discrimination (Harrell's C-index) and calibration were calculated. Results were compared between the development (CPRD GOLD) and validation (CPRD Aurum) datasets.

RESULTS

Psychosis risk increased with values of the P Risk prognostic index. Incidence was highest in younger age groups and, in the main, higher in males. Harrell's C was 0.79 (95% confidence interval = 0.78 to 0.79) in the validation dataset and 0.77 in the development dataset. A risk threshold of 1.0% gave sensitivity of 65.9% and specificity of 86.6%.

CONCLUSION

Further testing is required, but P Risk has the potential to be used in primary care to detect future risk of psychosis.

Development and validation of a risk prediction model for hospital admission in COVID-19 patients presenting to primary care

BACKGROUND

There is a paucity of prognostic models for COVID-19 that are usable for in-office patient assessment in general practice (GP).

OBJECTIVES

To develop and validate a risk prediction model for hospital admission with readily available predictors.

METHODS

A retrospective cohort study linking GP records from 8 COVID-19 centres and 55 general practices in the Netherlands to hospital admission records. The development cohort spanned March to June 2020, the validation cohort March to June 2021. The primary outcome was hospital admission within 14 days. We used geographic leave-region-out cross-validation in the development cohort and temporal validation in the validation cohort.

RESULTS

In the development cohort, 4,806 adult patients with COVID-19 consulted their GP (median age 56, 56% female); in the validation cohort 830 patients did (median age 56, 52% female). In the development and validation cohort respectively, 292 (6.1%) and 126 (15.2%) were admitted to the hospital within 14 days, respectively. A logistic regression model based on sex, smoking, symptoms, vital signs and comorbidities predicted hospital admission with a c-index of 0.84 (95% CI 0.83 to 0.86) at geographic cross-validation and 0.79 (95% CI 0.74 to 0.83) at temporal validation, and was reasonably well calibrated (intercept -0.08, 95% CI -0.98 to 0.52, slope 0.89, 95% CI 0.71 to 1.07 at geographic cross-validation and intercept 0.02, 95% CI -0.21 to 0.24, slope 0.82, 95% CI 0.64 to 1.00 at temporal validation).

CONCLUSION

We derived a risk model using readily available variables at GP assessment to predict hospital admission for COVID-19. It performed accurately across regions and waves. Further validation on cohorts with acquired immunity and newer SARS-CoV-2 variants is recommended.

Development and validation of risk prediction model for adverse outcomes in trauma patients

BACKGROUND

The prognosis of trauma patients is highly dependent on early medical diagnosis. By constructing a nomogram model, the risk of adverse outcomes can be displayed intuitively and individually, which has important clinical implications for medical diagnosis.

OBJECTIVE

To develop and evaluate models for predicting patients with adverse outcomes of trauma that can be used in different data availability settings in China.

METHODS

This was a retrospective prognostic study using data from 8 public tertiary hospitals in China from 2018. The data were randomly divided into a development set and a validation set. Simple, improved and extended models predicting adverse outcomes were developed, with adverse outcomes defined as in-hospital death or ICU transfer, and patient clinical characteristics, vital signs, diagnoses, and laboratory test values as predictors. The results of the models were presented in the form of nomograms, and performance was evaluated using area under the receiver operating characteristic curve (ROC-AUC), precision-recall (PR) curves (PR-AUC), Hosmer-Lemeshow goodness-of-fit test, calibration curve, and decision curve analysis (DCA).

RESULTS

Our final dataset consisted of 18,629 patients (40.2% female, mean age of 52.3), 1,089 (5.85%) of whom resulted in adverse outcomes. In the external validation set, three models achieved ROC-AUC of 0.872, 0.881, and 0.903, and a PR-AUC of 0.339, 0.337, and 0.403, respectively. In terms of the calibration curves and DCA, the models also performed well.

CONCLUSIONS

This prognostic study found that three prediction models and nomograms including the patient clinical characteristics, vital signs, diagnoses, and laboratory test values can support clinicians in more accurately identifying patients who are at risk of adverse outcomes in different settings based on data availability.

Nightingale Health Biobank Collaborative Group, Barrett JC, Esko T, Fischer K, Jostins-Dean L, Jousilahti P, Julkunen H, Jääskeläinen T, Kangas A, Kerimov N, Kerminen S, Kolde A, Koskela H, Kronberg J, Lundgren SN, Lundqvist A, Mäkelä V, Nybo K, Perola M, Salomaa V, Schut K, Soikkeli M, Soininen P, Tiainen M, Tillmann T, Würtz P. Metabolomic and genomic prediction of common diseases in 700,217 participants in three national biobanks

Identifying individuals at high risk of chronic diseases via easily measured biomarkers could enhance efforts to prevent avoidable illness and death. Using 'omic data can stratify risk for many diseases simultaneously from a single measurement that captures multiple molecular predictors of risk. Here we present nuclear magnetic resonance metabolomics in blood samples from 700,217 participants in three national biobanks. We built metabolomic scores that identify high-risk groups for diseases that cause the most morbidity in high-income countries and show consistent cross-biobank replication of the relative risk of disease for these groups. We show that these metabolomic scores are more strongly associated with disease onset than polygenic scores for most of these diseases. In a subset of 18,709 individuals with metabolomic biomarkers measured at two time points we show that people whose scores change have different risk of disease, suggesting that repeat measurements capture changes both to health status and disease risk possibly due to treatment, lifestyle changes or other factors. Lastly, we assessed the incremental predictive value of metabolomic scores over existing clinical risk scores for multiple diseases and found modest improvements in discrimination for several diseases whose clinical utility, while promising, remains to be determined.

Accuracy of the Canadian COVID-19 Mortality Score (CCMS) to predict in-hospital mortality among vaccinated and unvaccinated patients infected with Omicron: a cohort study

OBJECTIVE

The objective is to externally validate and assess the opportunity to update the Canadian COVID-19 Mortality Score (CCMS) to predict in-hospital mortality among consecutive non-palliative COVID-19 patients infected with Omicron subvariants at a time when vaccinations were widespread.

DESIGN

This observational study validated the CCMS in an external cohort at a time when Omicron variants were dominant. We assessed the potential to update the rule and improve its performance by recalibrating and adding vaccination status in a subset of patients from provinces with access to vaccination data and created the adjusted CCMS (CCMSadj). We followed discharged patients for 30 days after their index emergency department visit or for their entire hospital stay if admitted.

SETTING

External validation cohort for CCMS: 36 hospitals participating in the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN). Update cohort for CCMSadj: 14 hospitals in CCEDRRN in provinces with vaccination data.

PARTICIPANTS

Consecutive non-palliative COVID-19 patients presenting to emergency departments.

MAIN OUTCOME MEASURES

In-hospital mortality.

RESULTS

Of 39 682 eligible patients, 1654 (4.2%) patients died. The CCMS included age, sex, residence type, arrival mode, chest pain, severe liver disease, respiratory rate and level of respiratory support and predicted in-hospital mortality with an area under the curve (AUC) of 0.88 (95% CI 0.87 to 0.88) in external validation. Updating the rule by recalibrating and adding vaccination status to create the CCMSadj changed the weights for age, respiratory status and homelessness, but only marginally improved its performance, while vaccination status did not. The CCMSadj had an AUC of 0.91 (95% CI 0.89 to 0.92) in validation. CCMSadj scores of <10 categorised patients as low risk with an in-hospital mortality of <1.6%. A score>15 had observed mortality of >56.8%.

CONCLUSIONS

The CCMS remained highly accurate in predicting mortality from Omicron and improved marginally through recalibration. Adding vaccination status did not improve the performance. The CCMS can be used to inform patient prognosis, goals of care conversations and guide clinical decision-making for emergency department patients with COVID-19.

Development and validation of a model for predicting in-hospital mortality in patients with sepsis-associated kidney injury receiving renal replacement therapy: a retrospective cohort study based on the MIMIC-IV database

Background

SAKI is a common and serious complication of sepsis, contributing significantly to high morbidity and mortality, especially in patients requiring RRT. Early identification of high-risk patients enables timely interventions and improvement in clinical outcomes. The objective of this study was to develop and validate a predictive model for in-hospital mortality in patients with SAKI receiving RRT.

Methods

Patients with SAKI receiving RRT from the MIMIC-IV database were retrospectively enrolled and randomly assigned to either the training cohort or the testing cohort in a 7:3 ratio. LASSO regression and Boruta algorithm were utilized for feature selection. Subsequently, three machine learning models-CART, SVM and LR-were constructed, and their predictive efficacy was assessed using a comprehensive set of performance indicators. Feature importance analysis was performed to determine the contribution of each feature to a model's predictions. Finally, DCA was employed to evaluate the clinical utility of the prediction models. Additionally, a clinical nomogram was developed to facilitate the interpretation and visualization of the LR model.

Results

A total of 1663 adults were ultimately enrolled and randomly allocated into the training cohort (n = 1164) or the testing cohort (n = 499). Twenty-eight variables were evaluated for feature selection, with eight ultimately retained in the final model: age, MAP, RR, lactate, Cr, PT-INR, TBIL and CVP. The LR model demonstrated commendable performance, exhibiting robust discrimination in both the training cohort (AUROC: 0.73 (95% CI 0.70-0.76); AUPRC: 0.75 (95% CI 0.72-0.79); accuracy: 0.66 (95% CI 0.63-0.68)) and the testing cohort (AUROC: 0.72 (95% CI 0.68-0.76); AUPRC: 0.73 (95% CI 0.67-0.79); accuracy: 0.65 (95% CI 0.61-0.69)). Furthermore, there was good concordance between predicted and observed values in both the training cohort (χ2 = 4.41, p = 0.82) and the testing cohort (χ2 = 4.16, p = 0.84). The results of the DCA revealed that the LR model provided a greater net benefit compared to other prediction models.

Conclusions

The LR model exhibited superior performance in predicting in-hospital mortality in patients with SAKI receiving RRT, suggesting its potential utility in identifying high-risk patients and guiding clinical decision-making.

Validation and comparison of triage-based screening strategies for sepsis

OBJECTIVE

This study sought to externally validate and compare proposed methods for stratifying sepsis risk at emergency department (ED) triage.

METHODS

This nested case/control study enrolled ED patients from four hospitals in Utah and evaluated the performance of previously-published sepsis risk scores amenable to use at ED triage based on their area under the precision-recall curve (AUPRC, which balances positive predictive value and sensitivity) and area under the receiver operator characteristic curve (AUROC, which balances sensitivity and specificity). Score performance for predicting whether patients met Sepsis-3 criteria in the ED was compared to patients' assigned ED triage score (Canadian Triage Acuity Score [CTAS]) with adjustment for multiple comparisons.

RESULTS

Among 2000 case/control patients, 981 met Sepsis-3 criteria on final adjudication. The best performing sepsis risk scores were the Predict Sepsis version #3 (AUPRC 0.183, 95 % CI 0.148-0.256; AUROC 0.859, 95 % CI 0.843-0.875) and Borelli scores (AUPRC 0.127, 95 % CI 0.107-0.160, AUROC 0.845, 95 % CI 0.829-0.862), which significantly outperformed CTAS (AUPRC 0.038, 95 % CI 0.035-0.042, AUROC 0.650, 95 % CI 0.628-0.671, p < 0.001 for all AUPRC and AUROC comparisons). The Predict Sepsis and Borelli scores exhibited sensitivity of 0.670 and 0.678 and specificity of 0.902 and 0.834, respectively, at their recommended cutoff values and outperformed Systemic Inflammatory Response Syndrome (SIRS) criteria (AUPRC 0.083, 95 % CI 0.070-0.102, p = 0.052 and p = 0.078, respectively; AUROC 0.775, 95 % CI 0.756-0.795, p < 0.001 for both scores).

CONCLUSIONS

The Predict Sepsis and Borelli scores exhibited improved performance including increased specificity and positive predictive values for sepsis identification at ED triage compared to CTAS and SIRS criteria.

Boosting the accuracy of existing models by updating and extending: using a multicenter COVID-19 ICU cohort as a proxy

Most published prediction models for Coronavirus Disease 2019 (COVID-19) were poorly reported, at high risk of bias, and heterogeneous in model performance. To tackle methodological challenges faced in previous prediction studies, we investigated whether model updating and extending improves mortality prediction, using the Intensive Care Unit (ICU) as a proxy. All COVID-19 patients admitted to seven ICUs in the Euregio-Meuse Rhine during the first pandemic wave were included. The 4C Mortality and SEIMC scores were selected as promising prognostic models from an external validation study. Five predictors could be estimated based on cohort size. TRIPOD guidelines were followed and logistic regression analyses with the linear predictor, APACHE II score, and country were performed. Bootstrapping with backward selection was applied to select variables for the final model. Additionally, shrinkage was performed. Model discrimination was displayed as optimism-corrected areas under the ROC curve and calibration by calibration slopes and plots. The mortality rate of the 551 included patients was 36%. Discrimination of the 4C Mortality and SEIMC scores increased from 0.70 to 0.74 and 0.70 to 0.73 and calibration plots improved compared to the original models after updating and extending. Mortality prediction can be improved after updating and extending of promising models.

Predictive Models for Sustained, Uncontrolled Hypertension and Hypertensive Crisis Based on Electronic Health Record Data: Algorithm Development and Validation

Background

Assessing disease progression among patients with uncontrolled hypertension is important for identifying opportunities for intervention.

Objective

We aim to develop and validate 2 models, one to predict sustained, uncontrolled hypertension (≥2 blood pressure [BP] readings ≥140/90 mm Hg or ≥1 BP reading ≥180/120 mm Hg) and one to predict hypertensive crisis (≥1 BP reading ≥180/120 mm Hg) within 1 year of an index visit (outpatient or ambulatory encounter in which an uncontrolled BP reading was recorded).

Methods

Data from 142,897 patients with uncontrolled hypertension within Atrium Health Greater Charlotte in 2018 were used. Electronic health record-based predictors were based on the 1-year period before a patient's index visit. The dataset was randomly split (80:20) into a training set and a validation set. In total, 4 machine learning frameworks were considered: L2-regularized logistic regression, multilayer perceptron, gradient boosting machines, and random forest. Model selection was performed with 10-fold cross-validation. The final models were assessed on discrimination (C-statistic), calibration (eg, integrated calibration index), and net benefit (with decision curve analysis). Additionally, internal-external cross-validation was performed at the county level to assess performance with new populations and summarized using random-effect meta-analyses.

Results

In internal validation, the C-statistic and integrated calibration index were 0.72 (95% CI 0.71-0.72) and 0.015 (95% CI 0.012-0.020) for the sustained, uncontrolled hypertension model, and 0.81 (95% CI 0.79-0.82) and 0.009 (95% CI 0.007-0.011) for the hypertensive crisis model. The models had higher net benefit than the default policies (ie, treat-all and treat-none) across different decision thresholds. In internal-external cross-validation, the pooled performance was consistent with internal validation results; in particular, the pooled C-statistics were 0.70 (95% CI 0.69-0.71) and 0.79 (95% CI 0.78-0.81) for the sustained, uncontrolled hypertension model and hypertensive crisis model, respectively.

Conclusions

An electronic health record-based model predicted hypertensive crisis reasonably well in internal and internal-external validations. The model can potentially be used to support population health surveillance and hypertension management. Further studies are needed to improve the ability to predict sustained, uncontrolled hypertension.

Developing a logistic regression model to predict spontaneous preterm birth from maternal socio-demographic and obstetric history at initial pregnancy registration

BACKGROUND

Current predictive machine learning techniques for spontaneous preterm birth heavily rely on a history of previous preterm birth and/or costly techniques such as fetal fibronectin and ultrasound measurement of cervical length to the disadvantage of those considered at low risk and/or those who have no access to more expensive screening tools.

AIMS AND OBJECTIVES

We aimed to develop a predictive model for spontaneous preterm delivery < 37 weeks using socio-demographic and clinical data readily available at booking -an approach which could be suitable for all women regardless of their previous obstetric history.

METHODS

We developed a logistic regression model using seven feature variables derived from maternal socio-demographic and obstetric history from a preterm birth (n = 917) and a matched full-term (n = 100) cohort in 2018 and 2020 at a tertiary obstetric unit in the UK. A three-fold cross-validation technique was applied with subsets for data training and testing in Python® (version 3.8) using the most predictive factors. The model performance was then compared to the previously published predictive algorithms.

RESULTS

The retrospective model showed good predictive accuracy with an AUC of 0.76 (95% CI: 0.71-0.83) for spontaneous preterm birth, with a sensitivity and specificity of 0.71 (95% CI: 0.66-0.76) and 0.78 (95% CI: 0.63-0.88) respectively based on seven variables: maternal age, BMI, ethnicity, smoking, gestational type, substance misuse and parity/obstetric history.

CONCLUSION

Pending further validation, our observations suggest that key maternal demographic features, incorporated into a traditional mathematical model, have promising predictive utility for spontaneous preterm birth in pregnant women in our region without the need for cervical length and/or fetal fibronectin.

Towards accurate screening and prevention for PTSD (2-ASAP): protocol of a longitudinal prospective cohort study

BACKGROUND

Effective preventive interventions for PTSD rely on early identification of individuals at risk for developing PTSD. To establish early post-trauma who are at risk, there is a need for accurate prognostic risk screening instruments for PTSD that can be widely implemented in recently trauma-exposed adults. Achieving such accuracy and generalizability requires external validation of machine learning classification models. The current 2-ASAP cohort study will perform external validation on both full and minimal feature sets of supervised machine learning classification models assessing individual risk to follow an adverse PTSD symptom trajectory over the course of 1 year. We will derive these models from the TraumaTIPS cohort, separately for men and women.

METHOD

The 2-ASAP longitudinal cohort will include N = 863 adults (N = 436 females, N = 427 males) who were recently exposed to acute civilian trauma. We will include civilian victims of accidents, crime and calamities at Victim Support Netherlands; and who were presented for medical evaluation of (suspected) traumatic injuries by emergency transportation to the emergency department. The baseline assessment within 2 months post-trauma will include self-report questionnaires on demographic, medical and traumatic event characteristics; potential risk and protective factors for PTSD; PTSD symptom severity and other adverse outcomes; and current best-practice PTSD screening instruments. Participants will be followed at 3, 6, 9, and 12 months post-trauma, assessing PTSD symptom severity and other adverse outcomes via self-report questionnaires.

DISCUSSION

The ultimate goal of our study is to improve accurate screening and prevention for PTSD in recently trauma-exposed civilians. To enable future large-scale implementation, we will use self-report data to inform the prognostic models; and we will derive a minimal feature set of the classification models. This can be transformed into a short online screening instrument that is user-friendly for recently trauma-exposed adults to fill in. The eventual short online screening instrument will classify early post-trauma which adults are at risk for developing PTSD. Those at risk can be targeted and may subsequently benefit from preventive interventions, aiming to reduce PTSD and relatedly improve psychological, functional and economic outcomes.

Indirect estimation of the prevalence of type 2 diabetes mellitus in the sub-population of Tehran: using non-laboratory risk-score models in Iran

BACKGROUND

The prevalence of type 2 diabetes mellitus (T2DM) in the population covered by the Tehran University of Medical Sciences is unclear but crucial for healthcare programs. This study aims to validate four non-laboratory risk-score models, the American Diabetes Association (ADA) Risk Score, Australian Type 2 Diabetes Risk Assessment Tool (AUSDRISK), Finnish Diabetes Risk Score (FINDRISC), and TOPICS Diabetes Screening Score, for identifying undiagnosed diabetes and indirectly estimate the prevalence of T2DM in a subset of the Tehranian population using the selected model.

METHODS

This research consisted of two main parts. In the first part, non-laboratory risk-score models to identify undiagnosed T2DM were validated using Iranian data from STEPs 2016 survey. The model performance was evaluated through the Area Under the Curve (AUC) and calibration via the observed-to-expected (O/E) ratio. Additional independent data from STEPs 2011 survey in Iran were utilized to test the model results by comparing indirect prevalence estimates with observed estimates. In the second part, the prevalence of T2DM was estimated indirectly by applying the selected model to a representative random sample from a Tehranian population telephone survey conducted in 2023.

RESULTS

Among the different models used, AUSDRISK showed the best performance in both discrimination (AUC (95% confidence interval (CI)): 0.80 (0.78, 0.81)) and calibration (O/E ratio = 1.01). After updating the original model, there was no change in the AUC value or calibration. Additionally, our findings indicate that the indirect estimates are nearly identical to the observed values in STEPs 2011 survey. In the second part of the study, by applying the recalibrated model to a subsample, the indirect prevalence of undiagnosed diabetes and T2DM (95% CI) were estimated at 4.18% (3.87, 4.49) and 11.1% (9.34, 13.1), respectively.

CONCLUSION

Given the strong performance of the model, it appears that indirect method can provide a cost-effective and simple approach to assess disease prevalence and intervention effectiveness.

Deep brain stimulation versus vagus nerve stimulation for the motor function of poststroke hemiplegia: study protocol for a multicentre randomised controlled trial

INTRODUCTION

Deep brain stimulation (DBS) and vagus nerve stimulation (VNS) can improve motor function in patients with poststroke hemiplegia. No comparison study exists.

METHODS AND ANALYSIS

This is a randomised, double-blind, controlled clinical trial involving 64 patients who had their first stroke at least 6 months ago and are experiencing poststroke limb dysfunction. These patients must receive necessary support at home and consent to participate. The aim is to evaluate the effectiveness and safety of DBS and VNS therapies. Patients are excluded if they have implantable devices that are sensitive to electrical currents, severe abnormalities in their lower limbs or are unable to comply with the trial procedures. The study has two parallel, distinct treatment arms: the Stimulation Group and the Sham Group. Initially, the Stimulation Group will undergo immediate electrical stimulation postsurgery, while the Sham Group will receive non-stimulation 1 month later. After 3 months, these groups will swap treatments, with the Stimulation Group discontinuing stimulation and the Sham Group initiating stimulation. Six months later, both groups will resume active stimulation. Our primary outcomes will meticulously assess motor function improvements, using the Fugl-Meyer Assessment, and safety, monitored by tracking adverse reaction rates. Furthermore, we will gain a comprehensive view of patient outcomes by evaluating secondary measures, including clinical improvement (National Institutes of Health Stroke Scale), surgical complications/side effects, quality of life (36-item Short Form Questionnaire) and mental health status (Hamilton Anxiety Rating Scale/Hamilton Depression Rating Scale). To ensure a thorough understanding of the long-term effects, we will conduct follow-ups at 9 and 12 months postsurgery, with additional long-term assessments at 15 and 18 months. These follow-ups will assess the sustained performance and durability of the treatment effects. The statistical analysis will uncover the optimal treatment strategy for poststroke hemiplegia, providing valuable insights for clinicians and patients alike.

ETHICS AND DISSEMINATION

This study was reviewed and approved by the Ethical Committee of Chinese PLA General Hospital (S2022-789-01). The findings will be submitted for publication in peer-reviewed journals with online accessibility, ensuring adherence to the conventional scientific publishing process while clarifying how the research outcomes will be disseminated and accessed.

TRIAL REGISTRATION NUMBER

NCT06121947.

Prospective external validation of IOTA methods for classifying adnexal masses and retrospective assessment of two-step strategy using benign descriptors and ADNEX model: Portuguese multicenter study

OBJECTIVES

To externally and prospectively validate the International Ovarian Tumor Analysis (IOTA) Simple Rules (SRs), Logistic Regression model 2 (LR2) and Assessment of Different NEoplasias in the adneXa (ADNEX) model in a Portuguese population, comparing these approaches with subjective assessment and the risk-of-malignancy index (RMI), as well as with each other. This study also aimed to retrospectively validate the IOTA two-step strategy, using modified benign simple descriptors (MBDs) followed by the ADNEX model in cases in which MBDs were not applicable.

METHODS

This was a prospective multicenter diagnostic accuracy study conducted between January 2016 and December 2021 of consecutive patients with an ultrasound diagnosis of at least one adnexal tumor, who underwent surgery at one of three tertiary referral centers in Lisbon, Portugal. All ultrasound assessments were performed by Level-II or -III sonologists with IOTA certification. Patient clinical data and serum CA 125 levels were collected from hospital databases. Each adnexal mass was classified as benign or malignant using subjective assessment, RMI, IOTA SRs, LR2 and the ADNEX model (with and without CA 125). The reference standard was histopathological diagnosis. In the second phase, all adnexal tumors were classified retrospectively using the two-step strategy (MBDs + ADNEX). Sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios and overall accuracy were determined for all methods. Receiver-operating-characteristics curves were constructed and corresponding areas under the curve (AUC) were determined for RMI, LR2, the ADNEX model and the two-step strategy. The ADNEX model calibration plots were constructed using locally estimated scatterplot smoothing (LOESS).

RESULTS

Of the 571 patients included in the study, 428 had benign disease and 143 had malignant disease (prevalence of malignancy, 25.0%), of which 42 had borderline ovarian tumor, 93 had primary invasive adnexal cancer and eight had metastatic tumors in the adnexa. Subjective assessment had an overall sensitivity of 97.9% and a specificity of 83.6% for distinguishing between benign and malignant lesions. RMI showed high specificity (95.6%) but very low sensitivity (58.7%), with an AUC of 0.913. The IOTA SRs were applicable in 80.0% of patients, with a sensitivity of 94.8% and specificity of 98.6%. The IOTA LR2 had a sensitivity of 84.6%, specificity of 86.9% and an AUC of 0.939, at a malignancy risk cut-off of 10%. At the same cut-off, the sensitivity, specificity and AUC for the ADNEX model with vs without CA 125 were 95.8% vs 98.6%, 82.5% vs 79.7% and 0.962 vs 0.960, respectively. The ADNEX model gave heterogeneous results for distinguishing between benign masses and different subtypes of malignancy, with the highest AUC (0.991) for discriminating benign masses from primary invasive adnexal cancer Stages II-IV, and the lowest AUC (0.696) for discriminating primary invasive adnexal cancer Stage I from metastatic lesion in the adnexa. The calibration plot suggested underestimation of the risk by the ADNEX model compared with the observed proportion of malignancy. The MBDs were applicable in 26.3% (150/571) of cases, of which none was malignant. The two-step strategy using the ADNEX model in the second step only, with and without CA 125, had AUCs of 0.964 and 0.961, respectively, which was similar to applying the ADNEX model in all patients.

CONCLUSIONS

The IOTA methods showed good-to-excellent performance in the Portuguese population, outperforming RMI. The ADNEX model was superior to other methods in terms of accuracy, but interpretation of its ability to distinguish between malignant subtypes was limited by sample size and large differences in the prevalence of tumor subtypes. The IOTA MBDs are reliable in identifying benign disease. The two-step strategy comprising application of MBDs followed by the ADNEX model if MBDs are not applicable, is suitable for daily clinical practice, circumventing the need to calculate the risk of malignancy in all patients. © 2024 International Society of Ultrasound in Obstetrics and Gynecology.

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

BACKGROUND

External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential.

METHODS

A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule.

RESULTS

We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data.

DISCUSSION

Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.