Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration

Development of a Prediction Model for Surgery or Early Mortality at the Time of Initial Assessment for Necrotizing Enterocolitis

OBJECTIVE

 No available scale, at the time of initial evaluation for necrotizing enterocolitis (NEC), accurately predicts, that is, with an area under the curve (AUC) ≥0.9, which preterm infants will undergo surgery for NEC stage III or die within a week.

STUDY DESIGN

 This is a retrospective cohort study (n = 261) of preterm infants with <33 weeks' gestation or <1,500 g birth weight with either suspected or with definite NEC born at Parkland Hospital between 2009 and 2021. A prediction model using the new HASOFA score (Hyperglycemia, Hyperkalemia, use of inotropes for Hypotension during the prior week, Acidemia, Neonatal Sequential Organ Failure Assessment [nSOFA] score) was compared with a similar model using the nSOFA score.

RESULTS

 Among 261 infants, 112 infants had NEC stage I, 68 with NEC stage II, and 81 with NEC stage III based on modified Bell's classification. The primary outcome, surgery for NEC stage III or death within a week, occurred in 81 infants (surgery in 66 infants and death in 38 infants). All infants with pneumoperitoneum or abdominal compartment syndrome either died or had surgery. The HASOFA and the nSOFA scores were evaluated in 254 and 253 infants, respectively, at the time of the initial workup for NEC. Both models were internally validated. The HASOFA model was a better predictor of surgery for NEC stage III or death within a week than the nSOFA model, with greater AUC 0.909 versus 0.825, respectively, p < 0.001. Combining HASOFA at initial assessment with concurrent or later presence of abdominal wall erythema or portal gas improved the prediction surgery for NEC stage III or death with AUC 0.942 or 0.956, respectively.

CONCLUSION

 Using this new internally validated prediction model, surgery for NEC stage III or death within a week can be accurately predicted at the time of initial assessment for NEC.

KEY POINTS

· No available scale, at initial evaluation, accurately predicts which preterm infants will undergo surgery for NEC stage III or die within a week.. · In this retrospective cohort study of 261 preterm infants with either suspected or definite NEC we developed a new prediction model (HASOFA score).. · The HASOFA-model had high discrimination (AUC: 0.909) and excellent calibration and was internally validated..

Preoperative risk prediction models for acute kidney injury after noncardiac surgery: an independent external validation cohort study

BACKGROUND

Numerous models have been developed to predict acute kidney injury (AKI) after noncardiac surgery, yet there is a lack of independent validation and comparison among them.

METHODS

We conducted a systematic literature search to review published risk prediction models for AKI after noncardiac surgery. An independent external validation was performed using a retrospective surgical cohort at a large Chinese hospital from January 2019 to October 2022. The cohort included patients undergoing a wide range of noncardiac surgeries with perioperative creatinine measurements. Postoperative AKI was defined according to the Kidney Disease Improving Global Outcomes creatinine criteria. Model performance was assessed in terms of discrimination (area under the receiver operating characteristic curve, AUROC), calibration (calibration plot), and clinical utility (net benefit), before and after model recalibration through intercept and slope updates. A sensitivity analysis was conducted by including patients without postoperative creatinine measurements in the validation cohort and categorising them as non-AKI cases.

RESULTS

Nine prediction models were evaluated, each with varying clinical and methodological characteristics, including the types of surgical cohorts used for model development, AKI definitions, and predictors. In the validation cohort involving 13,186 patients, 650 (4.9%) developed AKI. Three models demonstrated fair discrimination (AUROC between 0.71 and 0.75); other models had poor or failed discrimination. All models exhibited some miscalibration; five of the nine models were well-calibrated after intercept and slope updates. Decision curve analysis indicated that the three models with fair discrimination consistently provided a positive net benefit after recalibration. The results were confirmed in the sensitivity analysis.

CONCLUSIONS

We identified three models with fair discrimination and potential clinical utility after recalibration for assessing the risk of acute kidney injury after noncardiac surgery.

A pilot clinical risk model to predict polymyxin-induced nephrotoxicity: a real-world, retrospective cohort study

OBJECTIVES

Polymyxin-induced nephrotoxicity (PIN) is a major safety concern and challenge in clinical practice, which limits the clinical use of polymyxins. This study aims to investigate the risk factors and to develop a scoring tool for the early prediction of PIN.

METHODS

Data on critically ill patients who received intravenous polymyxin B or colistin sulfate for over 24 h were collected. Logistic regression with the least absolute shrinkage and selection operator (LASSO) was used to identify variables that are associated with outcomes. The eXtreme Gradient Boosting (XGB) classifier algorithm was used to further visualize factors with significant differences. A prediction model for PIN was developed through binary logistic regression analysis and the model was assessed by temporal validation and external validation. Finally, a risk-scoring system was developed based on the prediction model.

RESULTS

Of 508 patients, 161 (31.6%) patients developed PIN. Polymyxin type, loading dose, septic shock, concomitant vasopressors and baseline blood urea nitrogen (BUN) level were identified as significant predictors of PIN. All validation exhibited great discrimination, with the AUC of 0.742 (95% CI: 0.696-0.787) for internal validation, of 0.708 (95% CI: 0.605-0.810) for temporal validation and of 0.874 (95% CI: 0.759-0.989) for external validation, respectively. A simple risk-scoring tool was developed with a total risk score ranging from -3 to 4, corresponding to a risk of PIN from 0.79% to 81.24%.

CONCLUSIONS

This study established a prediction model for PIN. Before using polymyxins, the simple risk-scoring tool can effectively identify patients at risk of developing PIN within a range of 7% to 65%.

CLEAR Thrombectomy Score: An Index to Estimate the Probability of Good Functional Outcome With or Without Endovascular Treatment in the Late Window for Anterior Circulation Occlusion

BACKGROUND

With the expanding eligibility for endovascular therapy (EVT) of patients presenting in the late window (6-24 hours after last known well), we aimed to derive a score to predict favorable outcomes associated with EVT versus best medical management.

METHODS AND RESULTS

A multinational observational cohort of patients from the CLEAR (Computed Tomography for Late Endovascular Reperfusion) study with proximal intracranial occlusion (2014-2022) was queried (n=58 sites). Logistic regression analyses were used to derive a 9-point score for predicting good functional outcome (modified Rankin Scale score 0-2 or return to premorbid modified Rankin Scale score) at 90 days, with sensitivity analyses for prespecified subgroups conducted using bootstrapped random forest regressions. Secondary outcomes included 90-day functional independence (modified Rankin Scale score 0-2), poor outcome (modified Rankin Scale score 5-6), and 90-day survival. The score was externally validated with a single-center cohort (2014-2023). Of the 3231 included patients (n=2499 EVT), a 9-point score included age, early computed tomography ischemic changes, and stroke severity, with higher points indicating a higher probability of a good functional outcome. The areas under the curve for the primary outcome among EVT and best medical management subgroups were 0.72 (95% CI, 0.70-0.74) and 0.87 (95% CI, 0.84-0.90), respectively, with similar performance in the external validation cohort (area under the curve, 0.71 [95% CI, 0.66-0.76]). There was a significant interaction between the score and EVT for good functional outcome, functional independence, and poor outcome (all Pinteraction<0.001), with greater benefit favoring patients with lower and midrange scores.

CONCLUSIONS

This score is a pragmatic tool that can estimate the probability of a good outcome with EVT in the late window.

REGISTRATION

URL: https://www.Clinicaltrials.gov; Unique identifier: NCT04096248.

Using SWATH-MS to identify new molecular biomarkers in gingival crevicular fluid for detecting periodontitis and its response to treatment

AIM

To identify new biomarkers to detect untreated and treated periodontitis in gingival crevicular fluid (GCF) using sequential window acquisition of all theoretical mass spectra (SWATH-MS).

MATERIALS AND METHODS

GCF samples were collected from 44 periodontally healthy subjects and 40 with periodontitis (Stages III-IV). In the latter, 25 improved clinically 2 months after treatment. Samples were analysed using SWATH-MS, and proteins were identified by the UniProt human-specific database. The diagnostic capability of the proteins was determined with generalized additive models to distinguish the three clinical conditions.

RESULTS

In the untreated periodontitis vs. periodontal health modelling, five proteins showed excellent or good bias-corrected (bc)-sensitivity/bc-specificity values of >80%. These were GAPDH, ZG16B, carbonic anhydrase 1, plasma protease inhibitor C1 and haemoglobin subunit beta. GAPDH with MMP-9, MMP-8, zinc-α-2-glycoprotein and neutrophil gelatinase-associated lipocalin and ZG16B with cornulin provided increased bc-sensitivity/bc-specificity of >95%. For distinguishing treated periodontitis vs. periodontal health, most of these proteins and their combinations revealed a predictive ability similar to previous modelling. No model obtained relevant results to differentiate between periodontitis conditions.

CONCLUSIONS

New single and dual GCF protein biomarkers showed outstanding results in discriminating untreated and treated periodontitis from periodontal health. Periodontitis conditions were indistinguishable. Future research must validate these findings.

Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors

OBJECTIVES

Ectopic pregnancy (EP) is a major high-risk outcome following a pregnancy of unknown location (PUL) classification. Biochemical markers are used to triage PUL as high vs low risk to guide appropriate follow-up. The M6 model is currently the best risk-prediction model. We aimed to update the M6 model and evaluate whether performance can be improved by including clinical factors.

METHODS

This prospective cohort study recruited consecutive PUL between January 2015 and January 2017 at eight units (Phase 1), with two centers continuing recruitment between January 2017 and March 2021 (Phase 2). Serum samples were collected routinely and sent for β-human chorionic gonadotropin (β-hCG) and progesterone measurement. Clinical factors recorded were maternal age, pain score, bleeding score and history of EP. Based on transvaginal ultrasonography and/or biochemical confirmation during follow-up, PUL were classified subsequently as failed PUL (FPUL), intrauterine pregnancy (IUP) or EP (including persistent PUL (PPUL)). The M6 models with (M6P ) and without (M6NP ) progesterone were refitted and extended with clinical factors. Model validation was performed using internal-external cross-validation (IECV) (Phase 1) and temporal external validation (EV) (Phase 2). Missing values were handled using multiple imputation.

RESULTS

Overall, 5473 PUL were recruited over both phases. A total of 709 PUL were excluded because maternal age was < 16 years or initial β-hCG was ≤ 25 IU/L, leaving 4764 (87%) PUL for analysis (2894 in Phase 1 and 1870 in Phase 2). For the refitted M6P model, the area under the receiver-operating-characteristics curve (AUC) for EP/PPUL vs IUP/FPUL was 0.89 for IECV and 0.84-0.88 for EV, with respective sensitivities of 94% and 92-93%. For the refitted M6NP model, the AUCs were 0.85 for IECV and 0.82-0.86 for EV, with respective sensitivities of 92% and 93-94%. Calibration performance was good overall, but with heterogeneity between centers. Net Benefit confirmed clinical utility. The change in AUC when M6P was extended to include maternal age, bleeding score and history of EP was between -0.02 and 0.01, depending on center and phase. The corresponding change in AUC when M6NP was extended was between -0.01 and 0.03. At the 5% threshold to define high risk of EP/PPUL, extending M6P altered sensitivity by -0.02 to -0.01, specificity by 0.03 to 0.04 and Net Benefit by -0.005 to 0.006. Extending M6NP altered sensitivity by -0.03 to -0.01, specificity by 0.05 to 0.07 and Net Benefit by -0.005 to 0.006.

CONCLUSIONS

The updated M6 model offers accurate diagnostic performance, with excellent sensitivity for EP. Adding clinical factors to the model improved performance in some centers, especially when progesterone levels were not suitable or unavailable. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Derivation and external validation of a prediction model for pneumococcal urinary antigen test positivity in patients with community-acquired pneumonia

Objective

Derive and externally validate a prediction model for pneumococcal urinary antigen test (pUAT) positivity.

Methods

Retrospective cohort study of adults admitted with community-acquired pneumonia (CAP) to 177 U.S. hospitals in the Premier Database (derivation and internal validation samples) or 12 Cleveland Clinic hospitals (external validation sample). We utilized multivariable logistic regression to predict pUAT positivity in the derivation dataset, followed by model performance evaluation in both validation datasets. Potential predictors included demographics, comorbidities, clinical findings, and markers of disease severity.

Results

Of 198,130 Premier patients admitted with CAP, 27,970 (14.1%) underwent pUAT; 1962 (7.0%) tested positive. The strongest predictors of pUAT positivity were history of pneumococcal infection in the previous year (OR 6.99, 95% CI 4.27-11.46), severe CAP on admission (OR 1.76, 95% CI 1.56-1.98), substance abuse (OR 1.57, 95% CI 1.27-1.93), smoking (OR 1.23, 95% CI 1.09-1.39), and hyponatremia (OR 1.35, 95% CI 1.17-1.55). Negative predictors included IV antibiotic use in past year (OR 0.65, 95% CI 0.52-0.82), congestive heart failure (OR 0.72, 95% CI 0.63-0.83), obesity (OR 0.71, 95% CI 0.60-0.85), and admission from skilled nursing facility (OR 0.60, 95% CI 0.45-0.78). Model c-statistics were 0.60 and 0.67 in the internal and external validation cohorts, respectively. Compared to guideline-recommended testing of severe CAP patients, our model would have detected 23% more cases with 5% fewer tests.

Conclusion

Readily available data can identify patients most likely to have a positive pUAT. Our model could be incorporated into automated clinical decision support to improve test efficiency and antimicrobial stewardship.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.