External Validation of the International Prognosis Prediction Model of IgA Nephropathy

BACKGROUND

The International IgA Nephropathy (IgAN) Network developed and validated two prognostic prediction models for IgAN, one incorporating a race parameter. These models could anticipate the risk of a 50% reduction in estimated glomerular filtration rate (eGFR) or progression to end-stage renal disease (ESRD) subsequent to an IgAN diagnosis via renal biopsy. This investigation aimed to validate the International IgA Nephropathy Prediction Tool (IIgANPT) within a contemporary Chinese cohort.

METHODS

Within this study,185 patients diagnosed with IgAN via renal biopsy at the Center for Kidney Disease, Second Affiliated Hospital of Nanjing Medical University, between January 2012 and December 2021, were encompassed. Each patient's risk of progression was assessed utilizing the IIgANPT formula. The primary outcome, a 50% decline in eGFR or progression to ESRD, was examined. Two predictive models, one inclusive and the other exclusive of a race parameter, underwent evaluation via receiver-operating characteristic (ROC) curves, subgroup survival analyses, calibration plots, and decision curve analyses.

RESULTS

The median follow-up duration within our cohort spanned 5.1 years, during which 18 patients encountered the primary outcome. The subgroup survival curves exhibited distinct separations, and the comparison of clinical and histological characteristics among the risk subgroups revealed significant differences. Both models demonstrated outstanding discrimination, evidenced by the areas under the ROC curve at five years: 0.882 and 0.878. Whether incorporating the race parameter or not, both prediction models exhibited acceptable calibration. Decision curve analysis affirmed the favorable clinical utility of both models.

CONCLUSIONS

Both prognostic risk evaluation models for IgAN exhibited remarkable discrimination, sound calibration, and acceptable clinical utility.

External validation of the prediction model of intradialytic hypotension: a multicenter prospective cohort study

OBJECTIVE

Intradialytic hypotension (IDH) is a common and serious complication in patients with Maintenance Hemodialysis (MHD). The purpose of this study is to externally verify three IDH risk prediction models recently developed by Ma et al. and recalibrate, update and present the optimal model to improve the accuracy and applicability of the model in clinical environment.

METHODS

A multicenter prospective cohort study of patients from 11 hemodialysis centers in Sichuan Province, China, was conducted using convenience sampling from March 2022 to July 2022, with a follow-up period of 1 month. Model performance was assessed by: (1) Discrimination: Evaluated through the computation of the Area Under Curve (AUC) and its corresponding 95% confidence intervals. (2) Calibration: scrutinized through visual inspection of the calibration plot and utilization of the Brier score. (3) The incremental value of risk prediction and the utility of updating the model were gauged using NRI (Net Reclassification Improvement) and IDI (Integrated Discrimination Improvement). Decision Curve Analysis (DCA) was employed to evaluate the clinical benefit of updating the model.

RESULTS

The final cohort comprised 2235 individuals undergoing maintenance hemodialysis, exhibiting a 14.6% occurrence rate of IDH. The externally validated Area Under the Curve (AUC) values for the three original prediction models were 0.746 (95% CI: 0.718 to 0.775), 0.709 (95% CI: 0.679 to 0.739), and 0.735 (95% CI: 0.706 to 0.764) respectively. Conversely, the AUC value for the recalibrated and updated columnar plot model reached 0.817 (95% CI: 0.791 to 0.842), accompanied by a Brier score of 0.081. Furthermore, Decision Curve Analysis (DCA) exhibited a net benefit within the threshold probability range of 15.2% to 87.1%.

CONCLUSION

Externally validated, recalibrated, updated, and presented IDH prediction models may serve as a valuable instrument for evaluating IDH risk in clinical practice. Furthermore, they hold the potential to guide clinical providers in discerning individuals at risk and facilitating judicious clinical intervention decisions.

An ultrasound-based nomogram for predicting axillary node pathologic complete response after neoadjuvant chemotherapy in breast cancer: Modeling and external validation

INTRODUCTION

The staging and treatment of axillary nodes in breast cancer have become a focus of research. For breast cancer patients with fine-needle aspiration-or core needle biopsy-confirmed positive nodes, axillary lymph node dissection (ALND) after neoadjuvant chemotherapy (NAC) is still a standard treatment. However, some patients achieve an axillary pathologic complete response (pCR) after NAC. In this study, the authors sought to construct a model to predict axillary pCR in patients with positive axillary lymph nodes (cN+) breast cancer.

METHODS

Data from patients with pathologically proven cN+ breast cancer treated with NAC followed by ALND between January 2010 and April 2019 at the Peking University Cancer Hospital were reviewed. Axillary lymph node status was assessed using ultrasonography before and after NAC. The patient cohort was assigned to the construction and internal validation cohorts according to admission time. A nomogram was constructed based on the significant factors associated with axillary pCR. The predictive performance of the model was externally validated using data from Peking University First Hospital.

RESULTS

This study included 953 and 267 patients from Peking University Cancer Hospital and Peking University First Hospital, respectively. In the construction cohort, 39.7% (238 of 600) of patients achieved axillary pCR after NAC. The result of multivariate logistic regression analysis showed that tumor grade, clinical nodal response, NAC regimen, tumor pCR, lymphovascular invasion, and tumor biologic subtype were significant independent predictors of ypN0 (p < 0.05). The areas under the receiver operating characteristic curves for the construction, validation, and independent testing cohorts were 0.87 (95% confidence interval [CI], 0.84-0.90), 0.83 (95% CI, 0.79-0.87), and 0.84 (0.79-0.89), respectively.

CONCLUSIONS

A nomogram was constructed to predict the pCR of axillary lymph nodes after NAC for breast cancer. Validation of both the internal and external cohorts achieved good predictive performance, indicating that the model has preliminary clinical application prospects.

Development of a Prognostic Model for Early Breast Cancer Integrating Neutrophil to Lymphocyte Ratio and Clinical-Pathological Characteristics

BACKGROUND

Breast cancer-related inflammation is critical in tumorigenesis, cancer progression, and patient prognosis. Several inflammatory markers derived from peripheral blood cells count, such as the neutrophil-lymphocyte ratio (NLR), platelet-lymphocyte ratio (PLR), monocyte-lymphocyte ratio (MLR), and systemic immune-inflammation index (SII) are considered as prognostic markers in several types of malignancy.

METHODS

We investigate and validate a prognostic model in early patients with breast cancer to predict disease-free survival (DFS) based on readily available baseline clinicopathological prognostic factors and preoperative peripheral blood-derived indexes.

RESULTS

We analyzed a training cohort of 710 patients and 2 external validation cohorts of 980 and 157 patients with breast cancer, respectively, with different demographic origins. An elevated preoperative NLR is a better DFS predictor than others scores. The prognostic model generated in this study was able to classify patients into 3 groups with different risks of relapse based on ECOG-PS, presence of comorbidities, T and N stage, PgR status, and NLR.

CONCLUSION

Prognostic models derived from the combination of clinicopathological features and peripheral blood indices, such as NLR, represent attractive markers mainly because they are easily detectable and applicable in daily clinical practice. More comprehensive prospective studies are needed to unveil their actual effectiveness.

A Modification of the American Joint Committee on Cancer Nomogram for Undifferentiated Sarcoma With External Validation and Risk Stratification

BACKGROUND

The aim of this study is to establish a model to predict the overall survival (OS) and stratify the risk of postoperative patients with undifferentiated sarcoma.

METHODS

A total of 452 postoperative patients with undifferentiated sarcoma in the trunk and extremity from the Surveillance, Epidemiology, and End Results database were enrolled as the training cohort. We collected a group of 163 undifferentiated sarcoma patients from our center as the external validation cohort. Cox proportional hazards regression model was used to screen survival-associated factors for the construction of the nomogram. Concordance-indexes (C-indexes), calibration curves, and receiver operating characteristics (ROCs) curves were applied for the discrimination and calibration of the nomogram. The cutoff value of nomogram-based total points was applied to stratify the risk of patients.

RESULTS

A nomogram was developed incorporating four independent factors: age, tumor site, eighth AJCC stage, and radiotherapy. The nomogram showed good prognostic accuracy and excellent agreement in the training and validation cohort, with C-indexes of .701 (95% confidence interval [CI]: .683-.719) and .700 (95% CI: 0.659-.741), respectively. Furthermore, we identified the best cutoff value of nomogram total points (103.2) as the predicted risk and divided the patients into a high-risk group and a low-risk group. Significant differences in OS between the two groups were indicated in the training cohort and external validation cohort, showing the appreciable clinical validity and clinical utility of the nomogram (P < .001).

CONCLUSION

This nomogram provides an insightful and applicable tool for individual evaluations and the distinguishment of risk for patients with undifferentiated sarcoma.

Validation of Fetal Medicine Foundation competing-risks model for small-for-gestational-age neonate in early third trimester

OBJECTIVE

To evaluate the new 36-week Fetal Medicine Foundation (FMF) competing-risks model for the prediction of small-for-gestational age (SGA) at an earlier gestation of 30 + 0 to 34 + 0 weeks.

METHODS

This was a retrospective multicenter cohort study of prospectively collected data on 3012 women with a singleton pregnancy undergoing ultrasound examination at 30 + 0 to 34 + 0 weeks' gestation as part of a universal screening program. We used the default FMF competing-risks model for prediction of SGA at 36 weeks' gestation combining maternal factors (age, obstetric and medical history, weight, height, smoking status, race, mode of conception), estimated fetal weight (EFW) and uterine artery pulsatility index (UtA-PI) to calculate risks for different cut-offs of birth-weight percentile and gestational age at delivery. We examined the accuracy of the model by means of discrimination and calibration.

RESULTS

The prediction of SGA < 3rd percentile improved with the addition of UtA-PI and with a shorter examination-to-delivery interval. For a 10% false-positive rate, maternal factors, EFW and UtA-PI predicted 88.0%, 74.4% and 72.8% of SGA < 3rd percentile delivered at < 37, < 40 and < 42 weeks' gestation, respectively. The respective values for SGA < 10th percentile were 86.1%, 69.3% and 66.2%. In terms of population stratification, if the biomarkers used are EFW and UtA-PI and the aim is to detect 90% of SGA < 10th percentile, then 10.8% of the population should be scanned within 2 weeks after the initial assessment, an additional 7.2% (total screen-positive rate (SPR), 18.0%) should be scanned within 2-4 weeks after the initial assessment and an additional 11.7% (total SPR, 29.7%) should be examined within 4-6 weeks after the initial assessment. The new model was well calibrated.

CONCLUSIONS

The 36-week FMF competing-risks model for SGA is also applicable and accurate at 30 + 0 to 34 + 0 weeks and provides effective risk stratification, especially for cases leading to delivery < 37 weeks of gestation. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Validation of the Klinrisk chronic kidney disease progression model in the FIDELITY population

Background

Chronic kidney disease (CKD) affects >800 million individuals worldwide and is often underrecognized. Early detection, identification and treatment can delay disease progression. Klinrisk is a proprietary CKD progression risk prediction model based on common laboratory data to predict CKD progression. We aimed to externally validate the Klinrisk model for prediction of CKD progression in FIDELITY (a prespecified pooled analysis of two finerenone phase III trials in patients with CKD and type 2 diabetes). In addition, we sought to identify evidence of an interaction between treatment and risk.

Methods

The validation cohort included all participants in FIDELITY up to 4 years. The primary and secondary composite outcomes included a ≥40% decrease in estimated glomerular filtration rate (eGFR) or kidney failure, and a ≥57% decrease in eGFR or kidney failure. Prediction discrimination was calculated using area under the receiver operating characteristic curve (AUC). Calibration plots were calculated by decile comparing observed with predicted risk.

Results

At time horizons of 2 and 4 years, 993 and 1795 patients experienced a primary outcome event, respectively. The model predicted the primary outcome accurately with an AUC of 0.81 for 2 years and 0.86 for 4 years. Calibration was appropriate at both 2 and 4 years, with Brier scores of 0.067 and 0.115, respectively. No evidence of interaction between treatment and risk was identified for the primary composite outcome (P = .31).

Conclusions

Our findings demonstrate the accuracy and utility of a laboratory-based prediction model for early identification of patients at the highest risk of CKD progression.

COVID-19: The Development and Validation of a New Mortality Risk Score

Background: The coronavirus disease 2019 (COVID-19) pandemic has found the whole world unprepared for its correct management. Italy was the first European country to experience the spread of the SARS-CoV-2 virus at the end of February 2020. As a result of hospital overcrowding, the quality of care delivered was not always optimal. A substantial number of patients admitted to non-ICU units could have been treated at home. It would have been extremely useful to have a score that, based on personal and clinical characteristics and simple blood tests, could have predicted with sufficient reliability the probability that a patient had or did not have a disease that could have led to their death. This study aims to develop a scoring system to identify which patients with COVID-19 are at high mortality risk upon hospital admission, to expedite and enhance clinical decision making. Methods: A retrospective analysis was performed to develop a multivariable prognostic prediction model. Results: Derivation and external validation cohorts were obtained from two Italian University Hospital databases, including 388 (10.31% deceased) and 1357 (7.68% deceased) patients with confirmed COVID-19, respectively. A multivariable logistic model was used to select seven variables associated with in-hospital death (age, baseline oxygen saturation, hemoglobin value, white blood cell count, percentage of neutrophils, platelet count, and creatinine value). Calibration and discrimination were satisfactory with a cumulative AUC for prediction mortality of 0.924 (95% CI: 0.893-0.944) in derivation cohorts and 0.808 (95% CI: 0.886-0.828) in external validation cohorts. The risk score obtained was compared with the ISARIC 4C Mortality Score, and with all the other most important scores considered so far, to evaluate the risk of death of patients with COVID-19. It performed better than all the above scores to evaluate the predictability of dying. Its sensitivity, specificity, and AUC were higher than the other COVID-19 scoring systems when the latter were calculated for the 388 patients in our derivation cohort. Conclusions: In conclusion, the CZ-COVID-19 Score may help all physicians by identifying those COVID-19 patients who require more attention to provide better therapeutic regimens or, on the contrary, by identifying those patients for whom hospitalization is not necessary and who could therefore be sent home without overcrowding healthcare facilities. We developed and validated a new risk score based on seven variables for upon-hospital admission of COVID-19 patients. It is very simple to calculate and performs better than all the other similar scores to evaluate the predictability of dying.

External Validation of Obese/Critically Ill Vancomycin Population Pharmacokinetic Models in Critically Ill Patients Who Are Obese

Obesity combined with critical illness might increase the risk of acquiring infections and hence mortality. In this patient population the pharmacokinetics of antimicrobials vary significantly, making antimicrobial dosing challenging. The objective of this study was to assess the predictive performance of published population pharmacokinetic models of vancomycin in patients who are critically ill or obese for a cohort of critically ill patients who are obese. This was a multi-center retrospective study conducted at 2 hospitals. Adult patients with a body mass index of ≥30 kg/m2 were included. PubMed was searched for published population pharmacokinetic studies in patients who were critically ill or obese. External validation was performed using Monolix software. A total of 4 models were identified in patients who were obese and 5 models were identified in patients who were critically ill. In total, 138 patients who were critically ill and obese were included, and the most accurate models for these patients were the Goti and Roberts models. In our analysis, models in patients who were critically ill outperformed models in patients who were obese. When looking at the most accurate models, both the Goti and the Roberts models had patient characteristics similar to ours in terms of age and creatinine clearance. This indicates that when selecting the proper model to apply in practice, it is important to account for all relevant variables, besides obesity.

External Validation of the Oakland Score for Predicting Safe Discharge in Patients Presenting With Lower Gastrointestinal Bleeding at the William Harvey Hospital in the United Kingdom

Introduction Lower gastrointestinal bleeds (LGIB) are defined by having a bleeding point in the gastrointestinal tract beyond the ligament of Treitz. The most common causes include diverticular bleeds, tumours, and colitis. There are no National Institute for Health and Care Excellence (NICE) guidelines regarding safe discharge of patients with LGIB. The aim of this study was to investigate the effectiveness and safety of the Oakland score, as suggested by the British Society of Gastroenterology (BSG) guidelines, in patients presenting with LGIB at William Harvey Hospital. Methods Patients with LGIB who presented to Accident & Emergency or inpatient referral from January to December 2023 were included in this retrospective study. Data was extracted from patients' Sunrise documentation. The Oakland score for each patient was calculated. Those with a score of ≤8 were deemed safe for discharge; those with a higher score were deemed unsuitable. Patients' admission, discharges, and adverse outcomes, such as representation, blood transfusion, or further intervention, were investigated. Patients with no adverse outcomes were deemed to have had a safe discharge. The area under the receiver-operating characteristic curve (AUROC) for the Oakland score and adverse outcome (and therefore safe discharge) were calculated. Results A total of 123 patients were included. These led to a total of 144 LGIB presentations to the hospital. Twenty-nine patients had an Oakland score of ≤8; 21 (72.4%) cases were initially discharged with four representations (19.0%) and eight (27.6%) were admitted although none of these suffered from any adverse outcomes. For those who scored ≤8, 25 (86.2%) were therefore deemed to have had a safe discharge. A total of 115 had a score >8; 43 (37.4%) were initially discharged, 72 (62.6%) admitted and 41 (35.7%) experienced at least one adverse outcome including 16 (13.9%) representations, 21 (18.3%) blood transfusions, three (2.6%) surgical interventions and one (0.9%) endoscopic haemostasis. Out of the 115 cases which scored >8, 74 (64.3%) were deemed to have had a safe discharge. The AUROC for safe discharge was 0.84. Conclusion The Oakland score seems to be a safe and reliable tool for identifying LGIB patients who could be safely discharged home without hospital intervention. However, further research is required to assess whether a score of >8 could be used as many patients with a higher score did not experience adverse outcomes.

Evaluation of screening performance of first-trimester competing-risks prediction model for small-for-gestational age in Asian population

OBJECTIVE

To examine the external validity of the Fetal Medicine Foundation (FMF) competing-risks model for the prediction of small-for-gestational age (SGA) at 11-14 weeks' gestation in an Asian population.

METHODS

This was a secondary analysis of a multicenter prospective cohort study in 10 120 women with a singleton pregnancy undergoing routine assessment at 11-14 weeks' gestation. We applied the FMF competing-risks model for the first-trimester prediction of SGA, combining maternal characteristics and medical history with measurements of mean arterial pressure (MAP), uterine artery pulsatility index (UtA-PI) and serum placental growth factor (PlGF) concentration. We calculated risks for different cut-offs of birth-weight percentile (< 10th , < 5th or < 3rd percentile) and gestational age at delivery (< 37 weeks (preterm SGA) or SGA at any gestational age). Predictive performance was examined in terms of discrimination and calibration.

RESULTS

The predictive performance of the competing-risks model for SGA was similar to that reported in the original FMF study. Specifically, the combination of maternal factors with MAP, UtA-PI and PlGF yielded the best performance for the prediction of preterm SGA with birth weight < 10th percentile (SGA < 10th ) and preterm SGA with birth weight < 5th percentile (SGA < 5th ), with areas under the receiver-operating-characteristics curve (AUCs) of 0.765 (95% CI, 0.720-0.809) and 0.789 (95% CI, 0.736-0.841), respectively. Combining maternal factors with MAP and PlGF yielded the best model for predicting preterm SGA with birth weight < 3rd percentile (SGA < 3rd ) (AUC, 0.797 (95% CI, 0.744-0.850)). After excluding cases with pre-eclampsia, the combination of maternal factors with MAP, UtA-PI and PlGF yielded the best performance for the prediction of preterm SGA < 10th and preterm SGA < 5th , with AUCs of 0.743 (95% CI, 0.691-0.795) and 0.762 (95% CI, 0.700-0.824), respectively. However, the best model for predicting preterm SGA < 3rd without pre-eclampsia was the combination of maternal factors and PlGF (AUC, 0.786 (95% CI, 0.723-0.849)). The FMF competing-risks model including maternal factors, MAP, UtA-PI and PlGF achieved detection rates of 42.2%, 47.3% and 48.1%, at a fixed false-positive rate of 10%, for the prediction of preterm SGA < 10th , preterm SGA < 5th and preterm SGA < 3rd , respectively. The calibration of the model was satisfactory.

CONCLUSION

The screening performance of the FMF first-trimester competing-risks model for SGA in a large, independent cohort of Asian women is comparable with that reported in the original FMF study in a mixed European population. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

A tool to predict the risk of lower extremity amputation in patients starting dialysis

BACKGROUND AND HYPOTHESIS

Non-traumatic lower extremity amputation (LEA) is a severe complication during dialysis. To inform decision-making for physicians, we developed a multivariable prediction model for LEA after starting dialysis.

METHODS

Data from the Swedish Renal Registry (SNR) between 2010 and 2020 were geographically split into a development and validation cohort. Data from NECOSAD between 1997 and 2009 were used for validation targeted at Dutch patients. Inclusion criteria were no previous LEA and kidney transplant and age ≥ 40 years at baseline. A Fine-Gray model was developed with LEA within 3 years after starting dialysis as outcome of interest. Death and kidney transplant were treated as competing events. One coefficient, ordered by expected relevance, per 20 events was estimated. Performance was assessed with calibration and discrimination.

RESULTS

SNR was split into an urban development cohort with 4 771 individuals experiencing 201 (4.8%) events and a rural validation cohort with 4.876 individuals experiencing 155 (3.2%) events. NECOSAD contained 1 658 individuals experiencing 61 (3.7%) events. Ten predictors were included: female sex, age, diabetes mellitus, peripheral artery disease, cardiovascular disease, congestive heart failure, obesity, albumin, haemoglobin and diabetic retinopathy. In SNR, calibration intercept and slope were -0.003 and 0.912 respectively. The C-index was estimated as 0.813 (0.783-0.843). In NECOSAD, calibration intercept and slope were 0.001 and 1.142 respectively. The C-index was estimated as 0.760 (0.697-0.824). Calibration plots showed good calibration.

CONCLUSION

A newly developed model to predict LEA after starting dialysis showed good discriminatory performance and calibration. By identifying high-risk individuals this model could help select patients for preventive measures.

External Validation of the Post-Concussion Symptoms Rule for Predicting Mild Traumatic Brain Injury Outcome

Persistent symptoms are common after a mild traumatic brain injury (mTBI). The Post-Concussion Symptoms (PoCS) Rule is a newly developed clinical decision rule for the prediction of persistent post-concussion symptoms (PPCS) 3 months after an mTBI. The PoCS Rule includes assessment of demographic and clinical characteristics and headache presence in the emergency department (ED), and follow-up assessment of symptoms at 7 days post-injury using two thresholds (lower/higher) for symptom scoring. We examined the PoCS Rule in an independent sample. We analyzed a clinical trial that recruited participants with mTBI from EDs in Greater Vancouver, Canada. The primary analysis used data from 236 participants, who were randomized to a usual care control group, and completed the Rivermead Postconcussion Symptoms Questionnaire at 3 months. The primary outcome was PPCS, as defined by the PoCS authors. We assessed the overall performance of the PoCS rule (area under the receiver operating characteristic curve [AUC]), sensitivity, and specificity. More than 40% of participants (median age 38 years, 59% female) reported PPCS at 3 months. Most participants (88%) were categorized as being at medium risk based on the ED assessment, and a majority were considered as being at high risk according to the final PoCS Rule (81% using a lower threshold and 72% using a higher threshold). The PoCS Rule showed a sensitivity of 93% (95% confidence interval [CI], 88-98; lower threshold) and 85% (95% CI, 78-92; higher threshold), and a specificity of 28% (95% CI, 21-36) and 37% (95% CI, 29-46), respectively. The overall performance was modest (AUC 0.61, 95% CI 0.59, 0.65). In conclusion, the PoCS Rule was sensitive for PPCS, but had a low specificity in our sample. Follow-up assessment of symptoms can improve risk stratification after mTBI.

Comparison of various prediction models in the effect of laparoscopic sleeve gastrectomy on type 2 diabetes mellitus in the Chinese population 5 years after surgery

BACKGROUND

The effect of bariatric surgery on type 2 diabetes mellitus (T2DM) control can be assessed based on predictive models of T2DM remission. Various models have been externally verified internationally. However, long-term validated results after laparoscopic sleeve gastrectomy (LSG) surgery are lacking. The best model for the Chinese population is also unknown.

METHODS

We retrospectively analyzed Chinese population data 5 years after LSG at Beijing Shijitan Hospital in China between March 2009 and December 2016. The independent t -test, Mann-Whitney U test, and chi-squared test were used to compare characteristics between T2DM remission and non-remission groups. We evaluated the predictive efficacy of each model for long-term T2DM remission after LSG by calculating the area under the curve (AUC), sensitivity, specificity, Youden index, positive predictive value (PPV), negative predictive value (NPV), and predicted-to-observed ratio, and performed calibration using Hosmer-Lemeshow test for 11 prediction models.

RESULTS

We enrolled 108 patients, including 44 (40.7%) men, with a mean age of 35.5 years. The mean body mass index was 40.3 ± 9.1 kg/m 2 , the percentage of excess weight loss (%EWL) was (75.9 ± 30.4)%, and the percentage of total weight loss (%TWL) was (29.1± 10.6)%. The mean glycated hemoglobin A1c (HbA1c) level was (7.3 ± 1.8)% preoperatively and decreased to (5.9 ± 1.0)% 5 years after LSG. The 5-year postoperative complete and partial remission rates of T2DM were 50.9% [55/108] and 27.8% [30/108], respectively. Six models, i.e., "ABCD", individualized metabolic surgery (IMS), advanced-DiaRem, DiaBetter, Dixon et al' s regression model, and Panunzi et al 's regression model, showed a good discrimination ability (all AUC >0.8). The "ABCD" (sensitivity, 74%; specificity, 80%; AUC, 0.82 [95% confidence interval [CI]: 0.74-0.89]), IMS (sensitivity, 78%; specificity, 84%; AUC, 0.82 [95% CI: 0.73-0.89]), and Panunzi et al' s regression models (sensitivity, 78%; specificity, 91%; AUC, 0.86 [95% CI: 0.78-0.92]) showed good discernibility. In the Hosmer-Lemeshow goodness-of-fit test, except for DiaRem ( P <0.01), DiaBetter ( P <0.01), Hayes et al ( P = 0.03), Park et al ( P = 0.02), and Ramos-Levi et al' s ( P <0.01) models, all models had a satifactory fit results ( P >0.05). The P values of calibration results of the "ABCD" and IMS were 0.07 and 0.14, respectively. The predicted-to-observed ratios of the "ABCD" and IMS were 0.87 and 0.89, respectively.

CONCLUSION

The prediction model IMS was recommended for clinical use because of excellent predictive performance, good statistical test results, and simple and practical design features.

External validation of the Norwegian anterior cruciate ligament reconstruction revision prediction model using patients from the STABILITY 1 Trial

PURPOSE

A machine learning-based anterior cruciate ligament (ACL) revision prediction model has been developed using Norwegian Knee Ligament Register (NKLR) data, but lacks external validation outside Scandinavia. This study aimed to assess the external validity of the NKLR model (https://swastvedt.shinyapps.io/calculator_rev/) using the STABILITY 1 randomized clinical trial (RCT) data set. The hypothesis was that model performance would be similar.

METHODS

The NKLR Cox Lasso model was selected for external validation owing to its superior performance in the original study. STABILITY 1 patients with all five predictors required by the Cox Lasso model were included. The STABILITY 1 RCT was a prospective study which randomized patients to receive either a hamstring tendon autograft (HT) alone or HT plus a lateral extra-articular tenodesis (LET). Since all patients in the STABILITY 1 trial received HT ± LET, three configurations were tested: 1: all patients coded as HT, 2: HT + LET group coded as bone-patellar tendon-bone (BPTB) autograft, 3: HT + LET group coded as unknown/other graft choice. Model performance was assessed via concordance and calibration.

RESULTS

In total, 591/618 (95.6%) STABILITY 1 patients were eligible for inclusion, with 39 undergoing revisions within 2 years (6.6%). Model performance was best when patients receiving HT + LET were coded as BPTB. Concordance was similar to the original NKLR prediction model for 1- and 2-year revision prediction (STABILITY: 0.71; NKLR: 0.68-0.69). Concordance 95% confidence interval (CI) ranged from 0.63 to 0.79. The model was well calibrated for 1-year prediction while the 2-year prediction demonstrated evidence of miscalibration.

CONCLUSION

When patients in STABILITY 1 who received HT + LET were coded as BPTB in the NKLR prediction model, concordance was similar to the index study. However, due to a wide 95% CI, the true performance of the prediction model with this Canadian and European cohort is unclear and a larger data set is required to definitively determine the external validity. Further, better calibration for 1-year predictions aligns with general prediction modelling challenges over longer periods. While not a large enough sample size to elicit the true accuracy and external validity of the prediction model when applied to North American patients, this analysis provides more support for the notion that HT plus LET performs similarly to BPTB reconstruction. In addition, despite the wide confidence interval, this study suggests optimism regarding the accuracy of the model when applied outside of Scandinavia.

LEVEL OF EVIDENCE

Level 3, cohort study.

Personality Disorder Symptoms in Intermittent Explosive Disorder: A Latent Class Analysis

Intermittent explosive disorder (IED) is characterized by recurrent reactive aggression. IED is associated with significant personality pathology that is suggestive of higher levels of general personality disorder (PD). However, little is known about how personality factors impact the severity and presentation of IED. The present study employed a latent class analysis to assess for distinct PD symptom classes within IED and to evaluate whether these classes differed in terms of severity and behavioral presentation. Statistical and clinical indicators revealed a four-class model, with latent classes distinguished primarily on general levels of PD symptoms (low, moderate, high). However, the two moderate PD symptom classes were distinguished from other classes on avoidant PD. In addition, classes differed in terms of severity and presentation, suggesting important implications for both general PD and avoidant PD comorbidity within IED. Results provide further insight into the heterogeneity within IED and suggest a more nuanced approach in treating this serious condition.

External Validation of the Risk Assessment Model of Venous Thromboembolism in Multicenter Internal Medicine Inpatients

To external validate the risk assessment model (RAM) of venous thromboembolism (VTE) in multicenter internal medicine inpatients. We prospectively collected 595 internal medical patients (310 with VTE patients, 285 non-VTE patients) were from Beijing Shijitan Hospital, Beijing Chaoyang Hospital, and the respiratory department of Beijing Tsinghua Changgeng Hospital from January 2022 to December 2022 for multicenter external validation. The prediction ability of Caprini RAM, Padua RAM, The International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) RAM, and Shijitan (SJT) RAM were compared. This study included a total of 595 internal medicine inpatients, including 242 (40.67%) in the respiratory department, 17 (2.86%) in the respiratory intensive care unit, 49 (8.24%) in the neurology department, 34 (5.71%) in the intensive care unit, 26 (4.37%) in the geriatric department, 22 (3.70%) in the emergency department, 71 (11.93%) in the nephrology department, 63 (10.59%) in the cardiology department, 24 (4.03%) in the hematology department, 6 (1.01%) in the traditional Chinese medicine department, 9 (1.51%) cases in the rheumatology department, 7 (1.18%) in the endocrinology department, 14 (2.35%) in the oncology department, and 11 (1.85%) in the gastroenterology department. Multivariate logistic regression analysis showed that among internal medicine inpatients, age > 60 years old, heart failure, nephrotic syndrome, tumors, history of VTE, and elevated D-dimer were significantly correlated with the occurrence of VTE (P < .05). The incidence of VTE increases with the increase of D-dimer. It was found that the effectiveness of SJT RAM (AUC = 0.80 ± 0.03) was better than Caprini RAM (AUC = 0.74 ± 0.03), Padua RAM (AUC = 0.72 ± 0.03) and IMPROVE RAM (AUC = 0.52 ± 0.03) (P < .05). The sensitivity and Yoden index of SJT RAM were higher than those of Caprini RAM, Pauda RAM, and IMPROVE RAM (P < .05), but specificity was not significantly different between the 4 models (P > .05). The SJT RAM derived from general hospitalized Chinese patients has effective and better predictive ability for internal medicine inpatients at risk of VTE.

External Validation and Updating of Published Models for Predicting 7-day Risk of Symptomatic Intracranial Hemorrhage after Receiving Alteplase for Acute Ischemic Stroke: A Retrospective Cohort Study

Background

Prediction scores for symptomatic intracranial hemorrhage (sICH) in acute ischemic stroke patients receiving thrombolytic therapy have been widely developed, but the external validation of these scores, especially in the Thai population, is lacking. This study aims to externally validate existing models and update the selected model to enhance its performance in our specific context.

Methods

This cohort study retrospectively collected data from medical records between 2013 and 2022. Acute ischemic stroke patients who received thrombolysis were included. All predictors were gathered at admission. External validation was performed on eight published prediction models; in addition, the observed and expected probabilities of sICH were compared. The most effective model for discrimination was then chosen for further updating using multivariable logistic regression and was bootstrapped for internal validation. Finally, a points-based system for clinical practice was developed from the optimism-corrected model.

Results

Fifty patients (10% of the 502 included cohort members) experienced sICH after undergoing thrombolysis. The SICH score outperformed the other seven models in terms of discrimination (area under the receiver operating characteristic [AuROC] curve = 0.74 [95% confidence interval {CI} 0.67 to 0.81]), but it still overstated risk (expected-to-observed outcomes [E/O] ratio = 1.7). Once updated, the optimism-corrected revised SICH model showed somewhat better calibration (E/O = 1 and calibration-in-the-large = 0), slightly worse underprediction in the moderate-to-high risk group (calibration slope = 1.152), and marginally better discrimination (AuROC = 0.78). The points-based system also demonstrated substantial agreement (88.1%) with the risk groups predicted by the logistic regression model (kappa statistic = 0.78).

Conclusion

Since the SICH score outperformed seven models in terms of discrimination, it was then modified to the Revised-SICH score, which predicted that patients with at least 5.5 points were at high risk of having sICH.

Transportability of bacterial infection prediction models for critically ill patients

OBJECTIVE

Bacterial infections (BIs) are common, costly, and potentially life-threatening in critically ill patients. Patients with suspected BIs may require empiric multidrug antibiotic regimens and therefore potentially be exposed to prolonged and unnecessary antibiotics. We previously developed a BI risk model to augment practices and help shorten the duration of unnecessary antibiotics to improve patient outcomes. Here, we have performed a transportability assessment of this BI risk model in 2 tertiary intensive care unit (ICU) settings and a community ICU setting. We additionally explored how simple multisite learning techniques impacted model transportability.

METHODS

Patients suspected of having a community-acquired BI were identified in 3 datasets: Medical Information Mart for Intensive Care III (MIMIC), Northwestern Medicine Tertiary (NM-T) ICUs, and NM "community-based" ICUs. ICU encounters from MIMIC and NM-T datasets were split into 70/30 train and test sets. Models developed on training data were evaluated against the NM-T and MIMIC test sets, as well as NM community validation data.

RESULTS

During internal validations, models achieved AUROCs of 0.78 (MIMIC) and 0.81 (NM-T) and were well calibrated. In the external community ICU validation, the NM-T model had robust transportability (AUROC 0.81) while the MIMIC model transported less favorably (AUROC 0.74), likely due to case-mix differences. Multisite learning provided no significant discrimination benefit in internal validation studies but offered more stability during transport across all evaluation datasets.

DISCUSSION

These results suggest that our BI risk models maintain predictive utility when transported to external cohorts.

CONCLUSION

Our findings highlight the importance of performing external model validation on myriad clinically relevant populations prior to implementation.

Analysis of Related Risk Factors and Prognostic Factors of Gastric Cancer with Liver Metastasis: A SEER and External Validation Based Study

Background

Gastric cancer (GC) has a poor prognosis, particularly in patients with liver metastasis (LM). This study aims to identify relevant factors associated with the occurrence of LM in GC patients and factors influencing the prognosis of gastric cancer with liver metastasis (GCLM) patients, in addition to developing diagnostic and prognostic nomograms specifically.

Patients and Methods

Overall, 6184 training data were from the Surveillance, Epidemiology, and End Results (SEER) database from 2011 to 2015. 1527 validation data were from our hospital between January 2018 and December 2022. Logistic regression was used to identify the risk factors associated with the occurrence of LM in GC patients, Cox regression was used to confirm the prognostic factors of GCLM patients. Two nomogram models were established to predict the risk and overall survival (OS) of patients with GCLM. The performance of the two models was evaluated using the area under the curve (AUC), concordance index (C-index), and calibration curves.

Results

A nomogram included five independent factors from multivariate logistic regression: sex, lymph node removal, chemotherapy, T stage and N stage were constructed to calculate the possibility of LM. Internal and external verifications of AUC were 0.786 and 0.885, respectively. The other nomogram included four independent factors from multivariate Cox regression: surgery at primary site, surgery at other site, chemotherapy, and N stage were constructed to predict OS. C-index for internal and external validations were 0.714 and 0.702, respectively, and the calibration curves demonstrated the robust discriminative ability of the models.

Conclusion

Based on the SEER database and validation data, we defined effective nomogram models to predict risk and OS in patients with GCLM. They have important value in clinical decision-making and personalized treatment.

External validation of unsupervised COVID-19 clinical phenotypes and their prognostic impact

BACKGROUND

Hospitalized patients with coronavirus disease 2019 (COVID-19) can be classified into different clinical phenotypes based on their demographic, clinical, radiology, and laboratory features. We aimed to validate in an external cohort of hospitalized COVID-19 patients the prognostic value of a previously described phenotyping system (FEN-COVID-19) and to assess the reproducibility of phenotypes development as a secondary analysis.

METHODS

Patients were classified in phenotypes A, B or C according to the severity of oxygenation impairment, inflammatory response, hemodynamic and laboratory tests according to the FEN-COVID-19 method.

RESULTS

Overall, 992 patients were included in the study, and 181 (18%), 757 (76%) and 54 (6%) of them were assigned to the FEN-COVID-19 phenotypes A, B, and C, respectively. An association with mortality was observed for phenotype C vs. A (hazard ratio [HR] 3.10, 95% confidence interval [CI] 1.81-5.30, p < 0.001) and for phenotype C vs. B (HR 2.20, 95% CI 1.50-3.23, p < 0.001). A non-statistically significant trend towards higher mortality was also observed for phenotype B vs. A (HR 1.41; 95% CI 0.92-2.15, p = 0.115). By means of cluster analysis, three different phenotypes were also identified in our cohort, with an overall similar gradient in terms of prognostic impact to that observed when patients were assigned to FEN-COVID-19 phenotypes.

CONCLUSIONS

The prognostic impact of FEN-COVID-19 phenotypes was confirmed in our external cohort, although with less difference in mortality between phenotypes A and B than in the original study.

Validation of a Cox prognostic model for tooth autotransplantation

OBJECTIVES

This study aimed to validate our Cox proportional hazards prognostic model for autotransplantation of teeth with complete root formation using prognostic index (PI) and determine whether the prognosis can be predicted.

PATIENTS AND METHODS

The Protocol group, as a training data set for validation, consisted of 259 autotransplanted teeth to create a PI using the Cox model, as described previously. The Pre-protocol group, as the first validation data set, consisted of 95 autotransplanted teeth treated without a protocol. The Post-protocol group, as the second validation data set, consisted of 61 autotransplanted teeth obtained after the establishment of the prognostic model. Because four prognostic factors, including history of root canal treatment (yes), number of roots (multirooted), source of donor tooth (maxillary tooth), and duration of edentulism (≥2.5 months), were selected as a Cox prognostic model, 16 patterns of PI were constructed. First, the autotransplantated teeth in the Protocol group were divided into low- and high-risk groups respectively according to the median of PI as the cutoff value. The survival curves of low- and high-risk groups were calculated using the Kaplan-Meier method and tested using the log-rank test. Then, in the Pre- and Post-protocol groups, all transplanted teeth were divided into low-and high-risk teeth by the median of PI and the survival curves of low- and high- risk teeth were analyzed statistically in a similar manner.

RESULTS

The survival curves of the low- and high-risk groups diverged significantly in the Protocol and Post-protocol groups. In the Pre-protocol group, the curves of the low- and high-risk groups were separated, and the low-risk survival rate was improved.

CONCLUSIONS

Our Cox prognostic model for autotransplantation of teeth with complete root formation was useful in predicting the prognosis by external validation using PI.

Drug-related readmissions in older hospitalized adults: External validation and updating of OPERAM DRA prediction tool

BACKGROUND

Drug-related readmissions (DRAs) are defined as rehospitalizations with an adverse drug event as their main or significant contributory cause. DRAs represent a major adverse health burden for older patients. A prediction model which identified older hospitalized patients at high risk of a DRA <1 year was previously developed using the OPERAM trial cohort, a European cluster randomized controlled trial including older hospitalized patients with multimorbidity and polypharmacy. This study has performed external validation and updated the prediction model consequently.

METHODS

The MedBridge trial cohort (a multicenter cluster randomized crossover trial performed in Sweden) was used as a validation cohort. It consisted of 2516 hospitalized patients aged ≥65 years. Model performance was assessed by: (1) discriminative power, assessed by the C-statistic with a 95% confidence interval (CI); (2) calibration, assessed by visual examination of the calibration plot and use of the Hosmer-Lemeshow goodness-of-fit test; and (3) overall accuracy, assessed by the scaled Brier score. Several updating methods were carried out to improve model performance.

RESULTS

In total, 2516 older patients were included in the validation cohort, of whom 582 (23.1%) experienced a DRA <1 year. In the validation cohort, the original model showed a good overall accuracy (scaled Brier score 0.03), but discrimination was moderate (C-statistic 0.62 [95% CI 0.59-0.64]), and calibration showed underestimation of risks. In the final updated model, the predictor "cirrhosis with portal hypertension" was removed and "polypharmacy" was added. This improved the model's discriminative capability to a C-statistic of 0.64 (95% CI 0.59-0.70) and enhanced calibration plots. Overall accuracy remained good.

CONCLUSIONS

The updated OPERAM DRA prediction model may be a useful tool in clinical practice to estimate the risk of DRAs in older hospitalized patients subsequent to discharge. Our efforts lay the groundwork for the future development of models with even better performance.

In-Hospital Mortality Prediction Model for Critically Ill Older Adult Patients Transferred from the Emergency Department to the Intensive Care Unit

Purpose

Studies on the prognosis of critically ill older adult patients admitted to the emergency department (ED) but requiring immediate admission to the intensive care unit (ICU) remain limited. This study aimed to develop an in-hospital mortality prediction model for critically ill older adult patients transferred from the ED to the ICU.

Patients and Methods

The training cohort was taken from the Medical Information Mart for Intensive Care IV (version 2.2) database, and the external validation cohort was taken from the Affiliated Dongyang Hospital of Wenzhou Medical University. In the training cohort, class balance was addressed using Random Over Sampling Examples (ROSE). Univariate and multivariate Cox regression analyses were performed to identify independent risk factors. These were then integrated into the predictive nomogram. In the validation cohort, the predictive performance of the nomogram was evaluated using the area under the curve (AUC) of the receiver operating characteristic curve, calibration curve, clinical utility decision curve analysis (DCA), and clinical impact curve (CIC).

Results

In the ROSE-balanced training cohort, univariate and multivariate Cox regression analysis identified that age, sex, Glasgow coma scale score, malignant cancer, sepsis, use of mechanical ventilation, use of vasoactive agents, white blood cells, potassium, and creatinine were independent predictors of in-hospital mortality in critically ill older adult patients, and were included in the nomogram. The nomogram showed good predictive performance in the ROSE-balanced training cohort (AUC [95% confidence interval]: 0.792 [0.783-0.801]) and validation cohort (AUC [95% confidence interval]: 0.780 [0.727-0.834]). The calibration curves were well-fitted. DCA and CIC demonstrated that the nomogram has good clinical application value.

Conclusion

This study developed a predictive model for early prediction of in-hospital mortality in critically ill older adult patients transferred from the ED to the ICU, which was validated by external data and has good predictive performance.

Development and external validation of dual online tools for prognostic assessment in elderly patients with high-grade glioma: a comprehensive study using SEER and Chinese cohorts

Background

Elderly individuals diagnosed with high-grade gliomas frequently experience unfavorable outcomes. We aimed to design two web-based instruments for prognosis to predict overall survival (OS) and cancer-specific survival (CSS), assisting clinical decision-making.

Methods

We scrutinized data from the SEER database on 5,245 elderly patients diagnosed with high-grade glioma between 2000-2020, segmenting them into training (3,672) and validation (1,573) subsets. An additional external validation cohort was obtained from our institution. Prognostic determinants were pinpointed using Cox regression analyses, which facilitated the construction of the nomogram. The nomogram's predictive precision for OS and CSS was gauged using calibration and ROC curves, the C-index, and decision curve analysis (DCA). Based on risk scores, patients were stratified into high or low-risk categories, and survival disparities were explored.

Results

Using multivariate Cox regression, we identified several prognostic factors for overall survival (OS) and cancer-specific survival (CSS) in elderly patients with high-grade gliomas, including age, tumor location, size, surgical technique, and therapies. Two digital nomograms were formulated anchored on these determinants. For OS, the C-index values in the training, internal, and external validation cohorts were 0.734, 0.729, and 0.701, respectively. We also derived AUC values for 3-, 6-, and 12-month periods. For CSS, the C-index values for the training and validation groups were 0.733 and 0.727, with analogous AUC metrics. The efficacy and clinical relevance of the nomograms were corroborated via ROC curves, calibration plots, and DCA for both cohorts.

Conclusion

Our investigation pinpointed pivotal risk factors in elderly glioma patients, leading to the development of an instrumental prognostic nomogram for OS and CSS. This instrument offers invaluable insights to optimize treatment strategies.

External Validation of Models for Predicting Disability in Community-Dwelling Older People in the Netherlands: A Comparative Study

Background

Advanced statistical modeling techniques may help predict health outcomes. However, it is not the case that these modeling techniques always outperform traditional techniques such as regression techniques. In this study, external validation was carried out for five modeling strategies for the prediction of the disability of community-dwelling older people in the Netherlands.

Methods

We analyzed data from five studies consisting of community-dwelling older people in the Netherlands. For the prediction of the total disability score as measured with the Groningen Activity Restriction Scale (GARS), we used fourteen predictors as measured with the Tilburg Frailty Indicator (TFI). Both the TFI and the GARS are self-report questionnaires. For the modeling, five statistical modeling techniques were evaluated: general linear model (GLM), support vector machine (SVM), neural net (NN), recursive partitioning (RP), and random forest (RF). Each model was developed on one of the five data sets and then applied to each of the four remaining data sets. We assessed the performance of the models with calibration characteristics, the correlation coefficient, and the root of the mean squared error.

Results

The models GLM, SVM, RP, and RF showed satisfactory performance characteristics when validated on the validation data sets. All models showed poor performance characteristics for the deviating data set both for development and validation due to the deviating baseline characteristics compared to those of the other data sets.

Conclusion

The performance of four models (GLM, SVM, RP, RF) on the development data sets was satisfactory. This was also the case for the validation data sets, except when these models were developed on the deviating data set. The NN models showed a much worse performance on the validation data sets than on the development data sets.

Development and validation of a predictive model associated with lymph node metastasis of gastric signet ring carcinoma patients

The risk factors for lymph node metastasis (LNM) in patients with gastric signet ring cell carcinoma (GSRC) have not been well-defined. This study was designed to prognosticate LNM in patients with GSRC by constructing and verifying a nomogram. A total of 2789 patients with GSRC from the Surveillance, Epidemiology, and End Results (SEER) database and Yijishan Hospital of Wannan Medical College (YJS) were retrospectively reviewed. A predictive model was established using logistic regression based on the SEER cohort. The performance of the model was evaluated using the concordance index (C-index) and decision curve analysis (DCA). In addition, its robustness was validated using the YJS cohort. Four independent predictors of LNM were identified in the SEER cohort. Next, a nomogram was constructed by incorporating these predictors. The C-index were 0.800 (95% confidence interval [CI] = 0.781-0.819) and 0.837 (95% CI = 0.784-0.890) in the training and external validation cohorts, respectively. The outcomes of DCA supported good clinical benefits. The proposed model for evaluating the LNM in patients with GSRC can help to avoid the misdiagnosis risk of N-stage, assist to screen the population suitable for neoadjuvant therapy and help clinicians to optimize clinical decisions.

Validating the SMART2 Score in a Racially Diverse High-Risk Nationwide Cohort of Patients Receiving Coronary Artery Bypass Grafting

Background We tested the potential of the Secondary Manifestations of Arterial Disease (SMART2) risk score for use in patients undergoing coronary artery bypass grafting. Methods and Results We conducted an external validation of the SMART2 score in a racially diverse high-risk national cohort (2010-2019) that underwent isolated coronary artery bypass grafting. We calculated the preoperative SMART2 score and modeled the 5-year major adverse cardiovascular event (cardiovascular mortality+myocardial infarction+stroke) incidence. We evaluated SMART2 score discrimination at 5 years using c-statistic and calibration with observed/expected ratio and calibration plots. We analyzed the potential clinical benefit using decision curves. We repeated these analyses in clinical subgroups, diabetes, chronic kidney disease, and polyvascular disease, and separately in White and Black patients. In 27 443 (mean age, 65 years; 10% Black individuals) US veterans undergoing coronary artery bypass grafting (2010-2019) nationwide, the 5-year major adverse cardiovascular event rate was 25%; 27% patients were in high predicted risk (>30% 5-year major adverse cardiovascular events). SMART2 score discrimination (c-statistic: 64) was comparable to the original study (c-statistic: 67) and was best in patients with chronic kidney disease (c-statistic: 66). However, it underpredicted major adverse cardiovascular event rates in the whole cohort (observed/expected ratio, 1.45) as well as in all studied subgroups. The SMART2 score performed better in White than Black patients. On decision curve analysis, the SMART2 score provides a net benefit over a wide range of risk thresholds. Conclusions The SMART2 model performs well in a racially diverse coronary artery bypass grafting cohort, with better predictive capabilities at the upper range of baseline risk, and can therefore be used to guide secondary preventive pharmacotherapy.

Applications of Machine Learning to Diagnosis of Parkinson's Disease

BACKGROUND

Accurate diagnosis of Parkinson's disease (PD) is challenging due to its diverse manifestations. Machine learning (ML) algorithms can improve diagnostic precision, but their generalizability across medical centers in China is underexplored.

OBJECTIVE

To assess the accuracy of an ML algorithm for PD diagnosis, trained and tested on data from different medical centers in China.

METHODS

A total of 1656 participants were included, with 1028 from Beijing (training set) and 628 from Fuzhou (external validation set). Models were trained using the least absolute shrinkage and selection operator-logistic regression (LASSO-LR), decision tree (DT), random forest (RF), eXtreme gradient boosting (XGboost), support vector machine (SVM), and k-nearest neighbor (KNN) techniques. Hyperparameters were optimized using five-fold cross-validation and grid search techniques. Model performance was evaluated using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, accuracy, sensitivity (recall), specificity, precision, and F1 score. Variable importance was assessed for all models.

RESULTS

SVM demonstrated the best differentiation between healthy controls (HCs) and PD patients (AUC: 0.928, 95% CI: 0.908-0.947; accuracy: 0.844, 95% CI: 0.814-0.871; sensitivity: 0.826, 95% CI: 0.786-0.866; specificity: 0.861, 95% CI: 0.820-0.898; precision: 0.849, 95% CI: 0.807-0.891; F1 score: 0.837, 95% CI: 0.803-0.868) in the validation set. Constipation, olfactory decline, and daytime somnolence significantly influenced predictability.

CONCLUSION

We identified multiple pivotal variables and SVM as a precise and clinician-friendly ML algorithm for prediction of PD in Chinese patients.

Risk-stratified approach by aMAP score for community population infected with hepatitis B and C to guide subsequent liver cancer screening practice: A cohort study with 10-year follow-up

The aim of this study was to determine whether the age-Male-ALBI-Platelet (aMAP) score is applicable in community settings and how to maximise its role in risk stratification. A total of thousand five hundred and three participants had an aMAP score calculated at baseline and were followed up for about 10 years to obtain information on liver cancer incidence and death. After assessing the ability of aMAP to predict liver cancer incidence and death in terms of differentiation and calibration, the optimal risk stratification threshold of the aMAP score was explored, based on absolute and relative risks. The aMAP score achieved higher area under curves (AUCs) (almost all above 0.8) within 10 years and exhibited a better calibration within 5 years. Regarding absolute risk, the risk of incidence of and death from liver cancer showed a rapid increase after an aMAP score of 55. The cumulative incidence (5-year: 8.3% vs. 1.3% and 10-year: 20.9% vs. 3.6%) and mortality (5-year: 6.7% vs. 1.1% and 10-year: 17.5% vs. 3.1%) of liver cancer in individuals with an aMAP score of ≥55 were significantly higher than in those with a score of <55 (Grey's test p  < .001). In terms of relative risk, the risk of death from liver cancer surpassed that from other causes after an aMAP score of ≥55 [HR = 1.38(1.02-1.87)]. Notably, the two types of death risk had opposite trends between the subpopulation with an aMAP score of ≥55 and < 55. To conclude, this study showed the value of the aMAP score in community settings and recommends using 55 as a new risk stratification threshold to guide subsequent liver cancer screening.

External validation of deep learning-based automated detection algorithm for chest radiograph: practical issues in outpatient clinic

BACKGROUND

There have been no reports on diagnostic performance of deep learning-based automated detection (DLAD) for thoracic diseases in real-world outpatient clinic.

PURPOSE

To validate DLAD for use at an outpatient clinic and analyze the interpretation time for chest radiographs.

MATERIAL AND METHODS

This is a retrospective single-center study. From 18 January 2021 to 18 February 2021, 205 chest radiographs with DLAD and paired chest CT from 205 individuals (107 men and 98 women; mean ± SD age: 63 ± 8 years) from an outpatient clinic were analyzed for external validation and observer performance. Two radiologists independently reviewed the chest radiographs by referring to the paired chest CT and made reference standards. Two pulmonologists and two thoracic radiologists participated in observer performance tests, and the total amount of time taken during the test was measured.

RESULTS

The performance of DLAD (area under the receiver operating characteristic curve [AUC] = 0.920) was significantly higher than that of pulmonologists (AUC = 0.756) and radiologists (AUC = 0.782) without assistance of DLAD. With help of DLAD, the AUCs were significantly higher for both groups (pulmonologists AUC = 0.853; radiologists AUC = 0.854). A greater than 50% decrease in mean interpretation time was observed in the pulmonologist group with assistance of DLAD compared to mean reading time without aid of DLAD (from 67 s per case to 30 s per case). No significant difference was observed in the radiologist group (from 61 s per case to 61 s per case).

CONCLUSION

DLAD demonstrated good performance in interpreting chest radiographs of patients at an outpatient clinic, and was especially helpful for pulmonologists in improving performance.

Development and External Validation of a Nomogram for Predicting the Effect of RTX on the Treatment of Membranous Nephropathy

Introduction

Rituximab (RTX) has been shown to be effective in inducing immunological remission in patients with membranous nephropathy (MN). Some patients required more than one course of RTX to achieve immunological remission. Identifying patients who need more courses of RTX to achieve immunological remission is beneficial for better physician-patient communication, the assessment of treatment course, and the evaluation of medical costs. This study aims to establish a practical model to predict the probability of immunological remission after receiving one cycle of RTX.

Methods

This study enrolled 106 patients from the First Affiliated Hospital of Zhengzhou University in the modeling group and 30 patients from Henan Provincial Hospital of Traditional Chinese Medicine in the external validation group. Patients in the modeling group were divided into responders or nonresponders according to whether they achieved immunological remission or not after following up for 6 months. A nomogram was established based on the results of logistic regression analysis. The predictive performance of the nomogram was evaluated by the area under the receiver operating characteristic curve (AUC), calibration curves, and decision curve analysis (DCAs).

Results

In the modeling group, 75 (70.8%) patients achieved immunological remission within 6 months after receiving one cycle of RTX. Significant differences were observed between nonresponders and responders. Risk factors used in nomogram included PLA2R antibody, hemoglobin, and gender. The AUC value of nomogram was 0.797 (95% CI 0.701-0.894, P<0.001). The calibration curves demonstrated acceptable agreement between the predicted outcomes by the nomogram and the actual values. DCA curves showed good positive net benefits in the predictive model. The external validation also demonstrated the reliability of the prediction nomogram.

Conclusion

A predictive nomogram including PLA2R antibody, hemoglobin, and gender may provide a basis to predict the doses of RTX needed in MN patients.

Cheese yield and nutrients recovery in the curd predicted by Fourier-transform spectra from individual sheep milk samples

The objectives of this study were to explore the use of Fourier-transform infrared (FTIR) spectroscopy on individual sheep milk samples for predicting cheese-making traits, and to test the effect of the farm variability on their prediction accuracy. For each of 121 ewes from 4 farms, a laboratory model cheese was produced, and 3 actual cheese yield traits (fresh cheese, cheese solids, and cheese water) and 4 milk nutrient recovery traits (fat, protein, total solids, and energy) in the curd were measured. Calibration equations were developed using a Bayesian approach with 2 different scenarios: (1) a random cross-validation (80% calibration; 20% validation set), and (2) a leave-one-out validation (3 farms used as calibration, and the remaining one as validation set) to assess the accuracy of prediction of samples from external farms, not included in calibration set. The best performance was obtained for predicting the yield and recovery of total solids, justifying for the practical application of the method at sheep population and dairy industry levels. Performances for the remaining traits were lower, but still useful for the monitoring of the milk processing in the case of fresh curd and recovery of energy. Insufficient accuracies were found for the recovery of protein and fat, highlighting the complex nature of the relationships among the milk nutrients and their recovery in the curd. The leave-one-out validation procedure, as expected, showed lower prediction accuracies, as a result of the characteristics of the farming systems, which were different between calibration and validation sets. In this regard, the inclusion of information related to the farm could help to improve the prediction accuracy of these traits. Overall, a large contribution to the prediction of the cheese-making traits came from the areas known as "water" and "fingerprint" regions. These findings suggest that, according to the traits studied, the inclusion of water regions for the development of the prediction equation models is fundamental to maintain a high prediction accuracy. However, further studies are necessary to better understand the role of specific absorbance peaks and their contribution to the prediction of cheese-making traits, to offer reliable tools applicable along the dairy ovine chain.

Machine learning for prediction of delirium in patients with extensive burns after surgery

AIMS

Machine learning-based identification of key variables and prediction of postoperative delirium in patients with extensive burns.

METHODS

Five hundred and eighteen patients with extensive burns who underwent surgery were included and randomly divided into a training set, a validation set, and a testing set. Multifactorial logistic regression analysis was used to screen for significant variables. Nine prediction models were constructed in the training and validation sets (80% of dataset). The testing set (20% of dataset) was used to further evaluate the model. The area under the receiver operating curve (AUROC) was used to compare model performance. SHapley Additive exPlanations (SHAP) was used to interpret the best one and to externally validate it in another large tertiary hospital.

RESULTS

Seven variables were used in the development of nine prediction models: physical restraint, diabetes, sex, preoperative hemoglobin, acute physiological and chronic health assessment, time in the Burn Intensive Care Unit and total body surface area. Random Forest (RF) outperformed the other eight models in terms of predictive performance (ROC:84.00%) When external validation was performed, RF performed well (accuracy: 77.12%, sensitivity: 67.74% and specificity: 80.46%).

CONCLUSION

The first machine learning-based delirium prediction model for patients with extensive burns was successfully developed and validated. High-risk patients for delirium can be effectively identified and targeted interventions can be made to reduce the incidence of delirium.

Internal and External Validation of the Generalizability of Machine Learning Algorithms in Predicting Non-home Discharge Disposition Following Primary Total Knee Joint Arthroplasty

BACKGROUND

Nonhome discharge disposition following primary total knee arthroplasty (TKA) is associated with a higher rate of complications and constitutes a socioeconomic burden on the health care system. While existing algorithms predicting nonhome discharge disposition varied in degrees of mathematical complexity and prediction power, their capacity to generalize predictions beyond the development dataset remains limited. Therefore, this study aimed to establish the machine learning model generalizability by performing internal and external validations using nation-scale and institutional cohorts, respectively.

METHODS

Four machine learning models were trained using the national cohort. Recursive feature elimination and hyper-parameter tuning were applied. Internal validation was achieved through five-fold cross-validation during model training. The trained models' performance was externally validated using the institutional cohort and assessed by discrimination, calibration, and clinical utility.

RESULTS

The national (424,354 patients) and institutional (10,196 patients) cohorts had non-home discharge rates of 19.4 and 36.4%, respectively. The areas under the receiver operating curve of the model predictions were 0.83 to 0.84 during internal validation and increased to 0.88 to 0.89 during external validation. Artificial neural network and histogram-based gradient boosting elicited the best performance with a mean area under the receiver operating curve of 0.89, calibration slope of 1.39, and Brier score of 0.14, which indicated that the two models were robust in distinguishing non-home discharge and well-calibrated with accurate predictions of the probabilities. The low inter-dataset similarity indicated reliable external validation. Length of stay, age, body mass index, and sex were the strongest predictors of discharge destination after primary TKA.

CONCLUSION

The machine learning models demonstrated excellent predictive performance during both internal and external validations, supporting their generalizability across different patient cohorts and potential applicability in the clinical workflow.

External validation of the first prognostic nomogram for older adults with cancer

Background

The geriatric oncology population tends to be complex because of multimorbidity, functional and cognitive decline, malnutrition and social frailty. Prognostic indices for predicting survival of elderly cancer patients to guide treatment remain scarce. A nomogram based on all domains of the geriatric assessment was previously developed at the National Cancer Centre Singapore (NCCS) to predict overall survival (OS) in elderly cancer patients. This nomogram comprised of six variables (age, eastern cooperative oncology group performance status, disease stage, geriatric depression scale (GDS), DETERMINE nutritional index and serum albumin).

Objectives

To externally validate the NCCS prognostic nomogram.

Design

This is a prospective cohort study.

Methods

The nomogram was developed based on a training cohort of 249 patients aged ⩾70 years who attended the NCCS outpatient geriatric oncology clinic between May 2007 and November 2010. External validation of the nomogram using the Royston and Altman approach was carried out on an independent testing cohort of 252 patients from the same clinic between July 2015 and June 2017. Model misspecification, discrimination and calibration were assessed.

Results

Median OS of the testing cohort was 3.1 years, which was significantly higher than the corresponding 1.0 year for the training cohort (log-rank p < 0.001). The nomogram achieved a high level of discrimination in the testing cohort (0.7112), comparable to the training cohort (0.7108). Predicted death probabilities were generally well calibrated with the observed death probabilities, as the joint test of calibration-in-the-large estimates at year 1, 2 and 3 from zeros and calibration slope from one was insignificant with p = 0.432. There were model misspecifications in GDS and serum albumin.

Conclusion

This study externally validated the prognostic nomogram in an independent cohort of geriatric oncology patients. This supports the use of this nomogram in clinical practice.

External Validation of Toulouse-Rangueil eGFR12 Prediction Model After Living Donor Nephrectomy

Decreased postdonation eGFR is associated with a higher risk of ESRD after living kidney donation, even when accounting for predonation characteristics. The Toulouse-Rangueil model (TRM) estimates 12 month postdonation eGFR (eGFR12) to inform counseling of candidates for living donation. The TRM was validated in several single-center European cohorts but has not been validated in US donors. We assessed the TRM in living kidney donors in the US using SRTR data 1/2000-6/2021. We compared the 2021 CKD-EPI equation eGFR12 observed estimates to the TRM eGFR12 predictions. Median (IQR) bias was -3.4 (-9.3, 3.4) mL/min/1.73 m2. Bias was higher for males vs. females (bias [IQR] -4.4 [-9.9, 1.8] vs. -2.9 [-8.8, 4.1]) and younger (31-40) vs. older donors (>50) (bias -4.9 [-10.6, 3.0] vs. -2.1 [-7.5, 4.0]). Bias was also larger for Black vs. White donors (bias (-6.7 [-12.1, -0.3], p < 0.001) vs. (-3.4 [-9.1, 3.1], p < 0.001)). Overall correlation was 0.71. In a sensitivity analysis using the 2009 CKD-EPI equation, results were generally consistent with exception to a higher overall bias (bias -4.2 [-9.8, 2.4]). The TRM overestimates postdonation renal function among US donors. Overestimation was greatest for those at higher risk for postdonation ESRD including male, Black, and younger donors. A new equation is needed to estimate postdonation renal function.

Development and external validation of a nomogram for predicting postoperative pneumonia in aneurysmal subarachnoid hemorrhage

Background

Postoperative pneumonia (POP) is a common complication after aneurysmal subarachnoid hemorrhage (aSAH) associated with increased mortality rates, prolonged hospitalization, and high medical costs. It is currently understood that identifying pneumonia early and implementing aggressive treatment can significantly improve patients' outcomes. The primary objective of this study was to explore risk factors and develop a logistic regression model that assesses the risks of POP.

Methods

An internal cohort of 613 inpatients with aSAH who underwent surgery at the Neurosurgical Department of First Affiliated Hospital of Wenzhou Medical University was retrospectively analyzed to develop a nomogram for predicting POP. We assessed the discriminative power, accuracy, and clinical validity of the predictions by using the area under the receiver operating characteristic curve (AUC), the calibration curve, and decision curve analysis (DCA). The final model was validated using an external validation set of 97 samples from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database.

Results

Among patients in our internal cohort, 15.66% (n = 96/613) of patients had POP. The least absolute shrinkage and selection operator (LASSO) regression analysis identified the Glasgow Coma Scale (GCS), mechanical ventilation time (MVT), albumin, C-reactive protein (CRP), smoking, and delayed cerebral ischemia (DCI) as potential predictors of POP. We then used multivariable logistic regression analysis to evaluate the effects of these predictors and create a final model. Eighty percentage of patients in the internal cohort were randomly assigned to the training set for model development, while the remaining 20% of patients were allocated to the internal validation set. The AUC values for the training, internal, and external validation sets were 0.914, 0.856, and 0.851, and the corresponding Brier scores were 0.084, 0.098, and 0.143, respectively.

Conclusion

We found that GCS, MVT, albumin, CRP, smoking, and DCI are independent predictors for the development of POP in patients with aSAH. Overall, our nomogram represents a reliable and convenient approach to predict POP in the patient population.

Population pharmacokinetics of Amisulpride in Chinese patients with schizophrenia with external validation: the impact of renal function

Introduction: Amisulpride is primarily eliminated via the kidneys. Given the clear influence of renal clearance on plasma concentration, we aimed to explicitly examine the impact of renal function on amisulpride pharmacokinetics (PK) via population PK modelling and Monte Carlo simulations. Method: Plasma concentrations from 921 patients (776 in development and 145 in validation) were utilized. Results: Amisulpride PK could be described by a one-compartment model with linear elimination where estimated glomerular filtration rate, eGFR, had a significant influence on clearance. All PK parameters (estimate, RSE%) were precisely estimated: apparent volume of distribution (645 L, 18%), apparent clearance (60.5 L/h, 2%), absorption rate constant (0.106 h-1, 12%) and coefficient of renal function on clearance (0.817, 10%). No other significant covariate was found. The predictive performance of the model was externally validated. Covariate analysis showed an inverse relationship between eGFR and exposure, where subjects with eGFR= 30 mL/min/1.73 m2 had more than 2-fold increase in AUC, trough and peak concentration. Simulation results further illustrated that, given a dose of 800 mg, plasma concentrations of all patients with renal impairment would exceed 640 ng/mL. Discussion: Our work demonstrated the importance of renal function in amisulpride dose adjustment and provided a quantitative framework to guide individualized dosing for Chinese patients with schizophrenia.

Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

External validation of the GAP model in Chinese patients with idiopathic pulmonary fibrosis

INTRODUCTION

The GAP model was widely used as a simple risk "screening" method for patients with idiopathic pulmonary fibrosis (IPF).

OBJECTIVES

We sought to validate the GAP model in Chinese patients with IPF to evaluate whether it can accurately predict the risk for mortality.

METHODS

A total of 212 patients with IPF diagnosed at China-Japan Friendship Hospital from 2015 to 2019 were enrolled. The latest follow-up ended in September 2022. Cumulative mortality of each GAP stage was calculated and compared based on Fine-Gray models for survival, and lung transplantation was treated as a competing risk. The performance of the model was evaluated in terms of both discrimination and calibration.

RESULTS

The cumulative mortality in patients with GAP stage III was significantly higher than that in those with GAP stage I or II (Gray's test p < 0.0001). The Harrell c-index for the GAP calculator was 0.736 (95% CI: 0.667-0.864). The discrimination for the GAP staging system were similar with that for the GAP calculator. The GAP model overestimated the mortality rate at 1- and 2-year in patients classified as GAP stage I (6.90% vs. 1.77% for 1-year, 14.20% vs. 6.78% for 2-year).

CONCLUSIONS

Our findings indicated that the GAP model overestimated the mortality rate in mild group.

Identification of reproducible radiomic features from on-board volumetric images: A multi-institutional phantom study

BACKGROUND

Radiomics analysis using on-board volumetric images has attracted research attention as a method for predicting prognosis during treatment; however, the lack of standardization is still one of the main concerns.

PURPOSE

This study investigated the factors that influence the reproducibility of radiomic features extracted from on-board volumetric images using an anthropomorphic radiomics phantom. Furthermore, a phantom experiment was conducted with different treatment machines from multiple institutions as external validation to identify reproducible radiomic features.

METHODS

The phantom was designed to be 35 × 20 × 20 cm with eight types of heterogeneous spheres (⌀ = 1, 2, and 3 cm). On-board volumetric images were acquired using 15 treatment machines from eight institutions. Of these, kilovoltage cone-beam computed tomography (kV-CBCT) image data acquired from four treatment machines at one institution were used as an internal evaluation dataset to explore the reproducibility of radiomic features. The remaining image data, including kV-CBCT, megavoltage-CBCT (MV-CBCT), and megavoltage computed tomography (MV-CT) provided by seven different institutions (11 treatment machines), were used as an external validation dataset. A total of 1,302 radiomic features, including 18 first-order, 75 texture, 465 (i.e., 93 × 5) Laplacian of Gaussian (LoG) filter-based, and 744 (i.e., 93 × 8) wavelet filter-based features, were extracted within the spheres. The intraclass correlation coefficient (ICC) was calculated to explore feature repeatability and reproducibility using an internal evaluation dataset. Subsequently, the coefficient of variation (COV) was calculated to validate the feature variability of external institutions. An absolute ICC exceeding 0.85 or COV under 5% was considered indicative of a highly reproducible feature.

RESULTS

For internal evaluation, ICC analysis showed that the median percentage of radiomic features with high repeatability was 95.2%. The ICC analysis indicated that the median percentages of highly reproducible features for inter-tube current, reconstruction algorithm, and treatment machine were decreased by 20.8%, 29.2%, and 33.3%, respectively. For external validation, the COV analysis showed that the median percentage of reproducible features was 31.5%. A total of 16 features, including nine LoG filter-based and seven wavelet filter-based features, were indicated as highly reproducible features. The gray-level run-length matrix (GLRLM) was classified as containing the most frequent features (N = 8), followed by the gray-level dependence matrix (N = 7) and gray-level co-occurrence matrix (N = 1) features.

CONCLUSIONS

We developed the standard phantom for radiomics analysis of kV-CBCT, MV-CBCT, and MV-CT images. With this phantom, we revealed that the differences in the treatment machine and image reconstruction algorithm reduce the reproducibility of radiomic features from on-board volumetric images. Specifically, the most reproducible features for external validation were LoG or wavelet filter-based GLRLM features. However, the acceptability of the identified features should be examined in advance at each institution before applying the findings to prognosis prediction.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

A machine learning-based model for predicting distant metastasis in patients with rectal cancer

Background

Distant metastasis from rectal cancer usually results in poorer survival and quality of life, so early identification of patients at high risk of distant metastasis from rectal cancer is essential.

Method

The study used eight machine-learning algorithms to construct a machine-learning model for the risk of distant metastasis from rectal cancer. We developed the models using 23867 patients with rectal cancer from the Surveillance, Epidemiology, and End Results (SEER) database between 2010 and 2017. Meanwhile, 1178 rectal cancer patients from Chinese hospitals were selected to validate the model performance and extrapolation. We tuned the hyperparameters by random search and tenfold cross-validation to construct the machine-learning models. We evaluated the models using the area under the receiver operating characteristic curves (AUC), the area under the precision-recall curve (AUPRC), decision curve analysis, calibration curves, and the precision and accuracy of the internal test set and external validation cohorts. In addition, Shapley's Additive explanations (SHAP) were used to interpret the machine-learning models. Finally, the best model was applied to develop a web calculator for predicting the risk of distant metastasis in rectal cancer.

Result

The study included 23,867 rectal cancer patients and 2,840 patients with distant metastasis. Multiple logistic regression analysis showed that age, differentiation grade, T-stage, N-stage, preoperative carcinoembryonic antigen (CEA), tumor deposits, perineural invasion, tumor size, radiation, and chemotherapy were-independent risk factors for distant metastasis in rectal cancer. The mean AUC value of the extreme gradient boosting (XGB) model in ten-fold cross-validation in the training set was 0.859. The XGB model performed best in the internal test set and external validation set. The XGB model in the internal test set had an AUC was 0.855, AUPRC was 0.510, accuracy was 0.900, and precision was 0.880. The metric AUC for the external validation set of the XGB model was 0.814, AUPRC was 0.609, accuracy was 0.800, and precision was 0.810. Finally, we constructed a web calculator using the XGB model for distant metastasis of rectal cancer.

Conclusion

The study developed and validated an XGB model based on clinicopathological information for predicting the risk of distant metastasis in patients with rectal cancer, which may help physicians make clinical decisions. rectal cancer, distant metastasis, web calculator, machine learning algorithm, external validation.

Clinical Validity of a Machine Learning Decision Support System for Early Detection of Hepatitis B Virus: A Binational External Validation Study

HepB LiveTest is a machine learning decision support system developed for the early detection of hepatitis B virus (HBV). However, there is a lack of evidence on its generalisability. In this study, we aimed to externally assess the clinical validity and portability of HepB LiveTest in predicting HBV infection among independent patient cohorts from Nigeria and Australia. The performance of HepB LiveTest was evaluated by constructing receiver operating characteristic curves and estimating the area under the curve. Delong's method was used to estimate the 95% confidence interval (CI) of the area under the receiver-operating characteristic curve (AUROC). Compared to the Australian cohort, patients in the derivation cohort of HepB LiveTest and the hospital-based Nigerian cohort were younger (mean age, 45.5 years vs. 38.8 years vs. 40.8 years, respectively; p < 0.001) and had a higher incidence of HBV infection (1.9% vs. 69.4% vs. 57.3%). In the hospital-based Nigerian cohort, HepB LiveTest performed optimally with an AUROC of 0.94 (95% CI, 0.91-0.97). The model provided tailored predictions that ensured most cases of HBV infection did not go undetected. However, its discriminatory measure dropped to 0.60 (95% CI, 0.56-0.64) in the Australian cohort. These findings indicate that HepB LiveTest exhibits adequate cross-site transportability and clinical validity in the hospital-based Nigerian patient cohort but shows limited performance in the Australian cohort. Whilst HepB LiveTest holds promise for reducing HBV prevalence in underserved populations, caution is warranted when implementing the model in older populations, particularly in regions with low incidence of HBV infection.

External validation of Fetal Medicine Foundation competing-risks model for midgestation prediction of small-for-gestational-age neonates in Spanish population

OBJECTIVE

To examine the external validity of the new Fetal Medicine Foundation (FMF) competing-risks model for prediction in midgestation of small-for-gestational-age (SGA) neonates.

METHODS

This was a single-center prospective cohort study of 25 484 women with a singleton pregnancy undergoing routine ultrasound examination at 19 + 0 to 23 + 6 weeks' gestation. The FMF competing-risks model for the prediction of SGA combining maternal factors and midgestation estimated fetal weight by ultrasound scan (EFW) and uterine artery pulsatility index (UtA-PI) was used to calculate risks for different cut-offs of birth-weight percentile and gestational age at delivery. The predictive performance was evaluated in terms of discrimination and calibration.

RESULTS

The validation cohort was significantly different in composition compared with the FMF cohort in which the model was developed. In the validation cohort, at a 10% false-positive rate (FPR), maternal factors, EFW and UtA-PI yielded detection rates of 69.6%, 38.7% and 31.7% for SGA < 10th percentile with delivery at < 32, < 37 and ≥ 37 weeks' gestation, respectively. The respective values for SGA < 3rd percentile were 75.7%, 48.2% and 38.1%. Detection rates in the validation cohort were similar to those reported in the FMF study for SGA with delivery at < 32 weeks but lower for SGA with delivery at < 37 and ≥ 37 weeks. Predictive performance in the validation cohort was similar to that reported in a subgroup of the FMF cohort consisting of nulliparous and Caucasian women. Detection rates in the validation cohort at a 15% FPR were 77.4%, 50.0% and 41.5% for SGA < 10th percentile with delivery at < 32, < 37 and ≥ 37 weeks, respectively, which were similar to the respective values reported in the FMF study at a 10% FPR. The model had satisfactory calibration.

CONCLUSION

The new competing-risks model for midgestation prediction of SGA developed by the FMF performs well in a large independent Spanish population. © 2023 International Society of Ultrasound in Obstetrics and Gynecology.

External validation of the ISARIC 4C Mortality Score to predict in-hospital mortality among patients with COVID-19 in a Canadian intensive care unit: a single-centre historical cohort study

PURPOSE

With uncertain prognostic utility of existing predictive scoring systems for COVID-19-related illness, the International Severe Acute Respiratory and Emerging Infection Consortium (ISARIC) 4C Mortality Score was developed by the International Severe Acute Respiratory and Emerging Infection Consortium as a COVID-19 mortality prediction tool. We sought to externally validate this score among critically ill patients admitted to an intensive care unit (ICU) with COVID-19 and compare its discrimination characteristics to that of the Acute Physiology and Chronic Health Evaluation (APACHE) II and Sequential Organ Failure Assessment (SOFA) scores.

METHODS

We enrolled all consecutive patients admitted with COVID-19-associated respiratory failure between 5 March 2020 and 5 March 2022 to our university-affiliated and intensivist-staffed ICU (Jewish General Hospital, Montreal, QC, Canada). After data abstraction, our primary outcome of in-hospital mortality was evaluated with an objective of determining the discriminative properties of the ISARIC 4C Mortality Score, using the area under the curve of a logistic regression model.

RESULTS

A total of 429 patients were included, 102 (23.8%) of whom died in hospital. The receiver operator curve of the ISARIC 4C Mortality Score had an area under the curve of 0.762 (95% confidence interval [CI], 0.717 to 0.811), whereas those of the SOFA and APACHE II scores were 0.705 (95% CI, 0.648 to 0.761) and 0.722 (95% CI, 0.667 to 0.777), respectively.

CONCLUSIONS

The ISARIC 4C Mortality Score is a tool that had a good predictive performance for in-hospital mortality in a cohort of patients with COVID-19 admitted to an ICU for respiratory failure. Our results suggest a good external validity of the 4C score when applied to a more severely ill population.

Predicting cancer relapse following lung stereotactic radiotherapy: an external validation study using real-world evidence

Purpose

For patients receiving lung stereotactic ablative radiotherapy (SABR), evidence suggests that high peritumor density predicts an increased risk of microscopic disease (MDE) and local-regional failure, but only if there is low or heterogenous incidental dose surrounding the tumor (GTV). A data-mining method (Cox-per-radius) has been developed to investigate this dose-density interaction. We apply the method to predict local relapse (LR) and regional failure (RF) in patients with non-small cell lung cancer.

Methods

199 patients treated in a routine setting were collated from a single institution for training, and 76 patients from an external institution for validation. Three density metrics (mean, 90th percentile, standard deviation (SD)) were studied in 1mm annuli between 0.5cm inside and 2cm outside the GTV boundary. Dose SD and fraction of volume receiving less than 30Gy were studied in annuli 0.5-2cm outside the GTV to describe incidental MDE dosage. Heat-maps were created that correlate with changes in LR and RF rates due to the interaction between dose heterogeneity and density at each distance combination. Regions of significant improvement were studied in Cox proportional hazards models, and explored with and without re-fitting in external data. Correlations between the dose component of the interaction and common dose metrics were reported.

Results

Local relapse occurred at a rate of 6.5% in the training cohort, and 18% in the validation cohort, which included larger and more centrally located tumors. High peritumor density in combination with high dose variability (0.5 - 1.6cm) predicts LR. No interactions predicted RF. The LR interaction improved the predictive ability compared to using clinical variables alone (optimism-adjusted C-index; 0.82 vs 0.76). Re-fitting model coefficients in external data confirmed the importance of this interaction (C-index; 0.86 vs 0.76). Dose variability in the 0.5-1.6 cm annular region strongly correlates with heterogeneity inside the target volume (SD; ρ = 0.53 training, ρ = 0.65 validation).

Conclusion

In these real-world cohorts, the combination of relatively high peritumor density and high dose variability predicts increase in LR, but not RF, following lung SABR. This external validation justifies potential use of the model to increase low-dose CTV margins for high-risk patients.

Development and External Validation of Partial Proportional Odds Risk Prediction Models for Cancer Stage at Diagnosis among Males and Females in Canada

Risk prediction models for cancer stage at diagnosis may identify individuals at higher risk of late-stage cancer diagnoses. Partial proportional odds risk prediction models for cancer stage at diagnosis for males and females were developed using data from Alberta's Tomorrow Project (ATP). Prediction models were validated on the British Columbia Generations Project (BCGP) cohort using discrimination and calibration measures. Among ATP males, older age at diagnosis was associated with an earlier stage at diagnosis, while full- or part-time employment, prostate-specific antigen testing, and former/current smoking were associated with a later stage at diagnosis. Among ATP females, mammogram and sigmoidoscopy or colonoscopy were associated with an earlier stage at diagnosis, while older age at diagnosis, number of pregnancies, and hysterectomy were associated with a later stage at diagnosis. On external validation, discrimination results were poor for both males and females while calibration results indicated that the models did not over- or under-fit to derivation data or over- or under-predict risk. Multiple factors associated with cancer stage at diagnosis were identified among ATP participants. While the prediction model calibration was acceptable, discrimination was poor when applied to BCGP data. Updating our models with additional predictors may help improve predictive performance.

Comparison of eight modern preoperative scoring systems for survival prediction in patients with extremity metastasis

BACKGROUND

Survival is an important factor to consider when clinicians make treatment decisions for patients with skeletal metastasis. Several preoperative scoring systems (PSSs) have been developed to aid in survival prediction. Although we previously validated the Skeletal Oncology Research Group Machine-learning Algorithm (SORG-MLA) in Taiwanese patients of Han Chinese descent, the performance of other existing PSSs remains largely unknown outside their respective development cohorts. We aim to determine which PSS performs best in this unique population and provide a direct comparison between these models.

METHODS

We retrospectively included 356 patients undergoing surgical treatment for extremity metastasis at a tertiary center in Taiwan to validate and compare eight PSSs. Discrimination (c-index), decision curve (DCA), calibration (ratio of observed:expected survivors), and overall performance (Brier score) analyses were conducted to evaluate these models' performance in our cohort.

RESULTS

The discriminatory ability of all PSSs declined in our Taiwanese cohort compared with their Western validations. SORG-MLA is the only PSS that still demonstrated excellent discrimination (c-indexes>0.8) in our patients. SORG-MLA also brought the most net benefit across a wide range of risk probabilities on DCA with its 3-month and 12-month survival predictions.

CONCLUSIONS

Clinicians should consider potential ethnogeographic variations of a PSS's performance when applying it onto their specific patient populations. Further international validation studies are needed to ensure that existing PSSs are generalizable and can be integrated into the shared treatment decision-making process. As cancer treatment keeps advancing, researchers developing a new prediction model or refining an existing one could potentially improve their algorithm's performance by using data gathered from more recent patients that are reflective of the current state of cancer care.