Percutaneous Left Atrial Appendage Occlusion in Comparison to Non-Vitamin K Antagonist Oral Anticoagulant Among Patients With Atrial Fibrillation

Background This study aimed to compare percutaneous left atrial appendage occlusion (LAAO) with non-vitamin K antagonist oral anticoagulants among patients with atrial fibrillation. Methods and Results Using a US administrative database, 562 850 patients with atrial fibrillation were identified, among whom 8397 were treated with LAAO and 554 453 were treated with non-vitamin K antagonist oral anticoagulants between March 13, 2015 and December 31, 2018. Propensity score overlap weighting was used to balance baseline characteristics. The primary outcome was a composite end point of ischemic stroke or systemic embolism, major bleeding, and all-cause mortality. The mean age was 76.4±7.6 years; 280 097 (49.8%) were female. Mean follow-up was 1.5±1.0 years. LAAO was associated with no significant difference in the risk of the primary composite end point (hazard ratio [HR], 0.93 [0.84-1.03]), or the secondary outcomes including ischemic stroke/systemic embolism (HR, 1.07 [0.81-1.41]), and intracranial bleeding (HR, 1.08 [0.72-1.61]). LAAO was associated with a higher risk of major bleeding (HR, 1.22 [1.05-1.42], P=0.01) and a lower risk of mortality (HR, 0.73 [0.64-0.84], P<0.001). The lower risk of mortality associated with LAAO was most pronounced in patients with a prior history of intracranial bleeding. Conclusions In comparison to non-vitamin K antagonist oral anticoagulants, LAAO was associated with no significant difference in the risk of the composite outcome and a lower risk of mortality, which suggests LAAO might be a reasonable option in select patients with atrial fibrillation. The observation of higher bleeding risk associated with LAAO highlights the need to optimize postprocedural antithrombotic regimens as well as systematic efforts to assess and address bleeding predispositions.

Development of Parkinson Disease and Its Relationship with Incidentally Discovered White Matter Disease and Covert Brain Infarction in a Real-World Cohort

OBJECTIVE

This study aimed to examine the relationship between covert cerebrovascular disease, comprised of covert brain infarction and white matter disease, discovered incidentally in routine care, and subsequent Parkinson disease.

METHODS

Patients were ≥50 years and received neuroimaging for non-stroke indications in the Kaiser Permanente Southern California system from 2009 to 2019. Natural language processing identified incidentally discovered covert brain infarction and white matter disease and classified white matter disease severity. The Parkinson disease outcome was defined as 2 ICD diagnosis codes.

RESULTS

230,062 patients were included (median follow-up 3.72 years). A total of 1,941 Parkinson disease cases were identified (median time-to-event 2.35 years). Natural language processing identified covert cerebrovascular disease in 70,592 (30.7%) patients, 10,622 (4.6%) with covert brain infarction and 65,814 (28.6%) with white matter disease. After adjustment for known risk factors, white matter disease was associated with Parkinson disease (hazard ratio 1.67 [95%CI, 1.44, 1.93] for patients <70 years and 1.33 [1.18, 1.50] for those ≥70 years). Greater severity of white matter disease was associated with increased incidence of Parkinson disease(/1,000 person-years), from 1.52 (1.43, 1.61) in patients without white matter disease to 4.90 (3.86, 6.13) in those with severe disease. Findings were robust when more specific definitions of Parkinson disease were used. Covert brain infarction was not associated with Parkinson disease (adjusted hazard ratio = 1.05 [0.88, 1.24]).

INTERPRETATION

Incidentally discovered white matter disease was associated with subsequent Parkinson disease, an association strengthened with younger age and increased white matter disease severity. Incidentally discovered covert brain infarction did not appear to be associated with subsequent Parkinson disease. ANN NEUROL 2022;92:620-630.

Predicting Chronic Opioid Use Among Patients With Osteoarthritis Using Electronic Health Record Data

OBJECTIVE

To estimate the risk of a patient with osteoarthritis (OA) developing chronic opioid use (COU) within 1 year of a new opioid prescription by using electronic health record (EHR) data and predictive models.

METHODS

We used EHR data from 13 health care organizations to identify patients with OA with an opioid prescription between March 1, 2017 and February 28, 2019 and no record of opioid use in the prior 6 months. We evaluated 4 machine learning models to estimate patients' risk of COU (≥3 prescriptions ≥84 days, maximum gap ≤60 days). We also estimated the transportability of models to organizations outside the training set.

RESULTS

The cohort consisted of 33,894 patients with OA, of whom 2,925 (8.6%) developed COU within 1 year. All models demonstrated good discrimination, with the best-performing model (random forest) achieving an area under the receiver operating characteristic curve (AUC) of 0.728 (95% CI 0.711-0.745), but the simplest regression model performed nearly as well (AUC 0.717 [95% CI 0.699-0.734]). Predicted risk deciles spanned a range of 2% risk for the 10th percentile to 18% risk for the 90th percentile for well-calibrated models. Models showed highly variable discrimination across organizations (AUC 0.571-0.842).

CONCLUSIONS

We found that EHR-based predictive models could estimate the risk of future COU among patients with OA to help inform care decisions. Black-box methods did not have significant advantages over more interpretable models. Care should be taken when extending all models into organizations not included in model training because of a high variability in performance across held-out organizations.

Stratifying Future Stroke Risk with Incidentally Discovered White Matter Disease Severity and Covert Brain Infarct Site

BACKGROUND

Covert cerebrovascular disease (CCD) includes white matter disease (WMD) and covert brain infarction (CBI). Incidentally discovered CCD is associated with increased risk of subsequent symptomatic stroke. However, it is unknown whether the severity of WMD or the location of CBI predicts risk.

OBJECTIVES

The aim of this study was to examine the association of incidentally discovered WMD severity and CBI location with risk of subsequent symptomatic stroke.

METHOD

This retrospective cohort study includes patients aged ≥50 years old in the Kaiser Permanente Southern California health system who received neuroimaging for a nonstroke indication between 2009 and 2019. Incidental CBI and WMD were identified via natural language processing of the neuroimage report, and WMD severity was classified into grades.

RESULTS

A total of 261,960 patients received neuroimaging; 78,555 patients (30.0%) were identified to have incidental WMD and 12,857 patients (4.9%) to have incidental CBI. Increasing WMD severity is associated with an increased incidence rate of future stroke. However, the stroke incidence rate in CT-identified WMD is higher at each level of severity compared to rates in MRI-identified WMD. Patients with mild WMD via CT have a stroke incidence rate of 24.9 per 1,000 person-years, similar to that of patients with severe WMD via MRI. Among incidentally discovered CBI patients with a determined CBI location, 97.9% are subcortical rather than cortical infarcts. CBI confers a similar risk of future stroke, whether cortical or subcortical or whether MRI- or CT-detected.

CONCLUSIONS

Increasing severity of incidental WMD is associated with an increased risk of future symptomatic stroke, dependent on the imaging modality. Subcortical and cortical CBI conferred similar risks.

Risk Factors for Silent Brain Infarcts and White Matter Disease in a Real-World Cohort Identified by Natural Language Processing

OBJECTIVE

To assess the frequency of silent brain infarcts (SBIs) and white matter disease (WMD) and associations with stroke risk factors (RFs) in a real-world population.

PATIENTS AND METHODS

This was an observational study of patients 50 years or older in the Kaiser Permanente Southern California health system from January 1, 2009, through June 30, 2019, with head computed tomography or magnetic resonance imaging for nonstroke indications and no history of stroke, transient ischemic attack, or dementia. A natural language processing (NLP) algorithm was applied to the electronic health record to identify individuals with reported SBIs or WMD. Multivariable Poisson regression estimated risk ratios of demographic characteristics, RFs, and scan modality on the presence of SBIs or WMD.

RESULTS

Among 262,875 individuals, the NLP identified 13,154 (5.0%) with SBIs and 78,330 (29.8%) with WMD. Stroke RFs were highly prevalent. Advanced age was strongly associated with increased risk of SBIs (adjusted relative risks [aRRs], 1.90, 3.23, and 4.72 for those aged in their 60s, 70s, and ≥80s compared with those in their 50s) and increased risk of WMD (aRRs, 1.79, 3.02, and 4.53 for those aged in their 60s, 70s, and ≥80s compared with those in their 50s). Magnetic resonance imaging was associated with a reduced risk of SBIs (aRR, 0.87; 95% CI, 0.83 to 0.91) and an increased risk of WMD (aRR, 2.86; 95% CI, 2.83 to 2.90). Stroke RFs had modest associations with increased risk of SBIs or WMD.

CONCLUSION

An NLP algorithm can identify a large cohort of patients with incidentally discovered SBIs and WMD. Advanced age is strongly associated with incidentally discovered SBIs and WMD.

Does poor methodological quality of prediction modeling studies translate to poor model performance? An illustration in traumatic brain injury

BACKGROUND

Prediction modeling studies often have methodological limitations, which may compromise model performance in new patients and settings. We aimed to examine the relation between methodological quality of model development studies and their performance at external validation.

METHODS

We systematically searched for externally validated multivariable prediction models that predict functional outcome following moderate or severe traumatic brain injury. Risk of bias and applicability of development studies was assessed with the Prediction model Risk Of Bias Assessment Tool (PROBAST). Each model was rated for its presentation with sufficient detail to be used in practice. Model performance was described in terms of discrimination (AUC), and calibration. Delta AUC (dAUC) was calculated to quantify the percentage change in discrimination between development and validation for all models. Generalized estimation equations (GEE) were used to examine the relation between methodological quality and dAUC while controlling for clustering.

RESULTS

We included 54 publications, presenting ten development studies of 18 prediction models, and 52 external validation studies, including 245 unique validations. Two development studies (four models) were found to have low risk of bias (RoB). The other eight publications (14 models) showed high or unclear RoB. The median dAUC was positive in low RoB models (dAUC 8%, [IQR - 4% to 21%]) and negative in high RoB models (dAUC - 18%, [IQR - 43% to 2%]). The GEE showed a larger average negative change in discrimination for high RoB models (- 32% (95% CI: - 48 to - 15) and unclear RoB models (- 13% (95% CI: - 16 to - 10)) compared to that seen in low RoB models.

CONCLUSION

Lower methodological quality at model development associates with poorer model performance at external validation. Our findings emphasize the importance of adherence to methodological principles and reporting guidelines in prediction modeling studies.

External Validation of the FREEDOM Score for Individualized Decision Making Between CABG and PCI

BACKGROUND

Although randomized trials have established that coronary artery bypass grafting (CABG) is, on average, the most effective revascularization strategy compared with percutaneous coronary intervention (PCI) in patients with diabetes and multivessel disease (MVD), individual patients differ in many characteristics that can affect the benefits and harms of treatment. The FREEDOM (Future Revascularization Evaluation in Patients with Diabetes Mellitus) score was developed to predict different outcomes with CABG vs PCI on the basis of 8 patient characteristics and the smoking-treatment interaction.

OBJECTIVES

This study aimed to assess the ability of the 5-year major adverse cardiovascular event (MACE) model to predict treatment benefit of CABG vs PCI in the SYNTAX (Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery) and BEST (Bypass Surgery and Everolimus-Eluting Stent Implantation in the Treatment of Patients with Multivessel Coronary Artery Disease) trials.

METHODS

This study identified 702 patients with diabetes and MVD to mirror the FREEDOM participants. Discrimination was assessed by C-index, and calibration was assessed by calibration plots in the PCI and CABG arms, respectively. The ability of the FREEDOM score to predict treatment benefit of CABG vs PCI was assessed.

RESULTS

Overall, CABG was associated with a lower rate of 5-year MACE compared with PCI (12.4% vs 20.3%; log-rank P = 0.021) irrespective of a history of smoking (P_interaction = 0.975). Both discrimination and calibration were helpful in the PCI arm (C-index: 0.69; slope: 0.96, intercept: -0.24), but moderate in the CABG arm (C-index: 0.61; slope: 0.61; intercept: -0.53). The FREEDOM score showed some heterogeneity of treatment benefit.

CONCLUSIONS

The FREEDOM score could identify some heterogeneity of treatment benefit of CABG vs PCI for 5-year MACE. Until further prospective validations are performed, these results should be taken into consideration when using the FREEDOM score in patients with diabetes and MVD. (Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery [SYNTAX]; NCT00114972) (Bypass Surgery and Everolimus-Eluting Stent Implantation in the Treatment of Patients with Multivessel Coronary Artery Disease [BEST]; NCT00997828) (Future Revascularization Evaluation in Patients with Diabetes Mellitus [FREEDOM]; NCT00086450).

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Background:

While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood.

Methods:

We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed.

Results:

Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation.

Conclusions:

There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

An Electronic Health Record-Compatible Model to Predict Personalized Treatment Effects From the Diabetes Prevention Program: A Cross-Evidence Synthesis Approach Using Clinical Trial and Real-World Data

OBJECTIVE

To develop an electronic health record (EHR)-based risk tool that provides point-of-care estimates of diabetes risk to support targeting interventions to patients most likely to benefit.

PATIENTS AND METHODS

A risk prediction model was developed and validated in a large observational database of patients with an index visit date between January 1, 2012, and December 31, 2016, with treatment effect estimates from risk-based reanalysis of clinical trial data. The risk model development cohort included 1.1 million patients with prediabetes from the OptumLabs Data Warehouse (OLDW); the validation cohort included a distinct sample of 1.1 million patients in OLDW. The randomly assigned clinical trial cohort included 3081 people from the Diabetes Prevention Program (DPP) study.

RESULTS

Eleven variables reliably obtainable from the EHR were used to predict diabetes risk. This model validated well in the OLDW (C statistic = 0.76; observed 3-year diabetes rate was 1.8% (95% confidence interval [CI], 1.7 to 1.9) in the lowest-risk quarter and 19.6% (19.4 to 19.8) in the highest-risk quarter). In the DPP, the hazard ratio (HR) for lifestyle modification was constant across all levels of risk (HR, 0.43; 95% CI, 0.35 to 0.53), whereas the HR for metformin was highly risk dependent (HR, 1.1; 95% CI, 0.61 to 2.0 in the lowest-risk quarter vs HR, 0.45; 95% CI, 0.35 to 0.59 in the highest-risk quarter). Fifty-three percent of the benefits of population-wide dissemination of the DPP lifestyle modification and 73% of the benefits of population-wide metformin therapy can be obtained by targeting the highest-risk quarter of patients.

CONCLUSION

The Tufts-Predictive Analytics and Comparative Effectiveness DPP Risk model is an EHR-compatible tool that might support targeted diabetes prevention to more efficiently realize the benefits of the DPP interventions.

Abstract TMP106: Pooled Analysis Of Five Randomized Trials Comparing Device Closure Of Patent Foramen Ovale After Stroke To Medical Treatment: Impact Of Residual Shunt On Stroke Outcomes

Introduction: Patent foramen ovale (PFO) closure is effective for secondary stroke prevention in well selected patients. After device implantation some patients have residual shunting. There was no impact of incomplete closure on recurrent stroke risk in any of the trials but outcome rates are low with little power to detect an effect.

Hypothesis: Patients with residual shunts after PFO closure have more recurrent strokes than those with complete elimination of shunting.

Methods: We pooled individual patient data from 5 of 6 randomized clinical trials comparing medical therapy to PFO closure + medical therapy. Residual shunt data were not available for 1 trial. This analysis only includes PFO closure subjects. Hazard ratios (HR) and 95% confidence intervals (CI) were estimated with Cox proportional hazards regression, adjusted for age, sex, coronary disease, diabetes, hypertension, hyperlipidemia, prior stroke/TIA, smoking, index event (stroke vs TIA), hypermobile septum, shunt size (large vs small) and infarct location (superficial vs deep). Multiple imputation of missing values for adjusted variables was used.

Results: Among 1475 patients treated with PFO closure and with non-missing residual shunt data, 149 (10%) had incomplete closure. The median time from procedure to shunt measurement was 204 days (25 th -75 th percentile, 181-722). In the complete closure group the primary endpoint of recurrent ischemic stroke occurred in 30 patients (2.3%; 95% CI 1.5-3.2%) compared to 4 (2.7%; 95% CI 0.7-6.7%) with incomplete closure. Unadjusted and adjusted HRs comparing incomplete vs complete closure were 1.37 (0.48, 3.92) and 1.69 (0.37, 4.95) respectively. The secondary endpoint of recurrent ischemic stroke, TIA, or vascular death occurred in 66 (5.0%; 95% CI 3.9-6.3%) patients with complete closure compared to 9 (6.0%; 95% CI 2.8-11.2%) without. Unadjusted and adjusted HRs were 1.26 (0.63, 2.54) and 1.39 (0.68, 2.83) respectively.

Conclusions: Our pooled analysis of individual subjects after PFO closure from 5 of 6 relevant randomized clinical trials, showed no association between residual shunt and recurrent stroke events. This analysis was limited by low rates of residual shunt and recurrent events even in this large, combined dataset.

Heterogeneity of Treatment Effects in an Analysis of Pooled Individual Patient Data From Randomized Trials of Device Closure of Patent Foramen Ovale After Stroke

Importance

Patent foramen ovale (PFO)-associated strokes comprise approximately 10% of ischemic strokes in adults aged 18 to 60 years. While device closure decreases stroke recurrence risk overall, the best treatment for any individual is often unclear.

Objective

To evaluate heterogeneity of treatment effect of PFO closure on stroke recurrence based on previously developed scoring systems.

Design, Setting, and Participants

Investigators for the Systematic, Collaborative, PFO Closure Evaluation (SCOPE) Consortium pooled individual patient data from all 6 randomized clinical trials that compared PFO closure plus medical therapy vs medical therapy alone in patients with PFO-associated stroke, and included a total of 3740 participants. The trials were conducted worldwide from 2000 to 2017.

Exposures

PFO closure plus medical therapy vs medical therapy alone. Subgroup analyses used the Risk of Paradoxical Embolism (RoPE) Score (a 10-point scoring system in which higher scores reflect younger age and the absence of vascular risk factors) and the PFO-Associated Stroke Causal Likelihood (PASCAL) Classification System, which combines the RoPE Score with high-risk PFO features (either an atrial septal aneurysm or a large-sized shunt) to classify patients into 3 categories of causal relatedness: unlikely, possible, and probable.

Main Outcomes and Measures

Ischemic stroke.

Results

Over a median follow-up of 57 months (IQR, 24-64), 121 outcomes occurred in 3740 patients. The annualized incidence of stroke with medical therapy was 1.09% (95% CI, 0.88%-1.36%) and with device closure was 0.47% (95% CI, 0.35%-0.65%) (adjusted hazard ratio [HR], 0.41 [95% CI, 0.28-0.60]). The subgroup analyses showed statistically significant interaction effects. Patients with low vs high RoPE Score had HRs of 0.61 (95% CI, 0.37-1.00) and 0.21 (95% CI, 0.11-0.42), respectively (P for interaction = .02). Patients classified as unlikely, possible, and probable using the PASCAL Classification System had HRs of 1.14 (95% CI, 0.53-2.46), 0.38 (95% CI, 0.22-0.65), and 0.10 (95% CI, 0.03-0.35), respectively (P for interaction = .003). The 2-year absolute risk reduction was -0.7% (95% CI, -4.0% to 2.6%), 2.1% (95% CI, 0.6%-3.6%), and 2.1% (95% CI, 0.9%-3.4%) in the unlikely, possible, and probable PASCAL categories, respectively. Device-associated adverse events were generally higher among patients classified as unlikely; the absolute risk increases in atrial fibrillation beyond day 45 after randomization with a device were 4.41% (95% CI, 1.02% to 7.80%), 1.53% (95% CI, 0.33% to 2.72%), and 0.65% (95% CI, -0.41% to 1.71%) in the unlikely, possible, and probable PASCAL categories, respectively.

Conclusions and Relevance

Among patients aged 18 to 60 years with PFO-associated stroke, risk reduction for recurrent stroke with device closure varied across groups classified by their probabilities that the stroke was causally related to the PFO. Application of this classification system has the potential to guide individualized decision-making.

External Validation of the SYNTAX Score II 2020

BACKGROUND

The SYNTAX score II 2020 (SSII-2020) was derived from cross correlation and externally validated in randomized trials to predict death and major adverse cardiac and cerebrovascular events (MACE) following percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG) in patients with 3-vessel disease (3VD) and/or left main coronary artery disease (LMCAD).

OBJECTIVES

The authors aimed to investigate the SSII-2020's value in identifying the safest modality of revascularization in a non-randomized setting.

METHODS

Five-year mortality and MACE were assessed in 7,362 patients with 3VD and/or LMCAD enrolled in a Japanese PCI/CABG registry. The discriminative abilities of the SSII-2020 were assessed using Harrell's C statistic. Agreement between observed and predicted event rates following PCI or CABG and treatment benefit (absolute risk difference [ARD]) for these outcomes were assessed by calibration plots.

RESULTS

The SSII-2020 for 5-year mortality well predicted the prognosis after PCI and CABG (C-index = 0.72, intercept = -0.11, slope = 0.92). When patients were grouped according to the predicted 5-year mortality ARD, <4.5% (equipoise of PCI and CABG) and ≥4.5% (CABG better), the observed mortality rates after PCI and CABG were not significantly different in patients with lower predicted ARD (observed ARD: 2.1% [95% CI: -0.4% to 4.4%]), and the significant difference in survival in favor of CABG was observed in patients with higher predicted ARD (observed ARD: 9.7% [95% CI: 6.1%-13.3%]). For MACE, the SSII-2020 could not recommend a specific treatment with sufficient accuracy.

CONCLUSIONS

The SSII-2020 for predicting 5-year death has the potential to support decision making on revascularization in patients with 3VD and/or LMCAD.

COVID outcome prediction in the emergency department (COPE): using retrospective Dutch hospital data to develop simple and valid models for predicting mortality and need for intensive care unit admission in patients who present at the emergency department with suspected COVID-19

OBJECTIVES

Develop simple and valid models for predicting mortality and need for intensive care unit (ICU) admission in patients who present at the emergency department (ED) with suspected COVID-19.

DESIGN

Retrospective.

SETTING

Secondary care in four large Dutch hospitals.

PARTICIPANTS

Patients who presented at the ED and were admitted to hospital with suspected COVID-19. We used 5831 first-wave patients who presented between March and August 2020 for model development and 3252 second-wave patients who presented between September and December 2020 for model validation.

OUTCOME MEASURES

We developed separate logistic regression models for in-hospital death and for need for ICU admission, both within 28 days after hospital admission. Based on prior literature, we considered quickly and objectively obtainable patient characteristics, vital parameters and blood test values as predictors. We assessed model performance by the area under the receiver operating characteristic curve (AUC) and by calibration plots.

RESULTS

Of 5831 first-wave patients, 629 (10.8%) died within 28 days after admission. ICU admission was fully recorded for 2633 first-wave patients in 2 hospitals, with 214 (8.1%) ICU admissions within 28 days. A simple model-COVID outcome prediction in the emergency department (COPE)-with age, respiratory rate, C reactive protein, lactate dehydrogenase, albumin and urea captured most of the ability to predict death. COPE was well calibrated and showed good discrimination for mortality in second-wave patients (AUC in four hospitals: 0.82 (95% CI 0.78 to 0.86); 0.82 (95% CI 0.74 to 0.90); 0.79 (95% CI 0.70 to 0.88); 0.83 (95% CI 0.79 to 0.86)). COPE was also able to identify patients at high risk of needing ICU admission in second-wave patients (AUC in two hospitals: 0.84 (95% CI 0.78 to 0.90); 0.81 (95% CI 0.66 to 0.95)).

CONCLUSIONS

COPE is a simple tool that is well able to predict mortality and need for ICU admission in patients who present to the ED with suspected COVID-19 and may help patients and doctors in decision making.

Association of Silent Cerebrovascular Disease Identified Using Natural Language Processing and Future Ischemic Stroke

BACKGROUND AND OBJECTIVE

Silent cerebrovascular disease (SCD), comprised of silent brain infarction (SBI) and white matter disease (WMD), is commonly found incidentally on neuroimaging scans obtained in routine clinical care. However, their prognostic significance is not known. We aimed to estimate the incidence of, and risk increase in, future stroke in patients with incidentally-discovered SCD. METHODS: Patients in Kaiser Permanente Southern California (KPSC) health system aged ≥ 50, without prior ischemic stroke, transient ischemic attack, or dementia/Alzheimer's disease receiving a head CT or MRI between 2009-2019 were included. SBI and WMD were identified by natural language processing (NLP) from the neuroimage report.

RESULTS

Among 262,875 individuals receiving neuroimaging, NLP identified 13,154 (5.0%) with SBI and 78,330 (29.8%) with WMD. The incidence of future stroke was 32.5 (95% CI 31.1, 33·9) per 1,000 patient-years for patients with SBI; 1.·3 (95% CI 18.9, 19.8) for patients with WMD and 6.8 (95% CI 6.7, 7.0) for patients without SCD. The crude HR associated with SBI was 3.40 (95% CI 3.25 to 3.56); and for WMD was 2.63 (95% CI 2.54 to 2·71). With MRI-discovered SBI, the adjusted HR was 2.95 (95% CI 2.53 to 3.44) for those < age 65 and 2.15 (95% CI 1.91 to 2.41) for those ≥ age 65. With CT scan, the adjusted HR was 2.48 (95% CI 2.19 to 2.81) for those < age 65 and 1.81 (95% CI 1.71 to 1.91) for those >= age 65. The adjusted HR associated with a finding of WMD was 1.76 (95% CI 1.69 to 1.82) and was not modified by age or imaging modality.

DISCUSSION

Incidentally-discovered SBI and WMD are common and associated with increased risk of subsequent symptomatic stroke representing an important opportunity for stroke prevention.

External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

BACKGROUND

There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated.

METHODS

A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.

RESULTS

We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001).

CONCLUSIONS

Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

OBJECTIVE

To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.

STUDY DESIGN AND SETTING

We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147).

RESULTS

PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001).

CONCLUSION

High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Targeting of the diabetes prevention program leads to substantial benefits when capacity is constrained

OBJECTIVE

Approximately 84 million people in the USA have pre-diabetes, but only a fraction of them receive proven effective therapies to prevent type 2 diabetes. We estimated the value of prioritizing individuals at highest risk of progression to diabetes for treatment, compared to non-targeted treatment of individuals meeting inclusion criteria for the Diabetes Prevention Program (DPP).

METHODS

Using microsimulation to project outcomes in the DPP trial population, we compared two interventions to usual care: (1) lifestyle modification and (2) metformin administration. For each intervention, we compared targeted and non-targeted strategies, assuming either limited or unlimited program capacity. We modeled the individualized risk of developing diabetes and projected diabetic outcomes to yield lifetime costs and quality-adjusted life expectancy, from which we estimated net monetary benefits (NMB) for both lifestyle and metformin versus usual care.

RESULTS

Compared to usual care, lifestyle modification conferred positive benefits and reduced lifetime costs for all eligible individuals. Metformin's NMB was negative for the lowest population risk quintile. By avoiding use when costs outweighed benefits, targeted administration of metformin conferred a benefit of $500 per person. If only 20% of the population could receive treatment, when prioritizing individuals based on diabetes risk, rather than treating a 20% random sample, the difference in NMB ranged from $14,000 to $20,000 per person.

CONCLUSIONS

Targeting active diabetes prevention to patients at highest risk could improve health outcomes and reduce costs compared to providing the same intervention to a similar number of patients with pre-diabetes without targeted selection.

Agreement between neuroimages and reports for natural language processing-based detection of silent brain infarcts and white matter disease

BACKGROUND

There are numerous barriers to identifying patients with silent brain infarcts (SBIs) and white matter disease (WMD) in routine clinical care. A natural language processing (NLP) algorithm may identify patients from neuroimaging reports, but it is unclear if these reports contain reliable information on these findings.

METHODS

Four radiology residents reviewed 1000 neuroimaging reports (RI) of patients age > 50 years without clinical histories of stroke, TIA, or dementia for the presence, acuity, and location of SBIs, and the presence and severity of WMD. Four neuroradiologists directly reviewed a subsample of 182 images (DR). An NLP algorithm was developed to identify findings in reports. We assessed interrater reliability for DR and RI, and agreement between these two and with NLP.

RESULTS

For DR, interrater reliability was moderate for the presence of SBIs (k = 0.58, 95 % CI 0.46-0.69) and WMD (k = 0.49, 95 % CI 0.35-0.63), and moderate to substantial for characteristics of SBI and WMD. Agreement between DR and RI was substantial for the presence of SBIs and WMD, and fair to substantial for characteristics of SBIs and WMD. Agreement between NLP and DR was substantial for the presence of SBIs (k = 0.64, 95 % CI 0.53-0.76) and moderate (k = 0.52, 95 % CI 0.39-0.65) for the presence of WMD.

CONCLUSIONS

Neuroimaging reports in routine care capture the presence of SBIs and WMD. An NLP can identify these findings (comparable to direct imaging review) and can likely be used for cohort identification.

The cumulative incidence of dysphagia and dysphagia-free survival in persons diagnosed with amyotrophic lateral sclerosis

INTRODUCTION/AIMS

Dysphagia worsens mortality and quality of life for persons diagnosed with amyotrophic lateral sclerosis (ALS), yet our understanding of its incidence and timing remains limited. In this study we sought to estimate dysphagia incidence and dysphagia-free survival over time.

METHODS

Using data from the Pooled Resource Open-Access ALS Clinical Trials Database, we compared characteristics of persons with and without dysphagia upon study entry. To account for competing mortality risk, we used Kaplan-Meier curves to estimate the cumulative incidence of dysphagia and the median number of days until the development of dysphagia or death in those without dysphagia at study entry.

RESULTS

Patients with dysphagia upon study entry were more likely to have bulbar onset and had faster rates of functional decline and shorter diagnostic delays. The cumulative incidence of new-onset dysphagia was 44% at 1 year and 64% at 2 years after trial enrollment for those with spinal onset, and 85% and 92% for those with bulbar onset. The median duration of dysphagia-free survival after trial enrollment was 11.5 months for those with spinal onset and 3.2 months for those with bulbar onset.

DISCUSSION

Our findings underscore the high risk for dysphagia development and support the need for early dysphagia referral and evaluation to minimize the risk of serious dysphagia-related complications.

Abstract P595: Incidentally-Discovered Asymptomatic Ischemic Lesions Identified Using Natural Language Processing Are Strong Risk Factors for Dementia

Background: White matter disease (WMD) and silent brain infarction (SBI) are known to be risk markers for dementia. Nevertheless, the effects of incidentally-discovered WMD and SBI on routinely-obtained neuroimages is unknown.

Methods: In the retrospective cohort study, Kaiser Permanente-Southern California (KPSC) health plan enrollees aged ≥ 50 years old with a brain cross-sectional image (CT or MRI) between 2009-2019 and without a prior history of ischemic stroke, transient ischemic attack, or dementia were identified. Natural language processing (NLP) was used to identify patients with SBI and WMD on the first qualifying neuroimaging report. Cox proportional hazards regression model was applied to estimate the effects of WMD and SBI on dementia, controlling for major cardiovascular risk factors.

Results: Among 262,875 individuals receiving brain neuroimaging, 13,154 (5.0%) and 78,330 (29.8%) had SBI and WMD, respectively. The table below summarizes the crude dementia incidence rates. The crude hazard ratio (HR) was 3.69 (95% CI 3.58-3.80) for WMD and 2.80 (95% CI 2.68-2.93) for SBI. In the multivariable model controlling for all major cardiovascular risk factors, the effect of WMD on dementia risk was found to be stronger in younger compared to older patients and for CT- compared to MRI-discovered lesions. With CT, the adjusted HR was 3.08 (95% CI 2.84-3.34) for those < age 70 and 1.90 (95% CI 1.83-1.97) for those ≥ age 70. With MRI, the adjusted HR was 2.58 (95% CI 2.31-2.87) for those < age 70 and 1.59 (95% CI 1.45-1.75) for those ≥ age 70. The adjusted HR associated with a finding of SBI was 2.01 (95% CI 1.78-2.32) for patients age <70 and 1.26 (95% CI 1.20-1.32) for patients age ≥70, and was not modified by imaging modality.

Conclusion: Incidentally-discovered WMD and SBI are common in patients ≥ age 50 and are associated with substantial increase in the risk of subsequent dementia. The findings may represent an opportunity for dementia prevention.

Abstract P57: Risk Factors for Incidentally Discovered Silent Brain Infarcts and White Matter Disease in a Real World Cohort Identified by Artificial Intelligence

Introduction: Silent brain infarcts (SBIs) and white matter disease (WMD) are highly prevalent and associated with increased risk of ischemic stroke in patients with traditional stroke risk factors (RFs) in prospective cohort studies. Their frequency and associations with stroke RFs have not been well described in real world populations.

Methods: This was a cross-sectional study of patients age ≥ 50 in the Kaiser Permanente-Southern California (KPSC) health system between 2009-2019 with a head CT or MRI for non-stroke indications and no history of ischemic stroke, transient ischemic attack, or dementia. A natural language processing (NLP) algorithm developed at Mayo Clinic and Tufts Medical Center was applied to the KPSC EHR to identify individuals with reported SBIs or WMD. Multivariable Poisson regression with robust error variance was used to estimate risk ratios of demographics, stroke RFs (from the Framingham Stroke Risk Score), and scan modality on the presence of SBIs or WMD.

Results: Among 262,875 individuals, the NLP identified 13,154 (5.0%) with SBIs and 78,330 (29.8%) with WMD. Stroke RFs were highly prevalent in this cohort. The majority underwent CTs (74.8%) instead of MRIs as their initial neuroimaging. After adjustment for demographics and RFs, advanced age demonstrated a strong association with increased risk of SBIs and WMD (table). MRI was associated with a reduced risk of reported SBIs (ARR: 0.87, 95% CI 0.83-0.91) and an increased risk of reported WMD (ARR 2.86, 95% CI 2.83-2.90). Despite being prevalent, traditional stroke RFs had weak associations with increased risk of SBIs or increased risk of WMD.

Conclusions: Advanced age is strongly associated with incidentally discovered SBIs and WMD on neuroimaging studies obtained in routine care. The development of SBIs and WMD may not be fully attributable to traditional stroke RFs.