Validation of risk prediction models applied to longitudinal electronic health record data for the prediction of major cardiovascular events in the presence of data shifts

Aims

Deep learning has dominated predictive modelling across different fields, but in medicine it has been met with mixed reception. In clinical practice, simple, statistical models and risk scores continue to inform cardiovascular disease risk predictions. This is due in part to the knowledge gap about how deep learning models perform in practice when they are subject to dynamic data shifts; a key criterion that common internal validation procedures do not address. We evaluated the performance of a novel deep learning model, BEHRT, under data shifts and compared it with several ML-based and established risk models.

Methods and results

Using linked electronic health records of 1.1 million patients across England aged at least 35 years between 1985 and 2015, we replicated three established statistical models for predicting 5-year risk of incident heart failure, stroke, and coronary heart disease. The results were compared with a widely accepted machine learning model (random forests), and a novel deep learning model (BEHRT). In addition to internal validation, we investigated how data shifts affect model discrimination and calibration. To this end, we tested the models on cohorts from (i) distinct geographical regions; (ii) different periods. Using internal validation, the deep learning models substantially outperformed the best statistical models by 6%, 8%, and 11% in heart failure, stroke, and coronary heart disease, respectively, in terms of the area under the receiver operating characteristic curve.

Conclusion

The performance of all models declined as a result of data shifts; despite this, the deep learning models maintained the best performance in all risk prediction tasks. Updating the model with the latest information can improve discrimination but if the prior distribution changes, the model may remain miscalibrated.

Blood pressure-lowering treatment for the prevention of cardiovascular events in patients with atrial fibrillation: An individual participant data meta-analysis

BACKGROUND

Randomised evidence on the efficacy of blood pressure (BP)-lowering treatment to reduce cardiovascular risk in patients with atrial fibrillation (AF) is limited. Therefore, this study aimed to compare the effects of BP-lowering drugs in patients with and without AF at baseline.

METHODS AND FINDINGS

The study was based on the resource provided by the Blood Pressure Lowering Treatment Trialists' Collaboration (BPLTTC), in which individual participant data (IPD) were extracted from trials with over 1,000 patient-years of follow-up in each arm, and that had randomly assigned patients to different classes of BP-lowering drugs, BP-lowering drugs versus placebo, or more versus less intensive BP-lowering regimens. For this study, only trials that had collected information on AF status at baseline were included. The effects of BP-lowering treatment on a composite endpoint of major cardiovascular events (stroke, ischaemic heart disease or heart failure) according to AF status at baseline were estimated using fixed-effect one-stage IPD meta-analyses based on Cox proportional hazards models stratified by trial. Furthermore, to assess whether the associations between the intensity of BP reduction and cardiovascular outcomes are similar in those with and without AF at baseline, we used a meta-regression. From the full BPLTTC database, 28 trials (145,653 participants) were excluded because AF status at baseline was uncertain or unavailable. A total of 22 trials were included with 188,570 patients, of whom 13,266 (7%) had AF at baseline. Risk of bias assessment showed that 20 trials were at low risk of bias and 2 trials at moderate risk. Meta-regression showed that relative risk reductions were proportional to trial-level intensity of BP lowering in patients with and without AF at baseline. Over 4.5 years of median follow-up, a 5-mm Hg systolic BP (SBP) reduction lowered the risk of major cardiovascular events both in patients with AF (hazard ratio [HR] 0.91, 95% confidence interval [CI] 0.83 to 1.00) and in patients without AF at baseline (HR 0.91, 95% CI 0.88 to 0.93), with no difference between subgroups. There was no evidence for heterogeneity of treatment effects by baseline SBP or drug class in patients with AF at baseline. The findings of this study need to be interpreted in light of its potential limitations, such as the limited number of trials, limitation in ascertaining AF cases due to the nature of the arrhythmia and measuring BP in patients with AF.

CONCLUSIONS

In this meta-analysis, we found that BP-lowering treatment reduces the risk of major cardiovascular events similarly in individuals with and without AF. Pharmacological BP lowering for prevention of cardiovascular events should be recommended in patients with AF.