The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation

Predictive models of miscarriage on the basis of data from a preconception cohort study

OBJECTIVE

To use self-reported preconception data to derive models that predict the risk of miscarriage.

DESIGN

Prospective preconception cohort study.

SETTING

Not applicable.

PATIENTS

Study participants were female, aged 21-45 years, residents of the United States or Canada, and attempting spontaneous pregnancy at enrollment during 2013-2022. Participants were followed for up to 12 months of pregnancy attempts; those who conceived were followed through pregnancy and postpartum. We restricted analyses to participants who conceived during the study period.

EXPOSURE

On baseline and follow-up questionnaires completed every 8 weeks until pregnancy, we collected self-reported data on sociodemographic factors, reproductive history, lifestyle, anthropometrics, diet, medical history, and male partner characteristics. We included 160 potential predictor variables in our models.

MAIN OUTCOME MEASURES

The primary outcome was a miscarriage, defined as pregnancy loss before 20 weeks of gestation. We followed participants from their first positive pregnancy test until miscarriage or a censoring event (induced abortion, ectopic pregnancy, loss of follow-up, or 20 weeks of gestation), whichever occurred first. We fit both survival and static models using Cox proportional hazards models, logistic regression, support vector machines, gradient-boosted trees, and random forest algorithms. We evaluated model performance using the concordance index (survival models) and the weighted F1 score (static models).

RESULTS

Among the 8,720 participants who conceived, 20.4% reported miscarriage. In multivariable models, the strongest predictors of miscarriage were female age, history of miscarriage, and male partner age. The weighted F1 score ranged from 73%-89% for static models and the concordance index ranged from 53%-56% for survival models, indicating better discrimination for the static models compared with the survival models (i.e., the ability of the model to discriminate between individuals with and without miscarriage). No appreciable differences were observed across strata of miscarriage history or among models restricted to ≥8 weeks of gestation.

CONCLUSION

Our findings suggest that miscarriage is not easily predicted on the basis of preconception lifestyle characteristics and that advancing age and a history of miscarriage are the most important predictors of incident miscarriage.

Developing Clinical Prediction Models Using Primary Care Electronic Health Record Data: The Impact of Data Preparation Choices on Model Performance

Objective

To quantify prediction model performance in relation to data preparation choices when using electronic health records (EHR).

Study Design and Setting

Cox proportional hazards models were developed for predicting the first-ever main adverse cardiovascular events using Dutch primary care EHR data. The reference model was based on a 1-year run-in period, cardiovascular events were defined based on both EHR diagnosis and medication codes, and missing values were multiply imputed. We compared data preparation choices based on (i) length of the run-in period (2- or 3-year run-in); (ii) outcome definition (EHR diagnosis codes or medication codes only); and (iii) methods addressing missing values (mean imputation or complete case analysis) by making variations on the derivation set and testing their impact in a validation set.

Results

We included 89,491 patients in whom 6,736 first-ever main adverse cardiovascular events occurred during a median follow-up of 8 years. Outcome definition based only on diagnosis codes led to a systematic underestimation of risk (calibration curve intercept: 0.84; 95% CI: 0.83-0.84), while complete case analysis led to overestimation (calibration curve intercept: -0.52; 95% CI: -0.53 to -0.51). Differences in the length of the run-in period showed no relevant impact on calibration and discrimination.

Conclusion

Data preparation choices regarding outcome definition or methods to address missing values can have a substantial impact on the calibration of predictions, hampering reliable clinical decision support. This study further illustrates the urgency of transparent reporting of modeling choices in an EHR data setting.

Atrial fibrillation: trends in prevalence and antithrombotic prescriptions in the community

Introduction

In the past decade, the atrial fibrillation (AF) landscape, including the treatment modalities, has drastically changed. This raises the question how AF prevalence and choices in antithrombotic therapy prescription have developed in the community over time.

Methods

Routine care data from the Julius General Practitioners’ Network (JGPN) were used to calculate the yearly prevalence of AF and to quantify the percentage of all patients who were prescribed a platelet inhibitor, vitamin K antagonist (VKA), non-VKA oral anticoagulant (NOAC) or no antithrombotic medication. To explore whether certain patient characteristics are associated with selective prescription of oral anticoagulants (OAC), we applied logistic regression analyses.

Results

From 2008 through 2017, the JGPN database included 7459 unique AF patients. During this period, the prevalence of AF increased from 0.4% to 1.4%. The percentage of patients prescribed a VKA declined from 47% to 41%, whereas the percentage of patients prescribed a NOAC rose from 0% to 20%. In patients with new-onset AF, older age, heart failure, diabetes mellitus, vascular disease and dementia were independently associated with a higher likelihood of VKA rather than NOAC prescription. In 2017, 25% of all patients with AF and a CHA₂DS₂-VASc score ≥ 2 were not prescribed OAC therapy (i.e. 8% with platelet inhibitor monotherapy and 17% without any antithrombotic therapy).

Conclusion

Between 2008 and 2017, AF prevalence in the community more than tripled. Prescription patterns showed possible ‘channelling’ of VKAs over NOACs in frailer, elderly patients, whereas still about one in every four AF patients with a CHA₂DS₂-VASc score ≥ 2 was not prescribed any prophylactic OAC therapy.

Supplementary Information

The online version of this article (10.1007/s12471-022-01667-x) contains supplementary material, which is available to authorized users.

Predictive models of pregnancy based on data from a preconception cohort study

STUDY QUESTION

Can we derive adequate models to predict the probability of conception among couples actively trying to conceive?

SUMMARY ANSWER

Leveraging data collected from female participants in a North American preconception cohort study, we developed models to predict pregnancy with performance of ∼70% in the area under the receiver operating characteristic curve (AUC).

WHAT IS KNOWN ALREADY

Earlier work has focused primarily on identifying individual risk factors for infertility. Several predictive models have been developed in subfertile populations, with relatively low discrimination (AUC: 59-64%).

STUDY DESIGN, SIZE, DURATION

Study participants were female, aged 21-45 years, residents of the USA or Canada, not using fertility treatment, and actively trying to conceive at enrollment (2013-2019). Participants completed a baseline questionnaire at enrollment and follow-up questionnaires every 2 months for up to 12 months or until conception. We used data from 4133 participants with no more than one menstrual cycle of pregnancy attempt at study entry.

PARTICIPANTS/MATERIALS, SETTING, METHODS

On the baseline questionnaire, participants reported data on sociodemographic factors, lifestyle and behavioral factors, diet quality, medical history and selected male partner characteristics. A total of 163 predictors were considered in this study. We implemented regularized logistic regression, support vector machines, neural networks and gradient boosted decision trees to derive models predicting the probability of pregnancy: (i) within fewer than 12 menstrual cycles of pregnancy attempt time (Model I), and (ii) within 6 menstrual cycles of pregnancy attempt time (Model II). Cox models were used to predict the probability of pregnancy within each menstrual cycle for up to 12 cycles of follow-up (Model III). We assessed model performance using the AUC and the weighted-F1 score for Models I and II, and the concordance index for Model III.

MAIN RESULTS AND THE ROLE OF CHANCE

Model I and II AUCs were 70% and 66%, respectively, in parsimonious models, and the concordance index for Model III was 63%. The predictors that were positively associated with pregnancy in all models were: having previously breastfed an infant and using multivitamins or folic acid supplements. The predictors that were inversely associated with pregnancy in all models were: female age, female BMI and history of infertility. Among nulligravid women with no history of infertility, the most important predictors were: female age, female BMI, male BMI, use of a fertility app, attempt time at study entry and perceived stress.

LIMITATIONS, REASONS FOR CAUTION

Reliance on self-reported predictor data could have introduced misclassification, which would likely be non-differential with respect to the pregnancy outcome given the prospective design. In addition, we cannot be certain that all relevant predictor variables were considered. Finally, though we validated the models using split-sample replication techniques, we did not conduct an external validation study.

WIDER IMPLICATIONS OF THE FINDINGS

Given a wide range of predictor data, machine learning algorithms can be leveraged to analyze epidemiologic data and predict the probability of conception with discrimination that exceeds earlier work.

STUDY FUNDING/COMPETING INTEREST(S)

The research was partially supported by the U.S. National Science Foundation (under grants DMS-1664644, CNS-1645681 and IIS-1914792) and the National Institutes for Health (under grants R01 GM135930 and UL54 TR004130). In the last 3 years, L.A.W. has received in-kind donations for primary data collection in PRESTO from FertilityFriend.com, Kindara.com, Sandstone Diagnostics and Swiss Precision Diagnostics. L.A.W. also serves as a fibroid consultant to AbbVie, Inc. The other authors declare no competing interests.

TRIAL REGISTRATION NUMBER

N/A.

Oral Anticoagulation Use in High-Risk Patients Is Improved by Elimination of False-Positive and Inactive Atrial Fibrillation Cases

BACKGROUND

Multiple registries have reported that >40% of high-risk atrial fibrillation patients are not taking oral anticoagulants. The purpose of our study was to determine the presence or absence of active atrial fibrillation and CHA₂DS₂-VASc (Congestive heart failure, Hypertension, Age ≥75 y, Diabetes mellitus, prior Stroke [or transient ischemic attack or thromboembolism], Vascular disease, Age 65-74 y, Sex category) risk factors to accurately identify high-risk atrial fibrillation (CHA₂DS₂-VASc ≥2) patients requiring oral anticoagulants and the magnitude of the anticoagulant treatment gap.

METHODS

We retrospectively adjudicated 6514 patients with atrial fibrillation documented by at least one of: billing diagnosis, electronic medical record encounter diagnosis, electronic medical record problem list, or electrocardiogram interpretation.

RESULTS

After review, 4555/6514 (69.9%) had active atrial fibrillation, while 1201 had no documented history of atrial fibrillation and 758 had a history of atrial fibrillation that was no longer active. After removing the 1201 patients without a confirmed atrial fibrillation diagnosis, oral anticoagulant use in high-risk patients increased to 71.1% (P < .0001 compared with 62.9% at baseline). Oral anticoagulant use increased to 79.7% when the 758 inactive atrial fibrillation patients were also eliminated from the analysis (P < .0001 compared with baseline). In the active high-risk atrial fibrillation group, there was no significant difference in the use of oral anticoagulants between men (80.7%) and women (78.8%) with a CHA₂DS₂-VASc ≥2, or in women with a CHA₂DS₂-VASc ≥3 (79.9%).

CONCLUSIONS

Current registries and health system health records with unadjudicated diagnoses over-report the number of high-risk atrial fibrillation patients not taking oral anticoagulants. Expert adjudication identifies a smaller treatment gap than previously described.

Validation of the BATT score for prehospital risk stratification of traumatic haemorrhagic death: usefulness for tranexamic acid treatment criteria

BACKGROUND

Tranexamic acid reduces surgical blood loss and reduces deaths from bleeding in trauma patients. Tranexamic acid must be given urgently, preferably by paramedics at the scene of the injury or in the ambulance. We developed a simple score (Bleeding Audit Triage Trauma score) to predict death from bleeding.

METHODS

We conducted an external validation of the BATT score using data from the UK Trauma Audit Research Network (TARN) from 1st January 2017 to 31st December 2018. We evaluated the impact of tranexamic acid treatment thresholds in trauma patients.

RESULTS

We included 104,862 trauma patients with an injury severity score of 9 or above. Tranexamic acid was administered to 9915 (9%) patients. Of these 5185 (52%) received prehospital tranexamic acid. The BATT score had good accuracy (Brier score = 6%) and good discrimination (C-statistic 0.90; 95% CI 0.89-0.91). Calibration in the large showed no substantial difference between predicted and observed death due to bleeding (1.15% versus 1.16%, P = 0.81). Pre-hospital tranexamic acid treatment of trauma patients with a BATT score of 2 or more would avoid 210 bleeding deaths by treating 61,598 patients instead of avoiding 55 deaths by treating 9915 as currently.

CONCLUSION

The BATT score identifies trauma patient at risk of significant haemorrhage. A score of 2 or more would be an appropriate threshold for pre-hospital tranexamic acid treatment.

Validity of Acute Cardiovascular Outcome Diagnoses Recorded in European Electronic Health Records: A Systematic Review

BACKGROUND

Electronic health records are widely used in cardiovascular disease research. We appraised the validity of stroke, acute coronary syndrome and heart failure diagnoses in studies conducted using European electronic health records.

METHODS

Using a prespecified strategy, we systematically searched seven databases from dates of inception to April 2019. Two reviewers independently completed study selection, followed by partial parallel data extraction and risk of bias assessment. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value estimates were narratively synthesized and heterogeneity between sensitivity and PPV estimates were assessed using I2.

RESULTS

We identified 81 studies, of which 20 validated heart failure diagnoses, 31 validated acute coronary syndrome diagnoses with 29 specifically recording estimates for myocardial infarction, and 41 validated stroke diagnoses. Few studies reported specificity or negative predictive value estimates. Sensitivity was ≤66% in all but one heart failure study, ≥80% for 91% of myocardial infarction studies, and ≥70% for 73% of stroke studies. PPV was ≥80% in 74% of heart failure, 88% of myocardial infarction, and 70% of stroke studies. PPV by stroke subtype was variable, at ≥80% for 80% of ischaemic stroke but only 44% of haemorrhagic stroke. There was considerable heterogeneity (I2 >75%) between sensitivity and PPV estimates for all diagnoses.

CONCLUSION

Overall, European electronic health record stroke, acute coronary syndrome and heart failure diagnoses are accurate for use in research, although validity estimates for heart failure and individual stroke subtypes were lower. Where possible, researchers should validate data before use or carefully interpret the results of previous validation studies for their own study purposes.

Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability

Kappa statistics have been widely used in the pathology literature to compare interobserver diagnostic variability (IOV) among different pathologists but there has been limited discussion about the clinical significance of kappa scores. Five representative and recent pathology papers were queried using clinically relevant specific questions to learn how IOV was evaluated and how the clinical applicability of results was interpreted. The papers supported our anecdotal impression that pathologists usually assess IOV using Cohen's or Fleiss' kappa statistics and interpret the results using some variation of the scale proposed by Landis and Koch. The papers did not cite or propose specific guidelines to comment on the clinical applicability of results. The solutions proposed to decrease IOV included the development of better diagnostic criteria and additional educational efforts, but the possibility that the entities themselves represented a continuum of morphologic findings rather than distinct diagnostic categories was not considered in any of the studies. A dataset from a previous study of IOV reported by Thunnissen et al. was recalculated to estimate percent agreement among 19 international lung pathologists for the diagnosis of 74 challenging lung neuroendocrine neoplasms. Kappa scores and diagnostic sensitivity, specificity, positive and negative predictive values were calculated using the majority consensus diagnosis for each case as the gold reference diagnosis for that case. Diagnostic specificity estimates among multiple pathologists were > 90%, although kappa scores were considerably more variable. We explain why kappa scores are of limited clinical applicability in pathology and propose the use of positive and negative percent agreement and diagnostic specificity against a gold reference diagnosis to evaluate IOV among two and multiple raters, respectively.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.