Measurement Error and Misclassification in Electronic Medical Records: Methods to Mitigate Bias

Limitations Reported in Evaluating Effectiveness of Risk Minimization Measures in the EU during 2018-2021: A Qualitative Analysis of Industry-Sponsored Post-Authorization Safety Studies

Marketing-authorization holders evaluate the effectiveness of risk minimization measures (RMM) for medicines through the conduct of post-authorization safety studies (PASS). Earlier studies show that concluding on RMM effectiveness is challenging. The aim of this study was to describe reported limitations associated with RMM effectiveness assessments of industry-sponsored PASS that did not render a conclusion. We conducted a thematic analysis of study limitations extracted from assessment reports and study reports finalized by the Pharmacovigilance Risk Assessment Committee between 2018 and 2021. In 39 (61.0%) of the PASS a conclusion on RMM effectiveness was drawn, where 25 (39.0%) PASS was inconclusive. Most PASS had a cross-sectional design with surveys as primary data sources (73.4% and 65.6% respectively). Four main themes emerged: (i) survey-specific limitations, (ii) limitations specifically related to secondary use of data, (iii) general limitations related to study design, and (iv) limitations not related to study design. In general, frequently reported limitations were survey-related, such as selection bias or information bias. Interestingly, well-known study limitations related to secondary use of data such as missing or misclassification of data were more often presented in inconclusive compared with conclusive PASS. Given that about 40% of PASS did not allow a conclusion on RMM effectiveness, our results suggest prioritization for strategies to mitigate limitations related to the secondary use of data at the protocol stage, for example, through feasibility assessments. Although many databases may have incomplete registration of some variables, feasibility testing prior to conducting a PASS could contribute to meeting study objectives and concluding on RMM effectiveness.

Identifying erroneous height and weight values from adult electronic health records in the All of Us research program

INTRODUCTION

Electronic Health Records (EHR) are a useful data source for research, but their usability is hindered by measurement errors. This study investigated an automatic error detection algorithm for adult height and weight measurements in EHR for the All of Us Research Program (All of Us).

METHODS

We developed reference charts for adult heights and weights that were stratified on participant sex. Our analysis included 4,076,534 height and 5,207,328 wt measurements from ∼ 150,000 participants. Errors were identified using modified standard deviation scores, differences from their expected values, and significant changes between consecutive measurements. We evaluated our method with chart-reviewed heights (8,092) and weights (9,039) from 250 randomly selected participants and compared it with the current cleaning algorithm in All of Us.

RESULTS

The proposed algorithm classified 1.4 % of height and 1.5 % of weight errors in the full cohort. Sensitivity was 90.4 % (95 % CI: 79.0-96.8 %) for heights and 65.9 % (95 % CI: 56.9-74.1 %) for weights. Precision was 73.4 % (95 % CI: 60.9-83.7 %) for heights and 62.9 (95 % CI: 54.0-71.1 %) for weights. In comparison, the current cleaning algorithm has inferior performance in sensitivity (55.8 %) and precision (16.5 %) for height errors while having higher precision (94.0 %) and lower sensitivity (61.9 %) for weight errors.

DISCUSSION

Our proposed algorithm outperformed in detecting height errors compared to weights. It can serve as a valuable addition to the current All of Us cleaning algorithm for identifying erroneous height values.

Challenges in and Opportunities for Electronic Health Record-Based Data Analysis and Interpretation

Electronic health records (EHRs) have been increasingly adopted in clinical practices across the United States, providing a primary source of data for clinical research, particularly observational cohort studies. EHRs are a high-yield, low-maintenance source of longitudinal real-world data for large patient populations and provide a wealth of information and clinical contexts that are useful for clinical research and translation into practice. Despite these strengths, it is important to recognize the multiple limitations and challenges related to the use of EHR data in clinical research. Missing data are a major source of error and biases and can affect the representativeness of the cohort of interest, as well as the accuracy of the outcomes and exposures. Here, we aim to provide a critical understanding of the types of data available in EHRs and describe the impact of data heterogeneity, quality, and generalizability, which should be evaluated prior to and during the analysis of EHR data. We also identify challenges pertaining to data quality, including errors and biases, and examine potential sources of such biases and errors. Finally, we discuss approaches to mitigate and remediate these limitations. A proactive approach to addressing these issues can help ensure the integrity and quality of EHR data and the appropriateness of their use in clinical studies.

Leveraging External Validation Data: The Challenges of Transporting Measurement Error Parameters

Approaches to address measurement error frequently rely on validation data to estimate measurement error parameters (e.g., sensitivity and specificity). Acquisition of validation data can be costly, thus secondary use of existing data for validation is attractive. To use these external validation data, however, we may need to address systematic differences between these data and the main study sample. Here, we derive estimators of the risk and the risk difference that leverage external validation data to account for outcome misclassification. If misclassification is differential with respect to covariates that themselves are differentially distributed in the validation and study samples, the misclassification parameters are not immediately transportable. We introduce two ways to account for such covariates: (1) standardize by these covariates or (2) iteratively model the outcome. If conditioning on a covariate for transporting the misclassification parameters induces bias of the causal effect (e.g., M-bias), the former but not the latter approach is biased. We provide proof of identification, describe estimation using parametric models, and assess performance in simulations. We also illustrate implementation to estimate the risk of preterm birth and the effect of maternal HIV infection on preterm birth. Measurement error should not be ignored and it can be addressed using external validation data via transportability methods.

Long COVID risk and pre-COVID vaccination in an EHR-based cohort study from the RECOVER program

Long COVID, or complications arising from COVID-19 weeks after infection, has become a central concern for public health experts. The United States National Institutes of Health founded the RECOVER initiative to better understand long COVID. We used electronic health records available through the National COVID Cohort Collaborative to characterize the association between SARS-CoV-2 vaccination and long COVID diagnosis. Among patients with a COVID-19 infection between August 1, 2021 and January 31, 2022, we defined two cohorts using distinct definitions of long COVID-a clinical diagnosis (n = 47,404) or a previously described computational phenotype (n = 198,514)-to compare unvaccinated individuals to those with a complete vaccine series prior to infection. Evidence of long COVID was monitored through June or July of 2022, depending on patients' data availability. We found that vaccination was consistently associated with lower odds and rates of long COVID clinical diagnosis and high-confidence computationally derived diagnosis after adjusting for sex, demographics, and medical history.

A machine learning model identifies patients in need of autoimmune disease testing using electronic health records

Systemic autoimmune rheumatic diseases (SARDs) can lead to irreversible damage if left untreated, yet these patients often endure long diagnostic journeys before being diagnosed and treated. Machine learning may help overcome the challenges of diagnosing SARDs and inform clinical decision-making. Here, we developed and tested a machine learning model to identify patients who should receive rheumatological evaluation for SARDs using longitudinal electronic health records of 161,584 individuals from two institutions. The model demonstrated high performance for predicting cases of autoantibody-tested individuals in a validation set, an external test set, and an independent cohort with a broader case definition. This approach identified more individuals for autoantibody testing compared with current clinical standards and a greater proportion of autoantibody carriers among those tested. Diagnoses of SARDs and other autoimmune conditions increased with higher model probabilities. The model detected a need for autoantibody testing and rheumatology encounters up to five years before the test date and assessment date, respectively. Altogether, these findings illustrate that the clinical manifestations of a diverse array of autoimmune conditions are detectable in electronic health records using machine learning, which may help systematize and accelerate autoimmune testing.

Derivation and Validation of Clinical Phenotypes of the Cardiopulmonary Bypass-Induced Inflammatory Response

BACKGROUND

Precision medicine aims to change treatment from a "one-size-fits-all" approach to customized therapies based on the individual patient. Applying a precision medicine approach to a heterogeneous condition, such as the cardiopulmonary bypass (CPB)-induced inflammatory response, first requires identification of homogeneous subgroups that correlate with biological markers and postoperative outcomes. As a first step, we derived clinical phenotypes of the CPB-induced inflammatory response by identifying patterns in perioperative clinical variables using machine learning and simulation tools. We then evaluated whether these phenotypes were associated with biological response variables and clinical outcomes.

METHODS

This single-center, retrospective cohort study used Cleveland Clinic registry data from patients undergoing cardiac surgery with CPB from January 2010 to March 2020. Biomarker data from a subgroup of patients enrolled in a clinical trial were also included. Patients undergoing emergent surgery, off-pump surgery, transplantation, descending thoracoabdominal aortic surgery, and planned ventricular assist device placement were excluded. Preoperative and intraoperative variables of patient baseline characteristics (demographics, comorbidities, and laboratory data) and perioperative data (procedural data, CPB duration, and hemodynamics) were analyzed to derive clinical phenotypes using K-means-based consensus clustering analysis. Proportion of ambiguously clustered was used to assess cluster size and optimal cluster numbers. After clusters were formed, we summarized perioperative profiles, inflammatory biomarkers (eg, interleukin [IL]-6 and IL-8), kidney biomarkers (eg, urine neutrophil gelatinase-associated lipocalin [NGAL] and IL-18), and clinical outcomes (eg, mortality and hospital length of stay). Pairwise standardized difference was reported for all summarized variables.

RESULTS

Of 36,865 eligible cardiac surgery cases, 25,613 met inclusion criteria. Cluster analysis derived 3 clinical phenotypes: α, β, and γ. Phenotype α (n = 6157 [24%]) included older patients with more comorbidities, including heart and kidney failure. Phenotype β (n = 10,572 [41%]) patients were younger and mostly male. Phenotype γ (n = 8884 [35%]) patients were 58% female and had lower body mass index (BMI). Phenotype α patients had worse outcomes, including longer hospital length of stay (mean = 9 days for α versus 6 for both β [absolute standardized difference {ASD} = 1.15] and γ [ASD = 1.08]), more kidney failure, and higher mortality. Inflammatory biomarkers (IL-6 and IL-8) and kidney injury biomarkers (urine NGAL and IL-18) were higher with the α phenotype compared to β and γ immediately after surgery.

CONCLUSIONS

Deriving clinical phenotypes that correlate with response biomarkers and outcomes represents an initial step toward a precision medicine approach for the management of CPB-induced inflammatory response and lays the groundwork for future investigation, including an evaluation of the heterogeneity of treatment effect.

Diabetes as a risk factor for the onset of frozen shoulder: a systematic review and meta-analysis

OBJECTIVE

Summarise longitudinal observational studies to determine whether diabetes (types 1 and 2) is a risk factor for frozen shoulder.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

MEDLINE, Embase, AMED, PsycINFO, Web of Science Core Collection, CINAHL, Epistemonikos, Trip, PEDro, OpenGrey and The Grey Literature Report were searched on January 2019 and updated in June 2021. Reference screening and emailing professional contacts were also used.

ELIGIBILITY CRITERIA

Longitudinal observational studies that estimated the association between diabetes and developing frozen shoulder.

DATA EXTRACTION AND SYNTHESIS

Data extraction was completed by one reviewer and independently checked by another using a predefined extraction sheet. Risk of bias was judged using the Quality In Prognosis Studies tool. For studies providing sufficient data, random-effects meta-analysis was used to derive summary estimates of the association between diabetes and the onset of frozen shoulder.

RESULTS

A meta-analysis of six case-control studies including 5388 people estimated the odds of developing frozen shoulder for people with diabetes to be 3.69 (95% CI 2.99 to 4.56) times the odds for people without diabetes. Two cohort studies were identified, both suggesting diabetes was associated with frozen shoulder, with HRs of 1.32 (95% CI 1.22 to 1.42) and 1.67 (95% CI 1.46 to 1.91). Risk of bias was judged as high in seven studies and moderate in one study.

CONCLUSION

People with diabetes are more likely to develop frozen shoulder. Risk of unmeasured confounding was the main limitation of this systematic review. High-quality studies are needed to confirm the strength of, and understand reasons for, the association.

PROSPERO REGISTRATION NUMBER

CRD42019122963.

Diseases of the musculoskeletal system and connective tissue in relation to temporomandibular disorders-A SWEREG-TMD nationwide case-control study

INTRODUCTION

Temporomandibular disorders (TMD) are comprised by a heterogenous group of diagnoses with multifaceted and complex etiologies. Although diseases of the musculoskeletal system and connective tissue (MSD) have been reported as risk factors for developing TMD, no nationwide population-based registry studies have been conducted to investigate this possible link. The aim of this study was to investigate the association between MSD and TMD in a population-based sample using Swedish registry data, and to further investigate the difference in such association between patients diagnosed with TMD in a hospital setting and patients surgically treated for the condition.

MATERIALS AND METHODS

Population based case-control study using Swedish nationwide registry data. Data was collected between 1998 and 2016 from 33 315 incident cases and 333 122 controls aged ≥18, matched for sex, age, and living area. Cases were stratified into non-surgical (NS), surgically treated once (ST1) and surgically treated twice or more (ST2). Information on MSD exposure (ICD-10 M00-M99) was collected between 1964 and 2016. Odds ratios were calculated using conditional logistic regression, adjusted for country of birth, educational level, living area, and mental health comorbidity.

RESULTS

A significant association between MSD and the development of TMD was found for all diagnostic categories: arthropathies (OR 2.0, CI 1.9-2.0); systemic connective tissue disorders (OR 2.3, CI 2.1-2.4); dorsopathies (OR 2.2, CI 2.1-2.2); soft tissue disorders (OR 2.2, CI 2.2-2.3); osteopathies and chondropathies (OR 1.7, CI 1.6-1.8); and other disorders of the musculoskeletal system and connective tissue (OR 1.9, CI 1.8-2.1). The associations were generally much stronger for TMD requiring surgical treatment. The diagnostic group with the strongest association was inflammatory polyarthropathies, M05-M14 (OR 11.7, CI 8.6-15.9), which was seen in the ST2 group.

CONCLUSIONS

Patients with MSD diagnoses have a higher probability of being diagnosed with TMD, in comparison to individuals without MSD. This association is even stronger for TMD that requires surgery. The results are in line with earlier findings, but present new population-based evidence of a possible causal relationship between MSD and TMD, even after adjusting for known confounders. Both dentists and physicians should be aware of this association and be wary of early signs of painful TMD among patients with MSD, to make early referral and timely conservative treatment possible.

Long COVID Risk and Pre-COVID Vaccination: An EHR-Based Cohort Study from the RECOVER Program

Importance

Characterizing the effect of vaccination on long COVID allows for better healthcare recommendations.

Objective

To determine if, and to what degree, vaccination prior to COVID-19 is associated with eventual long COVID onset, among those a documented COVID-19 infection.

Design Settings and Participants

Retrospective cohort study of adults with evidence of COVID-19 between August 1, 2021 and January 31, 2022 based on electronic health records from eleven healthcare institutions taking part in the NIH Researching COVID to Enhance Recovery (RECOVER) Initiative, a project of the National Covid Cohort Collaborative (N3C).

Exposures

Pre-COVID-19 receipt of a complete vaccine series versus no pre-COVID-19 vaccination.

Main Outcomes and Measures

Two approaches to the identification of long COVID were used. In the clinical diagnosis cohort (n=47,752), ICD-10 diagnosis codes or evidence of a healthcare encounter at a long COVID clinic were used. In the model-based cohort (n=199,498), a computable phenotype was used. The association between pre-COVID vaccination and long COVID was estimated using IPTW-adjusted logistic regression and Cox proportional hazards.

Results

In both cohorts, when adjusting for demographics and medical history, pre-COVID vaccination was associated with a reduced risk of long COVID (clinic-based cohort: HR, 0.66; 95% CI, 0.55-0.80; OR, 0.69; 95% CI, 0.59-0.82; model-based cohort: HR, 0.62; 95% CI, 0.56-0.69; OR, 0.70; 95% CI, 0.65-0.75).

Conclusions and Relevance

Long COVID has become a central concern for public health experts. Prior studies have considered the effect of vaccination on the prevalence of future long COVID symptoms, but ours is the first to thoroughly characterize the association between vaccination and clinically diagnosed or computationally derived long COVID. Our results bolster the growing consensus that vaccines retain protective effects against long COVID even in breakthrough infections.

Key Points

Question: Does vaccination prior to COVID-19 onset change the risk of long COVID diagnosis?Findings: Four observational analyses of EHRs showed a statistically significant reduction in long COVID risk associated with pre-COVID vaccination (first cohort: HR, 0.66; 95% CI, 0.55-0.80; OR, 0.69; 95% CI, 0.59-0.82; second cohort: HR, 0.62; 95% CI, 0.56-0.69; OR, 0.70; 95% CI, 0.65-0.75).Meaning: Vaccination prior to COVID onset has a protective association with long COVID even in the case of breakthrough infections.

Considerations for observational study design: Comparing the evidence of opioid use between electronic health records and insurance claims

PURPOSE

Pharmacoepidemiology studies often use insurance claims and/or electronic health records (EHR) to capture information about medication exposure. The choice between these data sources has important implications.

METHODS

We linked EHR from a large academic health system (2015-2017) to Medicare insurance claims for patients undergoing surgery. Drug utilization was characterized based on medication order dates in the EHR, and prescription fill dates in Medicare claims. We compared opioid use documented in EHR orders to prescription claims in four time periods: 1) Baseline (182 days before surgery); 2) Perioperative period; 3) Discharge date; 4) Follow-up (90 days after surgery).

RESULTS

We identified 11 128 patients undergoing surgery. During baseline, 34.4% (EHR) versus 44.1% (claims) had evidence of opioid use, and 56.9% of all baseline use was reflected only in one data source. During the perioperative period, 78.8% (EHR) versus 47.6% (claims) had evidence of use. On the day of discharge, 59.6% (EHR) versus 45.5% (claims) had evidence of use, and 51.8% of all discharge use was reflected only in one data source. During follow-up, 4.3% (EHR) versus 10.4% (claims) were identified with prolonged opioid use following surgery with 81.4% of all prolonged use reflected only in one data source.

CONCLUSIONS

When characterizing opioid exposure, we found substantial discrepancies between EHR medication orders and prescription claims data. In all time periods assessed, most patients' use was reflected only in the EHR, or only in the claims, not both. The potential for misclassification of drug utilization must be evaluated carefully, and choice of data source may have large impacts on key study design elements.

Modern Learning from Big Data in Critical Care: Primum Non Nocere

Large and complex data sets are increasingly available for research in critical care. To analyze these data, researchers use techniques commonly referred to as statistical learning or machine learning (ML). The latter is known for large successes in the field of diagnostics, for example, by identification of radiological anomalies. In other research areas, such as clustering and prediction studies, there is more discussion regarding the benefit and efficiency of ML techniques compared with statistical learning. In this viewpoint, we aim to explain commonly used statistical learning and ML techniques and provide guidance for responsible use in the case of clustering and prediction questions in critical care. Clustering studies have been increasingly popular in critical care research, aiming to inform how patients can be characterized, classified, or treated differently. An important challenge for clustering studies is to ensure and assess generalizability. This limits the application of findings in these studies toward individual patients. In the case of predictive questions, there is much discussion as to what algorithm should be used to most accurately predict outcome. Aspects that determine usefulness of ML, compared with statistical techniques, include the volume of the data, the dimensionality of the preferred model, and the extent of missing data. There are areas in which modern ML methods may be preferred. However, efforts should be made to implement statistical frameworks (e.g., for dealing with missing data or measurement error, both omnipresent in clinical data) in ML methods. To conclude, there are important opportunities but also pitfalls to consider when performing clustering or predictive studies with ML techniques. We advocate careful valuation of new data-driven findings. More interaction is needed between the engineer mindset of experts in ML methods, the insight in bias of epidemiologists, and the probabilistic thinking of statisticians to extract as much information and knowledge from data as possible, while avoiding harm.

Propensity score trimming mitigates bias due to covariate measurement error in inverse probability of treatment weighted analyses: A plasmode simulation

BACKGROUND

Inverse probability of treatment weighting (IPTW) may be biased by influential observations, which can occur from misclassification of strong exposure predictors.

METHODS

We evaluated bias and precision of IPTW estimators in the presence of a misclassified confounder and assessed the effect of propensity score (PS) trimming. We generated 1000 plasmode cohorts of size N = 10 000, sampled with replacement from 6063 NHANES respondents (1999-2014) age 40 to 79 with labs and no statin use. We simulated statin exposure as a function of demographics and CVD risk factors; and outcomes as a function of 10-year CVD risk score and statin exposure (rate ratio [RR] = 0.5). For 5% of the people in selected populations (eg, all patients, exposed, those with outcomes), we randomly misclassified a confounder that strongly predicted exposure. We fit PS models and estimated RRs using IPTW and 1:1 PS matching, with and without asymmetric trimming.

RESULTS

IPTW bias was substantial when misclassification was differential by outcome (RR range: 0.38-0.63) and otherwise minimal (RR range: 0.51-0.53). However, trimming reduced bias for IPTW, nearly eliminating it at 5% trimming (RR range: 0.49-0.52). In one scenario, when the confounder was misclassified for 5% of those with outcomes (0.3% of cohort), untrimmed IPTW was more biased and less precise (RR = 0.37 [SE(logRR) = 0.21]) than matching (RR = 0.50 [SE(logRR) = 0.13]). After 1% trimming, IPTW estimates were unbiased and more precise (RR = 0.49 [SE(logRR) = 0.12]) than matching (RR = 0.51 [SE(logRR) = 0.14]).

CONCLUSIONS

Differential misclassification of a strong predictor of exposure resulted in biased and imprecise IPTW estimates. Asymmetric trimming reduced bias, with more precise estimates than matching.