Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network

Rule-Based Cohort Definitions for Acute Respiratory Failure: Electronic Phenotyping Algorithm

Background

Acute respiratory failure is generally treated with invasive mechanical ventilation or noninvasive respiratory support strategies. The efficacies of the various strategies are not fully understood. There is a need for accurate therapy-based phenotyping for secondary analyses of electronic health record data to answer research questions regarding respiratory management and outcomes with each strategy.

Objective

The objective of this study was to address knowledge gaps related to ventilation therapy strategies across diverse patient populations by developing an algorithm for accurate identification of patients with acute respiratory failure. To accomplish this objective, our goal was to develop rule-based computable phenotypes for patients with acute respiratory failure using remotely monitored intensive care unit (tele-ICU) data. This approach permits analyses by ventilation strategy across broad patient populations of interest with the ability to sub-phenotype as research questions require.

Methods

Tele-ICU data from ≥200 hospitals were used to create a rule-based algorithm for phenotyping patients with acute respiratory failure, defined as an adult patient requiring invasive mechanical ventilation or a noninvasive strategy. The dataset spans a wide range of hospitals and ICU types across all US regions. Structured clinical data, including ventilation therapy start and stop times, medication records, and nurse and respiratory therapy charts, were used to define clinical phenotypes. All adult patients of any diagnoses with record of ventilation therapy were included. Patients were categorized by ventilation type, and analysis of event sequences using record timestamps defined each phenotype. Manual validation was performed on 5% of patients in each phenotype.

Results

We developed 7 phenotypes: (0) invasive mechanical ventilation, (1) noninvasive positive-pressure ventilation, (2) high-flow nasal insufflation, (3) noninvasive positive-pressure ventilation subsequently requiring intubation, (4) high-flow nasal insufflation subsequently requiring intubation, (5) invasive mechanical ventilation with extubation to noninvasive positive-pressure ventilation, and (6) invasive mechanical ventilation with extubation to high-flow nasal insufflation. A total of 27,734 patients met our phenotype criteria and were categorized into these ventilation subgroups. Manual validation of a random selection of 5% of records from each phenotype resulted in a total accuracy of 88% and a precision and recall of 0.8789 and 0.8785, respectively, across all phenotypes. Individual phenotype validation showed that the algorithm categorizes patients particularly well but has challenges with patients that require ≥2 management strategies.

Conclusions

Our proposed computable phenotyping algorithm for patients with acute respiratory failure effectively identifies patients for therapy-focused research regardless of admission diagnosis or comorbidities and allows for management strategy comparisons across populations of interest.

The effect of number of healthcare visits on study sample selection in electronic health record data

Introduction

Few studies have addressed how to select a study sample when using electronic health record (EHR) data.

Objective

To examine how changing criterion for number of visits in EHR data required for inclusion in a study sample would impact one basic epidemiologic measure: estimates of disease period prevalence.

Methods

Year 2016 EHR data from three Midwestern health systems (Northwestern Medicine in Illinois, University of Iowa Health Care, and Froedtert & the Medical College of Wisconsin, all regional tertiary health care systems including hospitals and clinics) was used to examine how alternate definitions of the study sample, based on number of healthcare visits in one year, affected measures of disease period prevalence. In 2016, each of these health systems saw between 160,000 and 420,000 unique patients. Curated collections of ICD-9, ICD-10, and SNOMED codes (from CMS-approved electronic clinical quality measures) were used to define three diseases: acute myocardial infarction, asthma, and diabetic nephropathy).

Results

Across all health systems, increasing the minimum required number of visits to be included in the study sample monotonically increased crude period prevalence estimates. The rate at which prevalence estimates increased with number of visits varied across sites and across diseases.

Conclusion

In addition to providing thorough descriptions of case definitions, when using EHR data authors must carefully describe how a study sample is identified and report data for a range of sample definitions, including minimum number of visits, so that others can assess the sensitivity of reported results to sample definition in EHR data.

Key words

Electronic Health Records, Sampling Studies, Prevalence, Methods

Preparing Medical Imaging Data for Machine Learning

Artificial intelligence (AI) continues to garner substantial interest in medical imaging. The potential applications are vast and include the entirety of the medical imaging life cycle from image creation to diagnosis to outcome prediction. The chief obstacles to development and clinical implementation of AI algorithms include availability of sufficiently large, curated, and representative training data that includes expert labeling (eg, annotations). Current supervised AI methods require a curation process for data to optimally train, validate, and test algorithms. Currently, most research groups and industry have limited data access based on small sample sizes from small geographic areas. In addition, the preparation of data is a costly and time-intensive process, the results of which are algorithms with limited utility and poor generalization. In this article, the authors describe fundamental steps for preparing medical imaging data in AI algorithm development, explain current limitations to data curation, and explore new approaches to address the problem of data availability.

Using the electronic health record for genomics research

PURPOSE OF REVIEW

Although primarily designed for medical documentation and billing purposes, the electronic health record (EHR) has significant potential for translational research. In this article, we provide an overview of the use of the EHR for genomics research with a focus on heritable lipid disorders.

RECENT FINDINGS

Linking the EHR to genomic data enables repurposing of vast phenotype data for genomic discovery. EHR data can be used to study the genetic basis of common and rare disorders, identify subphenotypes of diseases, assess pathogenicity of novel genomic variants, investigate pleiotropy, and rapidly assemble cohorts for genomic medicine clinical trials. EHR-based discovery can inform clinical practice; examples include use of polygenic risk scores for assessing disease risk and use of phenotype data to interpret rare variants. Despite limitations such as missing data, variable use of standards and poor interoperablility between disparate systems, the EHR is a powerful resource for genomic research.

SUMMARY

When linked to genomic data, the EHR can be leveraged for genomic discovery, which in turn can inform clinical care, exemplifying the virtuous cycle of a learning healthcare system.

Computerized Adjudication of Coronary Heart Disease Events Using the Electronic Medical Record in HIV Clinical Research: Possibilities and Challenges Ahead

This pilot study assessed feasibility of computer-assisted electronic medical record (EMR) abstraction to ascertain coronary heart disease (CHD) event hospitalizations. We included a sample of 87 hospitalization records from participants the University of North Carolina (UNC) site of the Women's Interagency HIV Study (WIHS) and UNC Center for AIDS Research (CFAR) HIV Clinical Cohort who were hospitalized within UNC Healthcare System from July 2004 to July 2015. We compared a computer algorithm utilizing diagnosis/procedure codes, medications, and cardiac enzyme levels to adjudicate CHD events [myocardial infarction (MI)/coronary revascularization] from the EMR to standardized manual chart adjudication. Of 87 hospitalizations, 42 were classified as definite, 25 probable, and 20 non-CHD events by manual chart adjudication. A computer algorithm requiring presence of ≥1 CHD-related International Classification of Diseases, 9th Revision (ICD-9)/Current Procedural Terminology (CPT) code correctly identified 24 of 42 definite (57%), 29 of 67 probable/definite CHD (43%), and 95% of non-CHD events; additionally requiring clinically defined cardiac enzyme levels or administration of MI-related medications correctly identified 55%, 42%, and 95% of such events, respectively. Requiring any one of the ICD-9/CPT or cardiac enzyme criteria correctly identified 98% of definite, 97% of probable/definite CHD, and 85% of non-CHD events. Challenges included difficulty matching hospitalization dates, incomplete diagnosis code data, and multiple field names/locations of laboratory/medication data. Computer algorithms comprising only ICD-9/CPT codes failed to identify a sizable proportion of CHD events. Using a less restrictive algorithm yielded fewer missed events but increased the false-positive rate. Despite potential benefits of EMR-based research, there remain several challenges to fully computerized adjudication of CHD events.

Considerations for Improving the Portability of Electronic Health Record-Based Phenotype Algorithms

With the increased adoption of electronic health records, data collected for routine clinical care is used for health outcomes and population sciences research, including the identification of phenotypes. In recent years, research networks, such as eMERGE, OHDSI and PCORnet, have been able to increase statistical power and population diversity by combining patient cohorts. These networks share phenotype algorithms that are executed at each participating site. Here we observe experiences with phenotype algorithm portability across seven research networks and propose a generalizable framework for phenotype algorithm portability. Several strategies exist to increase the portability of phenotype algorithms, reducing the implementation effort needed by each site. These include using a common data model, standardized representation of the phenotype algorithm logic, and technical solutions to facilitate federated execution of queries. Portability is achieved by tradeoffs across three domains: Data, Authoring and Implementation, and multiple approaches were observed in representing portable phenotype algorithms. Our proposed framework will help guide future research in operationalizing phenotype algorithm portability at scale.

Optimizing Antihypertensive Medication Classification in Electronic Health Record-Based Data: Classification System Development and Methodological Comparison

Background

Computable phenotypes have the ability to utilize data within the electronic health record (EHR) to identify patients with certain characteristics. Many computable phenotypes rely on multiple types of data within the EHR including prescription drug information. Hypertension (HTN)-related computable phenotypes are particularly dependent on the correct classification of antihypertensive prescription drug information, as well as corresponding diagnoses and blood pressure information.

Objective

This study aimed to create an antihypertensive drug classification system to be utilized with EHR-based data as part of HTN-related computable phenotypes.

Methods

We compared 4 different antihypertensive drug classification systems based off of 4 different methodologies and terminologies, including 3 RxNorm Concept Unique Identifier (RxCUI)–based classifications and 1 medication name–based classification. The RxCUI-based classifications utilized data from (1) the Drug Ontology, (2) the new Medication Reference Terminology, and (3) the Anatomical Therapeutic Chemical Classification System and DrugBank, whereas the medication name–based classification relied on antihypertensive drug names. Each classification system was applied to EHR-based prescription drug data from hypertensive patients in the OneFlorida Data Trust.

Results

There were 13,627 unique RxCUIs and 8025 unique medication names from the 13,879,046 prescriptions. We observed a broad overlap between the 4 methods, with 84.1% (691/822) to 95.3% (695/729) of terms overlapping pairwise between the different classification methods. Key differences arose from drug products with multiple dosage forms, drug products with an indication of benign prostatic hyperplasia, drug products that contain more than 1 ingredient (combination products), and terms within the classification systems corresponding to retired or obsolete RxCUIs.

Conclusions

In total, 2 antihypertensive drug classifications were constructed, one based on RxCUIs and one based on medication name, that can be used in future computable phenotypes that require antihypertensive drug classifications.

Defining Major Depressive Disorder Cohorts Using the EHR: Multiple Phenotypes Based on ICD-9 Codes and Medication Orders

Background

Major Depressive Disorder (MDD) is one of the most common mental illnesses and a leading cause of disability worldwide. Electronic Health Records (EHR) allow researchers to conduct unprecedented large-scale observational studies investigating MDD, its disease development and its interaction with other health outcomes. While there exist methods to classify patients as clear cases or controls, given specific data requirements, there are presently no simple, generalizable, and validated methods to classify an entire patient population into varying groups of depression likelihood and severity.

Methods

We have tested a simple, pragmatic electronic phenotype algorithm that classifies patients into one of five mutually exclusive, ordinal groups, varying in depression phenotype. Using data from an integrated health system on 278,026 patients from a 10-year study period we have tested the convergent validity of these constructs using measures of external validation, including patterns of psychiatric prescriptions, symptom severity, indicators of suicidality, comorbidity, mortality, health care utilization, and polygenic risk scores for MDD.

Results

We found consistent patterns of increasing morbidity and/or adverse outcomes across the five groups, providing evidence for convergent validity.

Limitations

The study population is from a single rural integrated health system which is predominantly white, possibly limiting its generalizability.

Conclusion

Our study provides initial evidence that a simple algorithm, generalizable to most EHR data sets, provides categories with meaningful face and convergent validity that can be used for stratification of an entire patient population.

Multisite validation of a simple electronic health record algorithm for identifying diagnosed obstructive sleep apnea

STUDY OBJECTIVES

We examined the performance of a simple algorithm to accurately distinguish cases of diagnosed obstructive sleep apnea (OSA) and noncases using the electronic health record (EHR) across six health systems in the United States.

METHODS

Retrospective analysis of EHR data was performed. The algorithm defined cases as individuals with ≥ 2 instances of specific International Classification of Diseases (ICD)-9 and/or ICD-10 diagnostic codes (327.20, 327.23, 327.29, 780.51, 780.53, 780.57, G4730, G4733 and G4739) related to sleep apnea on separate dates in their EHR. Noncases were defined by the absence of these codes. Using chart reviews on 120 cases and 100 noncases at each site (n = 1,320 total), positive predictive value (PPV) and negative predictive value (NPV) were calculated.

RESULTS

The algorithm showed excellent performance across sites, with a PPV (95% confidence interval) of 97.1 (95.6, 98.2) and NPV of 95.5 (93.5, 97.0). Similar performance was seen at each site, with all NPV and PPV estimates ≥ 90% apart from a somewhat lower PPV of 87.5 (80.2, 92.8) at one site. A modified algorithm of ≥ 3 instances improved PPV to 94.9 (88.5, 98.3) at this site, but excluded an additional 18.3% of cases. Thus, performance may be further improved by requiring additional codes, but this reduces the number of determinate cases.

CONCLUSIONS

A simple EHR-based case-identification algorithm for diagnosed OSA showed excellent predictive characteristics in a multisite sample from the United States. Future analyses should be performed to understand the effect of undiagnosed disease in EHR-defined noncases. This algorithm has wide-ranging applications for EHR-based OSA research.

UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER

OBJECTIVE

Electronic health records (EHRs) are a rich source of information on human diseases, but the information is variably structured, fragmented, curated using different coding systems, and collected for purposes other than medical research. We describe an approach for developing, validating, and sharing reproducible phenotypes from national structured EHR in the United Kingdom with applications for translational research.

MATERIALS AND METHODS

We implemented a rule-based phenotyping framework, with up to 6 approaches of validation. We applied our framework to a sample of 15 million individuals in a national EHR data source (population-based primary care, all ages) linked to hospitalization and death records in England. Data comprised continuous measurements (for example, blood pressure; medication information; coded diagnoses, symptoms, procedures, and referrals), recorded using 5 controlled clinical terminologies: (1) read (primary care, subset of SNOMED-CT [Systematized Nomenclature of Medicine Clinical Terms]), (2) International Classification of Diseases-Ninth Revision and Tenth Revision (secondary care diagnoses and cause of mortality), (3) Office of Population Censuses and Surveys Classification of Surgical Operations and Procedures, Fourth Revision (hospital surgical procedures), and (4) DM+D prescription codes.

RESULTS

Using the CALIBER phenotyping framework, we created algorithms for 51 diseases, syndromes, biomarkers, and lifestyle risk factors and provide up to 6 validation approaches. The EHR phenotypes are curated in the open-access CALIBER Portal (https://www.caliberresearch.org/portal) and have been used by 40 national and international research groups in 60 peer-reviewed publications.

CONCLUSIONS

We describe a UK EHR phenomics approach within the CALIBER EHR data platform with initial evidence of validity and use, as an important step toward international use of UK EHR data for health research.

Effect of vocabulary mapping for conditions on phenotype cohorts

Objective

To study the effect on patient cohorts of mapping condition (diagnosis) codes from source billing vocabularies to a clinical vocabulary.

Materials and Methods

Nine International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) concept sets were extracted from eMERGE network phenotypes, translated to Systematized Nomenclature of Medicine - Clinical Terms concept sets, and applied to patient data that were mapped from source ICD9-CM and ICD10-CM codes to Systematized Nomenclature of Medicine - Clinical Terms codes using Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) vocabulary mappings. The original ICD9-CM concept set and a concept set extended to ICD10-CM were used to create patient cohorts that served as gold standards.

Results

Four phenotype concept sets were able to be translated to Systematized Nomenclature of Medicine - Clinical Terms without ambiguities and were able to perform perfectly with respect to the gold standards. The other 5 lost performance when 2 or more ICD9-CM or ICD10-CM codes mapped to the same Systematized Nomenclature of Medicine - Clinical Terms code. The patient cohorts had a total error (false positive and false negative) of up to 0.15% compared to querying ICD9-CM source data and up to 0.26% compared to querying ICD9-CM and ICD10-CM data. Knowledge engineering was required to produce that performance; simple automated methods to generate concept sets had errors up to 10% (one outlier at 250%).

Discussion

The translation of data from source vocabularies to Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) resulted in very small error rates that were an order of magnitude smaller than other error sources.

Conclusion

It appears possible to map diagnoses from disparate vocabularies to a single clinical vocabulary and carry out research using a single set of definitions, thus improving efficiency and transportability of research.

Development and Validation of a Prediction Model for Atrial Fibrillation Using Electronic Health Records

OBJECTIVES

This study sought to determine whether the risk of atrial fibrillation AF can be estimated accurately by using routinely ascertained features in the electronic health record (EHR) and whether AF risk is associated with stroke.

BACKGROUND

Early diagnosis of AF and treatment with anticoagulation may prevent strokes.

METHODS

Using a multi-institutional EHR, this study identified 412,085 individuals 45 to 95 years of age without prevalent AF between 2000 and 2014. A prediction model was derived and validated for 5-year AF risk by using split-sample validation and model performance was compared with other methods of AF risk assessment.

RESULTS

Within 5 years, 14,334 individuals developed AF. In the derivation sample (7,216 AF events of 206,042 total), the optimal risk model included sex, age, race, smoking, height, weight, diastolic blood pressure, hypertension, hyperlipidemia, heart failure, coronary heart disease, valvular disease, prior stroke, peripheral arterial disease, chronic kidney disease, hypothyroidism, and quadratic terms for height, weight, and age. In the validation sample (7,118 AF events of 206,043 total) the AF risk model demonstrated good discrimination (C-statistic: 0.777; 95% confidence interval [CI:] 0.771 to 0.783) and calibration (0.99; 95% CI: 0.96 to 1.01). Model discrimination and calibration were superior to CHARGE-AF (Cohorts for Heart and Aging Research in Genomic Epidemiology AF) (C-statistic: 0.753; 95% CI: 0.747 to 0.759; calibration slope: 0.72; 95% CI: 0.71 to 0.74), C2HEST (Coronary artery disease / chronic obstructive pulmonary disease; Hypertension; Elderly [age ≥75 years]; Systolic heart failure; Thyroid disease [hyperthyroidism]) (C-statistic: 0.754; 95% CI: 0.747 to 0.762; calibration slope: 0.44; 95% CI: 0.43 to 0.45), and CHA2DS2-VASc (Congestive heart failure, Hypertension, Age ≥75 years, Diabetes mellitus, Prior stroke, transient ischemic attack [TIA], or thromboembolism, Vascular disease, Age 65-74 years, Sex category [female]) scores (C-statistic: 0.702; 95% CI: 0.693 to 0.710; calibration slope: 0.37; 95% CI: 0.36 to 0.38). AF risk discriminated incident stroke (n = 4,814; C-statistic: 0.684; 95% CI: 0.677 to 0.692) and stroke within 90 days of incident AF (n = 327; C-statistic: 0.789; 95% CI: 0.764 to 0.814).

CONCLUSIONS

A model developed from a real-world EHR database predicted AF accurately and stratified stroke risk. Incorporating AF prediction into EHRs may enable risk-guided screening for AF.

A Review of Automatic Phenotyping Approaches using Electronic Health Records

Electronic Health Records (EHR) are a rich repository of valuable clinical information that exist in primary and secondary care databases. In order to utilize EHRs for medical observational research a range of algorithms for automatically identifying individuals with a specific phenotype have been developed. This review summarizes and offers a critical evaluation of the literature relating to studies conducted into the development of EHR phenotyping systems. This review describes phenotyping systems and techniques based on structured and unstructured EHR data. Articles published on PubMed and Google scholar between 2013 and 2017 have been reviewed, using search terms derived from Medical Subject Headings (MeSH). The popularity of using Natural Language Processing (NLP) techniques in extracting features from narrative text has increased. This increased attention is due to the availability of open source NLP algorithms, combined with accuracy improvement. In this review, Concept extraction is the most popular NLP technique since it has been used by more than 50% of the reviewed papers to extract features from EHR. High-throughput phenotyping systems using unsupervised machine learning techniques have gained more popularity due to their ability to efficiently and automatically extract a phenotype with minimal human effort.

Ensembles of natural language processing systems for portable phenotyping solutions

BACKGROUND

Manually curating standardized phenotypic concepts such as Human Phenotype Ontology (HPO) terms from narrative text in electronic health records (EHRs) is time consuming and error prone. Natural language processing (NLP) techniques can facilitate automated phenotype extraction and thus improve the efficiency of curating clinical phenotypes from clinical texts. While individual NLP systems can perform well for a single cohort, an ensemble-based method might shed light on increasing the portability of NLP pipelines across different cohorts.

METHODS

We compared four NLP systems, MetaMapLite, MedLEE, ClinPhen and cTAKES, and four ensemble techniques, including intersection, union, majority-voting and machine learning, for extracting generic phenotypic concepts. We addressed two important research questions regarding automated phenotype recognition. First, we evaluated the performance of different approaches in identifying generic phenotypic concepts. Second, we compared the performance of different methods to identify patient-specific phenotypic concepts. To better quantify the effects caused by concept granularity differences on performance, we developed a novel evaluation metric that considered concept hierarchies and frequencies. Each of the approaches was evaluated on a gold standard set of clinical documents annotated by clinical experts. One dataset containing 1,609 concepts derived from 50 clinical notes from two different institutions was used in both evaluations, and an additional dataset of 608 concepts derived from 50 case report abstracts obtained from PubMed was used for evaluation of identifying generic phenotypic concepts only.

RESULTS

For generic phenotypic concept recognition, the top three performers in the NYP/CUIMC dataset are union ensemble (F₁, 0.634), training-based ensemble (F₁, 0.632), and majority vote-based ensemble (F₁, 0.622). In the Mayo dataset, the top three are majority vote-based ensemble (F₁, 0.642), cTAKES (F₁, 0.615), and MedLEE (F₁, 0.559). In the PubMed dataset, the top three are majority vote-based ensemble (F₁, 0.719), training-based (F₁, 0.696) and MetaMapLite (F₁, 0.694). For identifying patient specific phenotypes, the top three performers in the NYP/CUIMC dataset are majority vote-based ensemble (F₁, 0.610), MedLEE (F₁, 0.609), and training-based ensemble (F₁, 0.585). In the Mayo dataset, the top three are majority vote-based ensemble (F₁, 0.604), cTAKES (F₁, 0.531) and MedLEE (F₁, 0.527).

CONCLUSIONS

Our study demonstrates that ensembles of natural language processing can improve both generic phenotypic concept recognition and patient specific phenotypic concept identification over individual systems. Among the individual NLP systems, each individual system performed best when they were applied in the dataset that they were primary designed for. However, combining multiple NLP systems to create an ensemble can generally improve the performance. Specifically, the ensemble can increase the results reproducibility across different cohorts and tasks, and thus provide a more portable phenotyping solution compared to individual NLP systems.

Detecting time-evolving phenotypic topics via tensor factorization on electronic health records: Cardiovascular disease case study

OBJECTIVE

Discovering subphenotypes of complex diseases can help characterize disease cohorts for investigative studies aimed at developing better diagnoses and treatments. Recent advances in unsupervised machine learning on electronic health record (EHR) data have enabled researchers to discover phenotypes without input from domain experts. However, most existing studies have ignored time and modeled diseases as discrete events. Uncovering the evolution of phenotypes - how they emerge, evolve and contribute to health outcomes - is essential to define more precise phenotypes and refine the understanding of disease progression. Our objective was to assess the benefits of an unsupervised approach that incorporates time to model diseases as dynamic processes in phenotype discovery.

METHODS

In this study, we applied a constrained non-negative tensor-factorization approach to characterize the complexity of cardiovascular disease (CVD) patient cohort based on longitudinal EHR data. Through tensor-factorization, we identified a set of phenotypic topics (i.e., subphenotypes) that these patients established over the 10 years prior to the diagnosis of CVD, and showed the progress pattern. For each identified subphenotype, we examined its association with the risk for adverse cardiovascular outcomes estimated by the American College of Cardiology/American Heart Association Pooled Cohort Risk Equations, a conventional CVD-risk assessment tool frequently used in clinical practice. Furthermore, we compared the subsequent myocardial infarction (MI) rates among the six most prevalent subphenotypes using survival analysis.

RESULTS

From a cohort of 12,380 adult CVD individuals with 1068 unique PheCodes, we successfully identified 14 subphenotypes. Through the association analysis with estimated CVD risk for each subtype, we found some phenotypic topics such as Vitamin D deficiency and depression, Urinary infections cannot be explained by the conventional risk factors. Through a survival analysis, we found markedly different risks of subsequent MI following the diagnosis of CVD among the six most prevalent topics (p < 0.0001), indicating these topics may capture clinically meaningful subphenotypes of CVD.

CONCLUSION

This study demonstrates the potential benefits of using tensor-decomposition to model diseases as dynamic processes from longitudinal EHR data. Our results suggest that this data-driven approach may potentially help researchers identify complex and chronic disease subphenotypes in precision medicine research.

Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network

BACKGROUND

Implementation of phenotype algorithms requires phenotype engineers to interpret human-readable algorithms and translate the description (text and flowcharts) into computable phenotypes - a process that can be labor intensive and error prone. To address the critical need for reducing the implementation efforts, it is important to develop portable algorithms.

METHODS

We conducted a retrospective analysis of phenotype algorithms developed in the Electronic Medical Records and Genomics (eMERGE) network and identified common customization tasks required for implementation. A novel scoring system was developed to quantify portability from three aspects: Knowledge conversion, clause Interpretation, and Programming (KIP). Tasks were grouped into twenty representative categories. Experienced phenotype engineers were asked to estimate the average time spent on each category and evaluate time saving enabled by a common data model (CDM), specifically the Observational Medical Outcomes Partnership (OMOP) model, for each category.

RESULTS

A total of 485 distinct clauses (phenotype criteria) were identified from 55 phenotype algorithms, corresponding to 1153 customization tasks. In addition to 25 non-phenotype-specific tasks, 46 tasks are related to interpretation, 613 tasks are related to knowledge conversion, and 469 tasks are related to programming. A score between 0 and 2 (0 for easy, 1 for moderate, and 2 for difficult portability) is assigned for each aspect, yielding a total KIP score range of 0 to 6. The average clause-wise KIP score to reflect portability is 1.37 ± 1.38. Specifically, the average knowledge (K) score is 0.64 ± 0.66, interpretation (I) score is 0.33 ± 0.55, and programming (P) score is 0.40 ± 0.64. 5% of the categories can be completed within one hour (median). 70% of the categories take from days to months to complete. The OMOP model can assist with vocabulary mapping tasks.

CONCLUSION

This study presents firsthand knowledge of the substantial implementation efforts in phenotyping and introduces a novel metric (KIP) to measure portability of phenotype algorithms for quantifying such efforts across the eMERGE Network. Phenotype developers are encouraged to analyze and optimize the portability in regards to knowledge, interpretation and programming. CDMs can be used to improve the portability for some 'knowledge-oriented' tasks.

Using natural language processing to construct a metastatic breast cancer cohort from linked cancer registry and electronic medical records data

Objectives

Most population-based cancer databases lack information on metastatic recurrence. Electronic medical records (EMR) and cancer registries contain complementary information on cancer diagnosis, treatment and outcome, yet are rarely used synergistically. To construct a cohort of metastatic breast cancer (MBC) patients, we applied natural language processing techniques within a semisupervised machine learning framework to linked EMR-California Cancer Registry (CCR) data.

Materials and Methods

We studied all female patients treated at Stanford Health Care with an incident breast cancer diagnosis from 2000 to 2014. Our database consisted of structured fields and unstructured free-text clinical notes from EMR, linked to CCR, a component of the Surveillance, Epidemiology and End Results Program (SEER). We identified de novo MBC patients from CCR and extracted information on distant recurrences from patient notes in EMR. Furthermore, we trained a regularized logistic regression model for recurrent MBC classification and evaluated its performance on a gold standard set of 146 patients.

Results

There were 11 459 breast cancer patients in total and the median follow-up time was 96.3 months. We identified 1886 MBC patients, 512 (27.1%) of whom were de novo MBC patients and 1374 (72.9%) were recurrent MBC patients. Our final MBC classifier achieved an area under the receiver operating characteristic curve (AUC) of 0.917, with sensitivity 0.861, specificity 0.878, and accuracy 0.870.

Discussion and Conclusion

To enable population-based research on MBC, we developed a framework for retrospective case detection combining EMR and CCR data. Our classifier achieved good AUC, sensitivity, and specificity without expert-labeled examples.

Drug-resistant epilepsy classified by a phenotyping algorithm associates with NTRK2

OBJECTIVE

Up to 40% of patients with epilepsy become drug resistant (DRE). Genetic factors are likely to play a role. While efforts have focused on the transporter and target hypotheses, neither of them fully explains the pan-pharmacoresistance seen in DRE.

MATERIALS AND METHODS

In this study, we developed and used a phenotyping algorithm for the identification of DRE, responders, and epilepsy-free controls that were sequenced using a gene panel developed by the Pharmacogenomics Research Network (PGRN), which includes 82 genes involved in drug response. We tested the transporter hypothesis of DRE, the association between drug resistance and variants in the ATP-binding cassette family of genes previously associated with DRE, and also investigated potential new genetic factors.

RESULTS

In the analysis of DRE vs controls, NTRK2 was significantly associated with DRE (rs76950094; P = 1.19 × 10^-7 and gene-based P-value = 1.67 × 10^-4 ). NTRK2 encodes TrkB, which is involved in the development and maturation of the central nervous system, and increased activation of TrkB signaling is suggested to promote epilepsy.

CONCLUSION

Although the role of NTRK2 in DRE needs to be elucidated, these results support alternative mechanisms underlying DRE, complementary to the existing hypotheses, that should be evaluated.

Augmented intelligence with natural language processing applied to electronic health records for identifying patients with non-alcoholic fatty liver disease at risk for disease progression

OBJECTIVE

Electronic health record (EHR) systems contain structured data (such as diagnostic codes) and unstructured data (clinical documentation). Clinical insights can be derived from analyzing both. The use of natural language processing (NLP) algorithms to effectively analyze unstructured data has been well demonstrated. Here we examine the utility of NLP for the identification of patients with non-alcoholic fatty liver disease, assess patterns of disease progression, and identify gaps in care related to breakdown in communication among providers.

MATERIALS AND METHODS

All clinical notes available on the 38,575 patients enrolled in the Mount Sinai BioMe cohort were loaded into the NLP system. We compared analysis of structured and unstructured EHR data using NLP, free-text search, and diagnostic codes with validation against expert adjudication. We then used the NLP findings to measure physician impression of progression from early-stage NAFLD to NASH or cirrhosis. Similarly, we used the same NLP findings to identify mentions of NAFLD in radiology reports that did not persist into clinical notes.

RESULTS

Out of 38,575 patients, we identified 2,281 patients with NAFLD. From the remainder, 10,653 patients with similar data density were selected as a control group. NLP outperformed ICD and text search in both sensitivity (NLP: 0.93, ICD: 0.28, text search: 0.81) and F2 score (NLP: 0.92, ICD: 0.34, text search: 0.81). Of 2281 NAFLD patients, 673 (29.5%) were believed to have progressed to NASH or cirrhosis. Among 176 where NAFLD was noted prior to NASH, the average progression time was 410 days. 619 (27.1%) NAFLD patients had it documented only in radiology notes and not acknowledged in other forms of clinical documentation. Of these, 170 (28.4%) were later identified as having likely developed NASH or cirrhosis after a median 1057.3 days.

DISCUSSION

NLP-based approaches were more accurate at identifying NAFLD within the EHR than ICD/text search-based approaches. Suspected NAFLD on imaging is often not acknowledged in subsequent clinical documentation. Many such patients are later found to have more advanced liver disease. Analysis of information flows demonstrated loss of key information that could have been used to help prevent the progression of early NAFLD (NAFL) to NASH or cirrhosis.

CONCLUSION

For identification of NAFLD, NLP performed better than alternative selection modalities. It then facilitated analysis of knowledge flow between physician and enabled the identification of breakdowns where key information was lost that could have slowed or prevented later disease progression.

Beyond the hype of big data and artificial intelligence: building foundations for knowledge and wisdom

Big data, coupled with the use of advanced analytical approaches, such as artificial intelligence (AI), have the potential to improve medical outcomes and population health. Data that are routinely generated from, for example, electronic medical records and smart devices have become progressively easier and cheaper to collect, process, and analyze. In recent decades, this has prompted a substantial increase in biomedical research efforts outside traditional clinical trial settings. Despite the apparent enthusiasm of researchers, funders, and the media, evidence is scarce for successful implementation of products, algorithms, and services arising that make a real difference to clinical care. This article collection provides concrete examples of how "big data" can be used to advance healthcare and discusses some of the limitations and challenges encountered with this type of research. It primarily focuses on real-world data, such as electronic medical records and genomic medicine, considers new developments in AI and digital health, and discusses ethical considerations and issues related to data sharing. Overall, we remain positive that big data studies and associated new technologies will continue to guide novel, exciting research that will ultimately improve healthcare and medicine-but we are also realistic that concerns remain about privacy, equity, security, and benefit to all.

Facilitating phenotype transfer using a common data model

BACKGROUND

Implementing clinical phenotypes across a network is labor intensive and potentially error prone. Use of a common data model may facilitate the process.

METHODS

Electronic Medical Records and Genomics (eMERGE) sites implemented the Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) Common Data Model across their electronic health record (EHR)-linked DNA biobanks. Two previously implemented eMERGE phenotypes were converted to OMOP and implemented across the network.

RESULTS

It was feasible to implement the common data model across sites, with laboratory data producing the greatest challenge due to local encoding. Sites were then able to execute the OMOP phenotype in less than one day, as opposed to weeks of effort to manually implement an eMERGE phenotype in their bespoke research EHR databases. Of the sites that could compare the current OMOP phenotype implementation with the original eMERGE phenotype implementation, specific agreement ranged from 100% to 43%, with disagreements due to the original phenotype, the OMOP phenotype, changes in data, and issues in the databases. Using the OMOP query as a standard comparison revealed differences in the original implementations despite starting from the same definitions, code lists, flowcharts, and pseudocode.

CONCLUSION

Using a common data model can dramatically speed phenotype implementation at the cost of having to populate that data model, though this will produce a net benefit as the number of phenotype implementations increases. Inconsistencies among the implementations of the original queries point to a potential benefit of using a common data model so that actual phenotype code and logic can be shared, mitigating human error in reinterpretation of a narrative phenotype definition.

A Roadmap for Foundational Research on Artificial Intelligence in Medical Imaging: From the 2018 NIH/RSNA/ACR/The Academy Workshop

Imaging research laboratories are rapidly creating machine learning systems that achieve expert human performance using open-source methods and tools. These artificial intelligence systems are being developed to improve medical image reconstruction, noise reduction, quality assurance, triage, segmentation, computer-aided detection, computer-aided classification, and radiogenomics. In August 2018, a meeting was held in Bethesda, Maryland, at the National Institutes of Health to discuss the current state of the art and knowledge gaps and to develop a roadmap for future research initiatives. Key research priorities include: 1, new image reconstruction methods that efficiently produce images suitable for human interpretation from source data; 2, automated image labeling and annotation methods, including information extraction from the imaging report, electronic phenotyping, and prospective structured image reporting; 3, new machine learning methods for clinical imaging data, such as tailored, pretrained model architectures, and federated machine learning methods; 4, machine learning methods that can explain the advice they provide to human users (so-called explainable artificial intelligence); and 5, validated methods for image de-identification and data sharing to facilitate wide availability of clinical imaging data sets. This research roadmap is intended to identify and prioritize these needs for academic research laboratories, funding agencies, professional societies, and industry.

Comorbidity Characterization Among eMERGE Institutions: A Pilot Evaluation with the Johns Hopkins Adjusted Clinical Groups® System

Electronic health records (EHR) are valuable to define phenotype selection algorithms used to identify cohorts ofpatients for sequencing or genome wide association studies (GWAS). To date, the electronic medical records and genomics (eMERGE) network institutions have developed and applied such algorithms to identify cohorts with associated DNA samples used to discover new genetic associations. For complex diseases, there are benefits to stratifying cohorts using comorbidities in order to identify their genetic determinants. The objective of this study was to: (a) characterize comorbidities in a range of phenotype-selected cohorts using the Johns Hopkins Adjusted Clinical Groups® (ACG®) System, (b) assess the frequency of important comorbidities in three commonly studied GWAS phenotypes, and (c) compare the comorbidity characterization of cases and controls. Our analysis demonstrates a framework to characterize comorbidities using the ACG system and identified differences in mean chronic condition count among GWAS cases and controls. Thus, we believe there is great potential to use the ACG system to characterize comorbidities among genetic cohorts selected based on EHR phenotypes.

Efficient Active Learning for Electronic Medical Record De-identification

Electronic medical records are often de-identified before disseminated for secondary uses. However, unstructured natural language records are challenging to de-identify while utilizing a considerable amount of expensive human annotation. In this investigation, we incorporate active learning into the de-identification workflow to reduce annotation requirements. We apply this approach to a real clinical trials dataset and a publicly available i2b2 dataset to illustrate that, when the machine learning de-identification system can actively request information to help create a better model from beyond the system (e.g., a knowledgeable human assistant), less training data will be needed to maintain or improve the performance of trained models in comparison to the typical passive learning framework. Specifically, with a batch size of 10 documents, it requires only 40 documents for an active learning approach to reach an F-measure of 0.9, while passive learning needs at least 25% more data for training a comparable model.

Validation of an Electronic Medical Record-Based Algorithm for Identifying Posttraumatic Stress Disorder in U.S. Veterans

We developed an algorithm for identifying U.S. veterans with a history of posttraumatic stress disorder (PTSD), using the Department of Veterans Affairs (VA) electronic medical record (EMR) system. This work was motivated by the need to create a valid EMR-based phenotype to identify thousands of cases and controls for a genome-wide association study of PTSD in veterans. We used manual chart review (n = 500) as the gold standard. For both the algorithm and chart review, three classifications were possible: likely PTSD, possible PTSD, and likely not PTSD. We used Lasso regression with cross-validation to select statistically significant predictors of PTSD from the EMR and then generate a predicted probability score of being a PTSD case for every participant in the study population (range: 0-1.00). Comparing the performance of our probabilistic approach (Lasso algorithm) to a rule-based approach (International Classification of Diseases [ICD] algorithm), the Lasso algorithm showed modestly higher overall percent agreement with chart review than the ICD algorithm (80% vs. 75%), higher sensitivity (0.95 vs. 0.84), and higher accuracy (AUC = 0.95 vs. 0.90). We applied a 0.7 probability cut-point to the Lasso results to determine final PTSD case-control status for the VA population. The final algorithm had a 0.99 sensitivity, 0.99 specificity, 0.95 positive predictive value, and 1.00 negative predictive value for PTSD classification (grouping possible PTSD and likely not PTSD) as determined by chart review. This algorithm may be useful for other research and quality improvement endeavors within the VA.

Machine Learning, Natural Language Processing, and the Electronic Health Record: Innovations in Mental Health Services Research

An unprecedented amount of clinical information is now available via electronic health records (EHRs). These massive data sets have stimulated opportunities to adapt computational approaches to track and identify target areas for quality improvement in mental health care. In this column, three key areas of EHR data science are described: EHR phenotyping, natural language processing, and predictive modeling. For each of these computational approaches, case examples are provided to illustrate their role in mental health services research. Together, adaptation of these methods underscores the need for standardization and transparency while recognizing the opportunities and challenges ahead.

Diagnostic Algorithms to Study Post-Concussion Syndrome Using Electronic Health Records: Validating a Method to Capture an Important Patient Population

Post-concussion syndrome (PCS) is characterized by persistent cognitive, somatic, and emotional symptoms after a mild traumatic brain injury (mTBI). Genetic and other biological variables may contribute to PCS etiology, and the emergence of biobanks linked to electronic health records (EHRs) offers new opportunities for research on PCS. We sought to validate the EHR data of PCS patients by comparing two diagnostic algorithms deployed in the Vanderbilt University Medical Center de-identified database of 2.8 million patient EHRs. The algorithms identified individuals with PCS by: 1) natural language processing (NLP) of narrative text in the EHR combined with structured demographic, diagnostic, and encounter data; or 2) coded billing and procedure data. The predictive value of each algorithm was assessed, and cases and controls identified by each approach were compared on demographic and medical characteristics. The NLP algorithm identified 507 cases and 10,857 controls. The negative predictive value in controls was 78% and the positive predictive value (PPV) in cases was 82%. Conversely, the coded algorithm identified 1142 patients with two or more PCS billing codes and had a PPV of 76%. Comparisons of PCS controls to both case groups recovered known epidemiology of PCS: cases were more likely than controls to be female and to have pre-morbid diagnoses of anxiety, migraine, and post-traumatic stress disorder. In contrast, controls and cases were equally likely to have attention deficit hyperactive disorder and learning disabilities, in accordance with the findings of recent systematic reviews of PCS risk factors. We conclude that EHRs are a valuable research tool for PCS. Ascertainment based on coded data alone had a predictive value comparable to an NLP algorithm, recovered known PCS risk factors, and maximized the number of included patients.

Feature extraction for phenotyping from semantic and knowledge resources

OBJECTIVE

Phenotyping algorithms can efficiently and accurately identify patients with a specific disease phenotype and construct electronic health records (EHR)-based cohorts for subsequent clinical or genomic studies. Previous studies have introduced unsupervised EHR-based feature selection methods that yielded algorithms with high accuracy. However, those selection methods still require expert intervention to tweak the parameter settings according to the EHR data distribution for each phenotype. To further accelerate the development of phenotyping algorithms, we propose a fully automated and robust unsupervised feature selection method that leverages only publicly available medical knowledge sources, instead of EHR data.

METHODS

SEmantics-Driven Feature Extraction (SEDFE) collects medical concepts from online knowledge sources as candidate features and gives them vector-form distributional semantic representations derived with neural word embedding and the Unified Medical Language System Metathesaurus. A number of features that are semantically closest and that sufficiently characterize the target phenotype are determined by a linear decomposition criterion and are selected for the final classification algorithm.

RESULTS

SEDFE was compared with the EHR-based SAFE algorithm and domain experts on feature selection for the classification of five phenotypes including coronary artery disease, rheumatoid arthritis, Crohn's disease, ulcerative colitis, and pediatric pulmonary arterial hypertension using both supervised and unsupervised approaches. Algorithms yielded by SEDFE achieved comparable accuracy to those yielded by SAFE and expert-curated features. SEDFE is also robust to the input semantic vectors.

CONCLUSION

SEDFE attains satisfying performance in unsupervised feature selection for EHR phenotyping. Both fully automated and EHR-independent, this method promises efficiency and accuracy in developing algorithms for high-throughput phenotyping.

Validation of the Use of Electronic Medical Records for Identification of Post-gastric Bypass Hypoglycemia Cases

OBJECTIVE

We sought to validate an algorithm designed to identify patients with post-gastric bypass hypoglycemia (PGBH) using clinician chart review.

METHODS

We conducted a chart review study of non-diabetic patients who underwent Roux-en-Y gastric bypass (RYGB) at our institution from 2004 to 2013. The electronic medical record (EMR) algorithm was based on any post-operative glucose <60 mg/dl, diagnosis of hypoglycemia, or medication use for treatment of PGBH and identified 158 charts as PGBH and 1048 charts without PGBH. Two clinicians independently reviewed a random selection of 80 cases and 80 control charts and determined the presence or absence of PGBH by searching the chart using keywords and reviewing laboratory results, medications, and clinic notes.

RESULTS

Of the 160 charts reviewed, the EMR algorithm agreed with the chart review for 130 (accuracy = 80%, 95% CI = 75-87%) with sensitivity of 89% (95% CI = 83-96%) and specificity of 86% (95% CI = 78-93%). We improved the algorithm's accuracy to 90% by limiting the search to data obtained 3 months or more following RYGB.

CONCLUSION

The EMR algorithm has high sensitivity, specificity, and accuracy to identify post-gastric bypass hypoglycemia within our patient cohort. The use EMR-based algorithms may be a useful tool for future research to improve our understanding of epidemiology and risk factors for post-bariatric surgery hypoglycemia.

Bias of Inaccurate Disease Mentions in Electronic Health Record-based Phenotyping

OBJECTIVES

Electronic health record (EHR)-based phenotyping is an automated technique for identifying patients diagnosed with a particular disease using EHR data. However, EHR-based phenotyping has difficulties in achieving satisfactorily high performance because clinical notes include disease mentions that ultimately signify something other than the patient's diagnosis (such as differential diagnosis or screening). Our objective is to quantify the influence of such disease mentions on EHR-based phenotyping performance.

METHODS

Physicians manually reviewed whether the disease mentions indicated the patients' diseases in 487,300 clinical notes of 4,430 patients. Particular focus was placed on disease mentions that did not signify the patient's diagnosis even though they did not have any syntactic modifier or indicator in the same sentences. Patients were then classified according to whether their clinical notes included such disease mentions.

RESULTS

Among the patients whose clinical notes included disease mentions without any modifier or indicator, the proportion of patients whose disease mentions signified the patients' diagnosis was 78.1% (on average). This value can be interpreted as the bias of disease mentions that did not signify the patient's diagnosis on the precision of EHR-based phenotyping by extracting disease mentions from clinical notes.

CONCLUSION

This study quantified the bias occurred owing to disease mentions that incorrectly signify a patient's diagnosis in the value of precision of EHR-based phenotyping from four dataset types. The results of this study will help researchers in diverse research environments with different available data types.

Validating laboratory defined chronic kidney disease in the electronic health record for patients in primary care

BACKGROUND

Electronic health record (EHR) data is increasingly used to identify patients with chronic kidney disease (CKD). EHR queries used to capture CKD status, identify comorbid conditions, measure awareness by providers, and track adherence to guideline-concordant processes of care have not been validated.

METHODS

We extracted EHR data for primary-care patients with two eGFRcreat 15-59 mL/min/1.73 m^2 at least 90 days apart. Two nephrologists manually reviewed a random sample of 50 charts to determine CKD status, associated comorbidities, and physician awareness of CKD. We also assessed the documentation of a CKD diagnosis with guideline-driven care.

RESULTS

Complete data were available on 1767 patients with query-defined CKD of whom 822 (47%) had a CKD diagnosis in their chart. Manual chart review confirmed the CKD diagnosis in 34 or 50 (68%) patients. Agreement between the reviewers and the EHR diagnoses on the presence of comorbidities was good (κ > 0.70, p < 0.05), except for congestive heart failure, (κ = 0.45, p < 0.05). Reviewers felt the providers were aware of CKD in 23 of 34 (68%) of the confirmed CKD cases. A CKD diagnosis was associated with higher odds of guideline-driven care including CKD-specific laboratory tests and prescriptions for statins. After adjustment, CKD diagnosis documentation was not significantly associated with ACE/ARB prescription.

CONCLUSIONS

Identifying CKD status by historical eGFRs overestimates disease prevalence. A CKD diagnosis in the patient chart was a reasonable surrogate for provider awareness of disease status, but CKD awareness remains relatively low. CKD in the patient chart was associated with higher rates of albuminuria testing and use of statins, but not use of ACE/ARB.

Lessons Learned From GWAS of Asthma

Asthma is a common complex disease of the airways. Genome-wide association studies (GWASs) of asthma have identified many risk alleles and loci that have been replicated in worldwide populations. Although the risk alleles identified by GWAS have small effects and explain only a small portion of prevalence, the discovery of asthma loci can provide an understanding of its genetic architecture and the molecular pathways involved in disease pathogenesis. These discoveries can translate into advances in clinical care by identifying therapeutic targets, preventive strategies and ultimately approaches for personalized medicine. In this review, we summarize results from GWAS of asthma from the past 10 years and the insights gleaned from these discoveries.

The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data

Electronic phenotyping is the task of ascertaining whether an individual has a medical condition of interest by analyzing their medical record and is foundational in clinical informatics. Increasingly, electronic phenotyping is performed via supervised learning. We investigate the effectiveness of multitask learning for phenotyping using electronic health records (EHR) data. Multitask learning aims to improve model performance on a target task by jointly learning additional auxiliary tasks and has been used in disparate areas of machine learning. However, its utility when applied to EHR data has not been established, and prior work suggests that its benefits are inconsistent. We present experiments that elucidate when multitask learning with neural nets improves performance for phenotyping using EHR data relative to neural nets trained for a single phenotype and to well-tuned baselines. We find that multitask neural nets consistently outperform single-task neural nets for rare phenotypes but underperform for relatively more common phenotypes. The effect size increases as more auxiliary tasks are added. Moreover, multitask learning reduces the sensitivity of neural nets to hyperparameter settings for rare phenotypes. Last, we quantify phenotype complexity and find that neural nets trained with or without multitask learning do not improve on simple baselines unless the phenotypes are sufficiently complex.

Training as an Intervention to Decrease Medical Record Abstraction Errors Multicenter Studies

Studies often rely on medical record abstraction as a major source of data. However, data quality from medical record abstraction has long been questioned. Electronic Health Records (EHRs) potentially add variability to the abstraction process due to the complexity of navigating and locating study data within these systems. We report training for and initial quality assessment of medical record abstraction for a clinical study conducted by the IDeA States Pediatric Clinical Trials Network (ISPCTN) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN) using medical record abstraction as the primary data source. As part of overall quality assurance, study-specific training for medical record abstractors was developed and deployed during study start-up. The training consisted of a didactic session with an example case abstraction and an independent abstraction of two standardized cases. Sixty-nine site abstractors from thirty sites were trained. The training was designed to achieve an error rate for each abstractor of no greater than 4.93% with a mean of 2.53%, at study initiation. Twenty-three percent of the trainees exceeded the acceptance limit on one or both of the training test cases, supporting the need for such training. We describe lessons learned in the design and operationalization of the study-specific, medical record abstraction training program.

From Big Data to Precision Medicine

For over a decade the term "Big data" has been used to describe the rapid increase in volume, variety and velocity of information available, not just in medical research but in almost every aspect of our lives. As scientists, we now have the capacity to rapidly generate, store and analyse data that, only a few years ago, would have taken many years to compile. However, "Big data" no longer means what it once did. The term has expanded and now refers not to just large data volume, but to our increasing ability to analyse and interpret those data. Tautologies such as "data analytics" and "data science" have emerged to describe approaches to the volume of available information as it grows ever larger. New methods dedicated to improving data collection, storage, cleaning, processing and interpretation continue to be developed, although not always by, or for, medical researchers. Exploiting new tools to extract meaning from large volume information has the potential to drive real change in clinical practice, from personalized therapy and intelligent drug design to population screening and electronic health record mining. As ever, where new technology promises "Big Advances," significant challenges remain. Here we discuss both the opportunities and challenges posed to biomedical research by our increasing ability to tackle large datasets. Important challenges include the need for standardization of data content, format, and clinical definitions, a heightened need for collaborative networks with sharing of both data and expertise and, perhaps most importantly, a need to reconsider how and when analytic methodology is taught to medical researchers. We also set "Big data" analytics in context: recent advances may appear to promise a revolution, sweeping away conventional approaches to medical science. However, their real promise lies in their synergy with, not replacement of, classical hypothesis-driven methods. The generation of novel, data-driven hypotheses based on interpretable models will always require stringent validation and experimental testing. Thus, hypothesis-generating research founded on large datasets adds to, rather than replaces, traditional hypothesis driven science. Each can benefit from the other and it is through using both that we can improve clinical practice.

Phenotyping through Semi-Supervised Tensor Factorization (PSST)

A computational phenotype is a set of clinically relevant and interesting characteristics that describe patients with a given condition. Various machine learning methods have been proposed to derive phenotypes in an automatic, high-throughput manner. Among these methods, computational phenotyping through tensor factorization has been shown to produce clinically interesting phenotypes. However, few of these methods incorporate auxiliary patient information into the phenotype derivation process. In this work, we introduce Phenotyping through Semi-Supervised Tensor Factorization (PSST), a method that leverages disease status knowledge about subsets of patients to generate computational phenotypes from tensors constructed from the electronic health records of patients. We demonstrate the potential of PSST to uncover predictive and clinically interesting computational phenotypes through case studies focusing on type-2 diabetes and resistant hypertension. PSST yields more discriminative phenotypes compared to the unsupervised methods and more meaningful phenotypes compared to a supervised method.

Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances

The importance of incorporating Natural Language Processing (NLP) methods in clinical informatics research has been increasingly recognized over the past years, and has led to transformative advances. Typically, clinical NLP systems are developed and evaluated on word, sentence, or document level annotations that model specific attributes and features, such as document content (e.g., patient status, or report type), document section types (e.g., current medications, past medical history, or discharge summary), named entities and concepts (e.g., diagnoses, symptoms, or treatments) or semantic attributes (e.g., negation, severity, or temporality). From a clinical perspective, on the other hand, research studies are typically modelled and evaluated on a patient- or population-level, such as predicting how a patient group might respond to specific treatments or patient monitoring over time. While some NLP tasks consider predictions at the individual or group user level, these tasks still constitute a minority. Owing to the discrepancy between scientific objectives of each field, and because of differences in methodological evaluation priorities, there is no clear alignment between these evaluation approaches. Here we provide a broad summary and outline of the challenging issues involved in defining appropriate intrinsic and extrinsic evaluation methods for NLP research that is to be used for clinical outcomes research, and vice versa. A particular focus is placed on mental health research, an area still relatively understudied by the clinical NLP research community, but where NLP methods are of notable relevance. Recent advances in clinical NLP method development have been significant, but we propose more emphasis needs to be placed on rigorous evaluation for the field to advance further. To enable this, we provide actionable suggestions, including a minimal protocol that could be used when reporting clinical NLP method development and its evaluation.

A case study evaluating the portability of an executable computable phenotype algorithm across multiple institutions and electronic health record environments

Electronic health record (EHR) algorithms for defining patient cohorts are commonly shared as free-text descriptions that require human intervention both to interpret and implement. We developed the Phenotype Execution and Modeling Architecture (PhEMA, http://projectphema.org) to author and execute standardized computable phenotype algorithms. With PhEMA, we converted an algorithm for benign prostatic hyperplasia, developed for the electronic Medical Records and Genomics network (eMERGE), into a standards-based computable format. Eight sites (7 within eMERGE) received the computable algorithm, and 6 successfully executed it against local data warehouses and/or i2b2 instances. Blinded random chart review of cases selected by the computable algorithm shows PPV ≥90%, and 3 out of 5 sites had >90% overlap of selected cases when comparing the computable algorithm to their original eMERGE implementation. This case study demonstrates potential use of PhEMA computable representations to automate phenotyping across different EHR systems, but also highlights some ongoing challenges.

The eMERGE genotype set of 83,717 subjects imputed to ~40 million variants genome wide and association with the herpes zoster medical record phenotype

The Electronic Medical Records and Genomics (eMERGE) network is a network of medical centers with electronic medical records linked to existing biorepository samples for genomic discovery and genomic medicine research. The network sought to unify the genetic results from 78 Illumina and Affymetrix genotype array batches from 12 contributing medical centers for joint association analysis of 83,717 human participants. In this report, we describe the imputation of eMERGE results and methods to create the unified imputed merged set of genome‐wide variant genotype data. We imputed the data using the Michigan Imputation Server, which provides a missing single‐nucleotide variant genotype imputation service using the minimac3 imputation algorithm with the Haplotype Reference Consortium genotype reference set. We describe the quality control and filtering steps used in the generation of this data set and suggest generalizable quality thresholds for imputation and phenotype association studies. To test the merged imputed genotype set, we replicated a previously reported chromosome 6 HLA‐B herpes zoster (shingles) association and discovered a novel zoster‐associated loci in an epigenetic binding site near the terminus of chromosome 3 (3p29).

Genome-wide association analysis of common genetic variants of resistant hypertension

Resistant hypertension (RHTN), defined as uncontrolled blood pressure (BP) ≥ 140/90 using three or more drugs or controlled BP (<140/90) using four or more drugs, is associated with adverse outcomes, including decline in kidney function. We conducted a genome-wide association analysis in 1194 White and Hispanic participants with hypertension and coronary artery disease from the INternational VErapamil-SR Trandolapril STudy-GENEtic Substudy (INVEST-GENES). Top variants associated with RHTN at p < 10^-4 were tested for replication in 585 White and Hispanic participants with hypertension and subcortical strokes from the Secondary Prevention of Subcortical Strokes GENEtic Substudy (SPS3-GENES). A genetic risk score for RHTN was created by summing the risk alleles of replicated RHTN signals. rs11749255 in MSX2 was associated with RHTN in INVEST (odds ratio (OR) (95% CI) = 1.50 (1.2-1.8), p = 7.3 × 10^-5) and replicated in SPS3 (OR = 2.0 (1.4-2.8), p = 4.3 × 10^-5), with genome-wide significance in meta-analysis (OR = 1.60 (1.3-1.9), p = 3.8 × 10^-8). Other replicated signals were in IFLTD1 and PTPRD. IFLTD1 rs6487504 was associated with RHTN in INVEST (OR = 1.90 (1.4-2.5), p = 1.1 × 10^-5) and SPS3 (OR = 1.70 (1.2-2.5), p = 4 × 10^-3). PTPRD rs324498, a previously reported RHTN signal, was among the top signals in INVEST (OR = 1.60 (1.3-2.0), p = 3.4 × 10^-5) and replicated in SPS3 (OR = 1.60 (1.1-2.4), one-sided p = 0.005). Participants with the highest number of risk alleles were at increased risk of RHTN compared to participants with a lower number (p-trend = 1.8 × 10^-15). Overall, we identified and replicated associations with RHTN in the MSX2, IFLTD1, and PTPRD regions, and combined these associations to create a genetic risk score.

Using whole genome scores to compare three clinical phenotyping methods in complex diseases

Genome-wide association studies depend on accurate ascertainment of patient phenotype. However, phenotyping is difficult, and it is often treated as an afterthought in these studies because of the expense involved. Electronic health records (EHRs) may provide higher fidelity phenotypes for genomic research than other sources such as administrative data. We used whole genome association models to evaluate different EHR and administrative data-based phenotyping methods in a cohort of 16,858 Caucasian subjects for type 1 diabetes mellitus, type 2 diabetes mellitus, coronary artery disease and breast cancer. For each disease, we trained and evaluated polygenic models using three different phenotype definitions: phenotypes derived from billing data, the clinical problem list, or a curated phenotyping algorithm. We observed that for these diseases, the curated phenotype outperformed the problem list, and the problem list outperformed administrative billing data. This suggests that using advanced EHR-derived phenotypes can further increase the power of genome-wide association studies.

A Census of Disease Ontologies

For centuries, humans have sought to classify diseases based on phenotypic presentation and available treatments. Today, a wide landscape of strategies, resources, and tools exist to classify patients and diseases. Ontologies can provide a robust foundation of logic for precise stratification and classification along diverse axes such as etiology, development, treatment, and genetics. Disease and phenotype ontologies are used in four primary ways: ( a) search, retrieval, and annotation of knowledge; ( b) data integration and analysis; ( c) clinical decision support; and ( d) knowledge discovery. Computational inference can connect existing knowledge and generate new insights and hypotheses about drug targets, prognosis prediction, or diagnosis. In this review, we examine the rise of disease and phenotype ontologies and the diverse ways they are represented and applied in biomedicine.

Identifying Asthma Exacerbation-Related Emergency Department Visit Using Electronic Medical Record and Claims Data

Background  Asthma exacerbation leading to emergency department (ED) visit is prevalent, an indicator of poor control of asthma, and is a potentially preventable clinical outcome.

Objective  We propose to utilize multiple data elements available in electronic medical records (EMRs) and claims database to create separate algorithms with high validity for clinical and research purposes to identify asthma exacerbation-related ED visit among the general population.

Methods  We performed a retrospective study with inclusion criteria of patients aged 4 to 40 years, a visit to Geisinger ED from January 1, 2006, to October 28, 2013, with asthma on their problem list. Different electronic data elements including chief complaints, vitals, season, smoking, medication use, and discharge diagnoses were obtained to create the algorithm. A stratified random sample was generated to select the charts for review. Chart review was performed to classify patients with asthma-related ED visit, that is, the gold standard. Two reviewers performed the chart review and validation was done on a small subset.

Results  There were 966 eligible ED visits in the EMR sample and 731 in the claims sample. Agreement between reviewers was 95.45% and kappa statistic was 0.91. Mean age of the EMR sample was 22 years, and mostly white (93%). Multiple models conventionally used in studies were evaluated and the final model chosen included principal diagnosis, bronchodilator, and steroid use for both algorithms, chief complaints for EMR, and secondary diagnosis for claims. Area under the curve was 0.93 (95% confidence interval: 0.91–0.94) and 0.94 (0.93–0.96), respectively, for EMR and claims data, with positive predictive value of > 94%. The algorithms are visually presented using nomograms.

Conclusion  We were able to develop two separate algorithms for EMR and claims to identify asthma exacerbation-related ED visit with excellent diagnostic ability and varying discrimination threshold for clinical and research purposes.

Characteristics and treatment of African-American and European-American patients with resistant hypertension identified using the electronic health record in an academic health centre: a case-control study

OBJECTIVE

To identify patients with hypertension with resistant and controlled blood pressure (BP) using electronic health records (EHRs) in order to elucidate practices in the real-world clinical treatment of hypertension and to enable future genetic studies.

DESIGN

Using EHRs, we developed and validated algorithms to identify patients with resistant and controlled hypertension.

SETTING

An academic medical centre in Nashville, Tennessee.

POPULATION

European-American (EA) and African-American (AA) patients with hypertension.

MAIN OUTCOME MEASURES

Demographic characteristics: race, age, gender, body mass index, outpatient BPs and the history of diabetes mellitus, chronic kidney disease stage 3, ischaemic heart disease, transient ischaemic attack, atrial fibrillation and sleep apnoea.

MEDICATION TREATMENT

All antihypertensive medication classes prescribed to a patient at the time of classification and ever prescribed following classification.

RESULTS

The algorithms had performance metrics exceeding 92%. The prevalence of resistant hypertension in the total hypertensive population was 7.3% in EA and 10.5% in AA. At diagnosis, AA were younger, heavier, more often female and had a higher incidence of type 2 diabetes and higher BPs than EA. AA with resistant hypertension were more likely to be treated with vasodilators, dihydropyridine calcium channel blockers and alpha-2 agonists while EA were more likely to be treated with angiotensin receptor blockers, renin inhibitors and beta blockers. Mineralocorticoid receptor antagonists use was increased in patients treated with more than four antihypertensive medications compared with patients treated with three (12.4% vs 2.6% in EA, p<0.001; 12.3% vs 2.8% in AA, p<0.001). The number of patients treated with a mineralocorticoid receptor antagonist increased to 37.4% in EA and 41.2% in AA over a mean follow-up period of 7.4 and 8.7 years, respectively.

CONCLUSIONS

Clinical treatment of resistant hypertension differs in EA and AA patients. These results demonstrate the feasibility of identifying resistant hypertension using an EHR.

Optimizing research in symptomatic uterine fibroids with development of a computable phenotype for use with electronic health records

BACKGROUND

Women with symptomatic uterine fibroids can report a myriad of symptoms, including pain, bleeding, infertility, and psychosocial sequelae. Optimizing fibroid research requires the ability to enroll populations of women with image-confirmed symptomatic uterine fibroids.

OBJECTIVE

Our objective was to develop an electronic health record-based algorithm to identify women with symptomatic uterine fibroids for a comparative effectiveness study of medical or surgical treatments on quality-of-life measures. Using an iterative process and text-mining techniques, an effective computable phenotype algorithm, composed of demographics, and clinical and laboratory characteristics, was developed with reasonable performance. Such algorithms provide a feasible, efficient way to identify populations of women with symptomatic uterine fibroids for the conduct of large traditional or pragmatic trials and observational comparative effectiveness studies. Symptomatic uterine fibroids, due to menorrhagia, pelvic pain, bulk symptoms, or infertility, are a source of substantial morbidity for reproductive-age women. Comparing Treatment Options for Uterine Fibroids is a multisite registry study to compare the effectiveness of hormonal or surgical fibroid treatments on women's perceptions of their quality of life. Electronic health record-based algorithms are able to identify large numbers of women with fibroids, but additional work is needed to develop electronic health record algorithms that can identify women with symptomatic fibroids to optimize fibroid research. We sought to develop an efficient electronic health record-based algorithm that can identify women with symptomatic uterine fibroids in a large health care system for recruitment into large-scale observational and interventional research in fibroid management.

STUDY DESIGN

We developed and assessed the accuracy of 3 algorithms to identify patients with symptomatic fibroids using an iterative approach. The data source was the Carolina Data Warehouse for Health, a repository for the health system's electronic health record data. In addition to International Classification of Diseases, Ninth Revision diagnosis and procedure codes and clinical characteristics, text data-mining software was used to derive information from imaging reports to confirm the presence of uterine fibroids. Results of each algorithm were compared with expert manual review to calculate the positive predictive values for each algorithm.

RESULTS

Algorithm 1 was composed of the following criteria: (1) age 18-54 years; (2) either ≥1 International Classification of Diseases, Ninth Revision diagnosis codes for uterine fibroids or mention of fibroids using text-mined key words in imaging records or documents; and (3) no International Classification of Diseases, Ninth Revision or Current Procedural Terminology codes for hysterectomy and no reported history of hysterectomy. The positive predictive value was 47% (95% confidence interval 39-56%). Algorithm 2 required ≥2 International Classification of Diseases, Ninth Revision diagnosis codes for fibroids and positive text-mined key words and had a positive predictive value of 65% (95% confidence interval 50-79%). In algorithm 3, further refinements included ≥2 International Classification of Diseases, Ninth Revision diagnosis codes for fibroids on separate outpatient visit dates, the exclusion of women who had a positive pregnancy test within 3 months of their fibroid-related visit, and exclusion of incidentally detected fibroids during prenatal or emergency department visits. Algorithm 3 achieved a positive predictive value of 76% (95% confidence interval 71-81%).

CONCLUSION

An electronic health record-based algorithm is capable of identifying cases of symptomatic uterine fibroids with moderate positive predictive value and may be an efficient approach for large-scale study recruitment.

Scalable and accurate deep learning with electronic health records

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient's chart.