Implementation of a commercial federated network of electronic health record data to enable sponsor-initiated clinical trials at an academic medical center

BACKGROUND

A commercial federated network called TriNetX has connected electronic health record (EHR) data from academic medical centers (AMCs) with biopharmaceutical sponsors in a privacy-preserving manner to promote sponsor-initiated clinical trials. Little is known about how AMCs have implemented TriNetX to support clinical trials.

FINDINGS

At our AMC over a six-year period, TriNetX integrated into existing institutional workflows enabled 402 requests for sponsor-initiated clinical trials, 14 % (n = 56) of which local investigators expressed interest in conducting. Although clinical trials administrators indicated TriNetX yielded unique study opportunities, measurement of impact of institutional participation in the network was challenging due to lack of a common trial identifier shared across TriNetX, sponsor, and our institution.

CONCLUSION

To the best of our knowledge, this study is among the first to describe integration of a federated network of EHR data into institutional workflows for sponsor-initiated clinical trials. This case report may inform efforts at other institutions.

Comparative Merits of Available Mortality Data Sources for Clinical Research

Obtaining reliable data on patient mortality is a critical challenge facing observational researchers seeking to conduct studies using real-world data. As these analyses are conducted more broadly using newly-available sources of real-world evidence, missing data can serve as a rate-limiting factor. We conducted a comparison of mortality data sources from different stakeholder perspectives - academic medical center (AMC) informatics service providers, AMC research coordinators, industry analytics professionals, and academics - to understand the strengths and limitations of differing mortality data sources: locally generated data from sites conducting research, data provided by governmental sources, and commercially available data sets. Researchers seeking to conduct observational studies using extant data should consider these factors in sourcing outcomes data for their populations of interest.

A method to automate the discharge summary hospital course for neurology patients

OBJECTIVE

Generation of automated clinical notes has been posited as a strategy to mitigate physician burnout. In particular, an automated narrative summary of a patient's hospital stay could supplement the hospital course section of the discharge summary that inpatient physicians document in electronic health record (EHR) systems. In the current study, we developed and evaluated an automated method for summarizing the hospital course section using encoder-decoder sequence-to-sequence transformer models.

MATERIALS AND METHODS

We fine-tuned BERT and BART models and optimized for factuality through constraining beam search, which we trained and tested using EHR data from patients admitted to the neurology unit of an academic medical center.

RESULTS

The approach demonstrated good ROUGE scores with an R-2 of 13.76. In a blind evaluation, 2 board-certified physicians rated 62% of the automated summaries as meeting the standard of care, which suggests the method may be useful clinically.

DISCUSSION AND CONCLUSION

To our knowledge, this study is among the first to demonstrate an automated method for generating a discharge summary hospital course that approaches a quality level of what a physician would write.

A RE-AIM Evaluation of a Visualization-Based Electronic Patient-Reported Outcome System

OBJECTIVES

Health care systems are primarily collecting patient-reported outcomes (PROs) for research and clinical care using proprietary, institution- and disease-specific tools for remote assessment. The purpose of this study was to conduct a Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation of a scalable electronic PRO (ePRO) reporting and visualization system in a single-arm study.

METHODS

The "mi.symptoms" ePRO system was designed using gerontechnological design principles to ensure high usability among older adults. The system enables longitudinal reporting of disease-agnostic ePROs and includes patient-facing PRO visualizations. We conducted an evaluation of the implementation of the system guided by the RE-AIM framework. Quantitative data were analyzed using basic descriptive statistics, and qualitative data were analyzed using directed content analysis.

RESULTS

Reach-the total reach of the study was 70 participants (median age: 69, 31% female, 17% Black or African American, 27% reported not having enough financial resources). Effectiveness-half (51%) of participants completed the 2-week follow-up survey and 36% completed all follow-up surveys. Adoption-the desire for increased self-knowledge, the value of tracking symptoms, and altruism motivated participants to adopt the tool. Implementation-the predisposing factor was access to, and comfort with, computers. Three enabling factors were incorporation into routines, multimodal nudges, and ease of use. Maintenance-reinforcing factors were perceived usefulness of viewing symptom reports with the tool and understanding the value of sustained symptom tracking in general.

CONCLUSION

Challenges in ePRO reporting, particularly sustained patient engagement, remain. Nonetheless, freely available, scalable, disease-agnostic systems may pave the road toward inclusion of a more diverse range of health systems and patients in ePRO collection and use.

Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records

Background

In the global effort to prevent death by suicide, many academic medical institutions are implementing natural language processing (NLP) approaches to detect suicidality from unstructured clinical text in electronic health records (EHRs), with the hope of targeting timely, preventative interventions to individuals most at risk of suicide. Despite the international need, the development of these NLP approaches in EHRs has been largely local and not shared across healthcare systems.

Methods

In this study, we developed a process to share NLP approaches that were individually developed at King's College London (KCL), UK and Weill Cornell Medicine (WCM), US - two academic medical centers based in different countries with vastly different healthcare systems. We tested and compared the algorithms' performance on manually annotated clinical notes (KCL: n = 4,911 and WCM = 837).

Results

After a successful technical porting of the NLP approaches, our quantitative evaluation determined that independently developed NLP approaches can detect suicidality at another healthcare organization with a different EHR system, clinical documentation processes, and culture, yet do not achieve the same level of success as at the institution where the NLP algorithm was developed (KCL approach: F1-score 0.85 vs. 0.68, WCM approach: F1-score 0.87 vs. 0.72).

Limitations

Independent NLP algorithm development and patient cohort selection at the two institutions comprised direct comparability.

Conclusions

Shared use of these NLP approaches is a critical step forward towards improving data-driven algorithms for early suicide risk identification and timely prevention.

Design and implementation of an integrated data model to support clinical and translational research administration

OBJECTIVE

Both academic medical centers and biomedical research sponsors need to understand impact of scientific funding to determine value. For the National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) hubs, tracking research activities can be complex, often involving multiple institutions and continually changing federal reporting requirements. Existing research administrative systems are institution-specific and tend to focus only on parts of a greater whole. The goal of this case report is to describe a comprehensive data model that addresses this gap.

MATERIALS AND METHODS

Web-based Center Administrative Management Program (WebCAMP) has been developed over a period of over 15 years in the context of CTSA hubs, with the recent addition of T32 programs. Its data model centers around the key concepts of people, projects, resources (inputs), and outcomes (outputs).

RESULTS

The WebCAMP data model and associated toolset for biomedical research administration integrates multiple components of the research enterprise, has been used by our CTSA hub for over 15 years and has been adopted by more than 20 other CTSA hubs.

DISCUSSION

To the best of our knowledge, this study is among the first to describe a comprehensive data model for biomedical research administration. Opportunities for future work include improved grant tracking through the development of a universal identifier that spans public and private funders, and a more generic outcomes tracking model able to rapidly incorporate new outcome types.

CONCLUSION

We propose that the WebCAMP data model, or a derivative of it, could serve as a future standard for research administrative data warehousing.

A Day-to-Day Approach for Automating the Hospital Course Section of the Discharge Summary

Optimal solutions for abstractive summarization of electronic health record content have yet to be discovered. Although studies have applied state-of-the-art transformers in the clinical domain to radiology reports and information extraction, little is known of transformers' performance with the hospital course section of the discharge summary. This paper compares two summarization approaches for automating the hospital course section within the discharge summary: (1) a truncation approach that uses all clinical notes and (2) a day-to-day approach that segments the notes per clinical day. We pair both approaches with different transformer encoder-decoder based-models - BART, BERT2GPT2, ClinicalBERT2GPT2, and ClinicalBERT2ClinicalBERT and evaluate the transformers that work best for each approach using ROUGE metrics. The results demonstrate that the day-to-day approach can overcome the limitations of longform document summarization for the patient clinical record.

Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative

OBJECTIVE

In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations.

MATERIALS AND METHODS

We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements.

RESULTS

Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback.

DISCUSSION

We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate.

CONCLUSION

By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

Understanding enterprise data warehouses to support clinical and translational research: enterprise information technology relationships, data governance, workforce, and cloud computing

OBJECTIVE

Among National Institutes of Health Clinical and Translational Science Award (CTSA) hubs, effective approaches for enterprise data warehouses for research (EDW4R) development, maintenance, and sustainability remain unclear. The goal of this qualitative study was to understand CTSA EDW4R operations within the broader contexts of academic medical centers and technology.

MATERIALS AND METHODS

We performed a directed content analysis of transcripts generated from semistructured interviews with informatics leaders from 20 CTSA hubs.

RESULTS

Respondents referred to services provided by health system, university, and medical school information technology (IT) organizations as "enterprise information technology (IT)." Seventy-five percent of respondents stated that the team providing EDW4R service at their hub was separate from enterprise IT; strong relationships between EDW4R teams and enterprise IT were critical for success. Managing challenges of EDW4R staffing was made easier by executive leadership support. Data governance appeared to be a work in progress, as most hubs reported complex and incomplete processes, especially for commercial data sharing. Although nearly all hubs (n = 16) described use of cloud computing for specific projects, only 2 hubs reported using a cloud-based EDW4R. Respondents described EDW4R cloud migration facilitators, barriers, and opportunities.

DISCUSSION

Descriptions of approaches to how EDW4R teams at CTSA hubs work with enterprise IT organizations, manage workforces, make decisions about data, and approach cloud computing provide insights for institutions seeking to leverage patient data for research.

CONCLUSION

Identification of EDW4R best practices is challenging, and this study helps identify a breadth of viable options for CTSA hubs to consider when implementing EDW4R services.

Assessment of structured data elements for social risk factors

OBJECTIVES

Computable social risk factor phenotypes derived from routinely collected structured electronic health record (EHR) or health information exchange (HIE) data may represent a feasible and robust approach to measuring social factors. This study convened an expert panel to identify and assess the quality of individual EHR and HIE structured data elements that could be used as components in future computable social risk factor phenotypes.

STUDY DESIGN

Technical expert panel.

METHODS

A 2-round Delphi technique included 17 experts with an in-depth knowledge of available EHR and/or HIE data. The first-round identification sessions followed a nominal group approach to generate candidate data elements that may relate to socioeconomics, cultural context, social relationships, and community context. In the second-round survey, panelists rated each data element according to overall data quality and likelihood of systematic differences in quality across populations (ie, bias).

RESULTS

Panelists identified a total of 89 structured data elements. About half of the data elements (n = 45) were related to socioeconomic characteristics. The panelists identified a diverse set of data elements. Elements used in reimbursement-related processes were generally rated as higher quality. Panelists noted that several data elements may be subject to implicit bias or reflect biased systems of care, which may limit their utility in measuring social factors.

CONCLUSIONS

Routinely collected structured data within EHR and HIE systems may reflect patient social risk factors. Identifying and assessing available data elements serves as a foundational step toward developing future computable social factor phenotypes.

Comparing automated vs. manual data collection for COVID-specific medications from electronic health records

INTRODUCTION

Data extraction from electronic health record (EHR) systems occurs through manual abstraction, automated extraction, or a combination of both. While each method has its strengths and weaknesses, both are necessary for retrospective observational research as well as sudden clinical events, like the COVID-19 pandemic. Assessing the strengths, weaknesses, and potentials of these methods is important to continue to understand optimal approaches to extracting clinical data. We set out to assess automated and manual techniques for collecting medication use data in patients with COVID-19 to inform future observational studies that extract data from the electronic health record (EHR).

MATERIALS AND METHODS

For 4,123 COVID-positive patients hospitalized and/or seen in the emergency department at an academic medical center between 03/03/2020 and 05/15/2020, we compared medication use data of 25 medications or drug classes collected through manual abstraction and automated extraction from the EHR. Quantitatively, we assessed concordance using Cohen's kappa to measure interrater reliability, and qualitatively, we audited observed discrepancies to determine causes of inconsistencies.

RESULTS

For the 16 inpatient medications, 11 (69%) demonstrated moderate or better agreement; 7 of those demonstrated strong or almost perfect agreement. For 9 outpatient medications, 3 (33%) demonstrated moderate agreement, but none achieved strong or almost perfect agreement. We audited 12% of all discrepancies (716/5,790) and, in those audited, observed three principal categories of error: human error in manual abstraction (26%), errors in the extract-transform-load (ETL) or mapping of the automated extraction (41%), and abstraction-query mismatch (33%).

CONCLUSION

Our findings suggest many inpatient medications can be collected reliably through automated extraction, especially when abstraction instructions are designed with data architecture in mind. We discuss quality issues, concerns, and improvements for institutions to consider when crafting an approach. During crises, institutions must decide how to allocate limited resources. We show that automated extraction of medications is feasible and make recommendations on how to improve future iterations.

An architecture for research computing in health to support clinical and translational investigators with electronic patient data

Objective

Obtaining electronic patient data, especially from electronic health record (EHR) systems, for clinical and translational research is difficult. Multiple research informatics systems exist but navigating the numerous applications can be challenging for scientists. This article describes Architecture for Research Computing in Health (ARCH), our institution’s approach for matching investigators with tools and services for obtaining electronic patient data.

Materials and Methods

Supporting the spectrum of studies from populations to individuals, ARCH delivers a breadth of scientific functions—including but not limited to cohort discovery, electronic data capture, and multi-institutional data sharing—that manifest in specific systems—such as i2b2, REDCap, and PCORnet. Through a consultative process, ARCH staff align investigators with tools with respect to study design, data sources, and cost. Although most ARCH services are available free of charge, advanced engagements require fee for service.

Results

Since 2016 at Weill Cornell Medicine, ARCH has supported over 1200 unique investigators through more than 4177 consultations. Notably, ARCH infrastructure enabled critical coronavirus disease 2019 response activities for research and patient care.

Discussion

ARCH has provided a technical, regulatory, financial, and educational framework to support the biomedical research enterprise with electronic patient data. Collaboration among informaticians, biostatisticians, and clinicians has been critical to rapid generation and analysis of EHR data.

Conclusion

A suite of tools and services, ARCH helps match investigators with informatics systems to reduce time to science. ARCH has facilitated research at Weill Cornell Medicine and may provide a model for informatics and research leaders to support scientists elsewhere.

Extracting social determinants of health from electronic health records using natural language processing: a systematic review

OBJECTIVE

Social determinants of health (SDoH) are nonclinical dispositions that impact patient health risks and clinical outcomes. Leveraging SDoH in clinical decision-making can potentially improve diagnosis, treatment planning, and patient outcomes. Despite increased interest in capturing SDoH in electronic health records (EHRs), such information is typically locked in unstructured clinical notes. Natural language processing (NLP) is the key technology to extract SDoH information from clinical text and expand its utility in patient care and research. This article presents a systematic review of the state-of-the-art NLP approaches and tools that focus on identifying and extracting SDoH data from unstructured clinical text in EHRs.

MATERIALS AND METHODS

A broad literature search was conducted in February 2021 using 3 scholarly databases (ACL Anthology, PubMed, and Scopus) following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. A total of 6402 publications were initially identified, and after applying the study inclusion criteria, 82 publications were selected for the final review.

RESULTS

Smoking status (n = 27), substance use (n = 21), homelessness (n = 20), and alcohol use (n = 15) are the most frequently studied SDoH categories. Homelessness (n = 7) and other less-studied SDoH (eg, education, financial problems, social isolation and support, family problems) are mostly identified using rule-based approaches. In contrast, machine learning approaches are popular for identifying smoking status (n = 13), substance use (n = 9), and alcohol use (n = 9).

CONCLUSION

NLP offers significant potential to extract SDoH data from narrative clinical notes, which in turn can aid in the development of screening tools, risk prediction models, and clinical decision support systems.

A Comparative Analysis of the Respiratory Subscore of the Sequential Organ Failure Assessment Scoring System

Rationale: The Sequential Organ Failure Assessment (SOFA) tool is a commonly used measure of illness severity. Calculation of the respiratory subscore of SOFA is frequently limited by missing arterial oxygen pressure (PaO2) data. Although missing PaO2 data are commonly replaced with normal values, the performance of different methods of substituting PaO2 for SOFA calculation is unclear. Objectives: The study objective was to compare the performance of different substitution strategies for missing PaO2 data for SOFA score calculation. Methods: This retrospective cohort study was performed using the Weill Cornell Critical Care Database for Advanced Research from a tertiary care hospital in the United States. All adult patients admitted to an intensive care unit (ICU) from 2011 to 2019 with an available respiratory SOFA score were included. We analyzed the availability of the PaO2/fraction of inspired oxygen (FiO2) ratio on the first day of ICU admission. In those without a PaO2/FiO2 ratio available, the ratio of oxygen saturation as measured by pulse oximetry to FiO2 was used to calculate a respiratory SOFA subscore according to four methods (linear substitution [Rice], nonlinear substitution [Severinghaus], modified respiratory SOFA, and multiple imputation by chained equations [MICE]) as well as the missing-as-normal technique. We then compared how well the different total SOFA scores discriminated in-hospital mortality. We performed several subgroup and sensitivity analyses. Results: We identified 35,260 unique visits, of which 9,172 included predominant respiratory failure. PaO2 data were available for 14,939 (47%). The area under the receiver operating characteristic curve for each substitution technique for discriminating in-hospital mortality was higher than that for the missing-as-normal technique (0.78 [0.77-0.79]) in all analyses (modified, 0.80 [0.79-0.81]; Rice, 0.80 [0.79-0.81]; Severinghaus, 0.80 [0.79-0.81]; and MICE, 0.80 [0.79-0.81]) (P < 0.01). Each substitution method had a higher accuracy for discriminating in-hospital mortality (MICE, 0.67; Rice, 0.67; modified, 0.66; and Severinghaus, 0.66) than the missing-as-normal technique. Model calibration for in-hospital mortality was less precise for the missing-as-normal technique than for the other substitution techniques at the lower range of SOFA and among the subgroups. Conclusions: Using physiologic and statistical substitution methods improved the total SOFA score's ability to discriminate mortality compared with the missing-as-normal technique. Treating missing data as normal may result in underreporting the severity of illness compared with using substitution. The simplicity of a direct oxygen saturation as measured by pulse oximetry/FiO2 ratio-modified SOFA technique makes it an attractive choice for electronic health record-based research. This knowledge can inform comparisons of severity of illness across studies that used different techniques.

Automated Extraction of Tumor Staging and Diagnosis Information From Surgical Pathology Reports

PURPOSE

Typically stored as unstructured notes, surgical pathology reports contain data elements valuable to cancer research that require labor-intensive manual extraction. Although studies have described natural language processing (NLP) of surgical pathology reports to automate information extraction, efforts have focused on specific cancer subtypes rather than across multiple oncologic domains. To address this gap, we developed and evaluated an NLP method to extract tumor staging and diagnosis information across multiple cancer subtypes.

METHODS

The NLP pipeline was implemented on an open-source framework called Leo. We used a total of 555,681 surgical pathology reports of 329,076 patients to develop the pipeline and evaluated our approach on subsets of reports from patients with breast, prostate, colorectal, and randomly selected cancer subtypes.

RESULTS

Averaged across all four cancer subtypes, the NLP pipeline achieved an accuracy of 1.00 for International Classification of Diseases, Tenth Revision codes, 0.89 for T staging, 0.90 for N staging, and 0.97 for M staging. It achieved an F1 score of 1.00 for International Classification of Diseases, Tenth Revision codes, 0.88 for T staging, 0.90 for N staging, and 0.24 for M staging.

CONCLUSION

The NLP pipeline was developed to extract tumor staging and diagnosis information across multiple cancer subtypes to support the research enterprise in our institution. Although it was not possible to demonstrate generalizability of our NLP pipeline to other institutions, other institutions may find value in adopting a similar NLP approach-and reusing code available at GitHub-to support the oncology research enterprise with elements extracted from surgical pathology reports.

Identifying organ dysfunction trajectory-based subphenotypes in critically ill patients with COVID-19

COVID-19-associated respiratory failure offers the unprecedented opportunity to evaluate the differential host response to a uniform pathogenic insult. Understanding whether there are distinct subphenotypes of severe COVID-19 may offer insight into its pathophysiology. Sequential Organ Failure Assessment (SOFA) score is an objective and comprehensive measurement that measures dysfunction severity of six organ systems, i.e., cardiovascular, central nervous system, coagulation, liver, renal, and respiration. Our aim was to identify and characterize distinct subphenotypes of COVID-19 critical illness defined by the post-intubation trajectory of SOFA score. Intubated COVID-19 patients at two hospitals in New York city were leveraged as development and validation cohorts. Patients were grouped into mild, intermediate, and severe strata by their baseline post-intubation SOFA. Hierarchical agglomerative clustering was performed within each stratum to detect subphenotypes based on similarities amongst SOFA score trajectories evaluated by Dynamic Time Warping. Distinct worsening and recovering subphenotypes were identified within each stratum, which had distinct 7-day post-intubation SOFA progression trends. Patients in the worsening suphenotypes had a higher mortality than those in the recovering subphenotypes within each stratum (mild stratum, 29.7% vs. 10.3%, p = 0.033; intermediate stratum, 29.3% vs. 8.0%, p = 0.002; severe stratum, 53.7% vs. 22.2%, p < 0.001). Pathophysiologic biomarkers associated with progression were distinct at each stratum, including findings suggestive of inflammation in low baseline severity of illness versus hemophagocytic lymphohistiocytosis in higher baseline severity of illness. The findings suggest that there are clear worsening and recovering subphenotypes of COVID-19 respiratory failure after intubation, which are more predictive of outcomes than baseline severity of illness. Distinct progression biomarkers at differential baseline severity of illness suggests a heterogeneous pathobiology in the progression of COVID-19 respiratory failure.

Critical carE Database for Advanced Research (CEDAR): An automated method to support intensive care units with electronic health record data

Patients treated in an intensive care unit (ICU) are critically ill and require life-sustaining organ failure support. Existing critical care data resources are limited to a select number of institutions, contain only ICU data, and do not enable the study of local changes in care patterns. To address these limitations, we developed the Critical carE Database for Advanced Research (CEDAR), a method for automating extraction and transformation of data from an electronic health record (EHR) system. Compared to an existing gold standard of manually collected data at our institution, CEDAR was statistically similar in most measures, including patient demographics and sepsis-related organ failure assessment (SOFA) scores. Additionally, CEDAR automated data extraction obviated the need for manual collection of 550 variables. Critically, during the spring 2020 COVID-19 surge in New York City, a modified version of CEDAR supported pandemic response efforts, including clinical operations and research. Other academic medical centers may find value in using the CEDAR method to automate data extraction from EHR systems to support ICU activities.

Shotgun transcriptome, spatial omics, and isothermal profiling of SARS-CoV-2 infection reveals unique host responses, viral diversification, and drug interactions

In less than nine months, the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) killed over a million people, including >25,000 in New York City (NYC) alone. The COVID-19 pandemic caused by SARS-CoV-2 highlights clinical needs to detect infection, track strain evolution, and identify biomarkers of disease course. To address these challenges, we designed a fast (30-minute) colorimetric test (LAMP) for SARS-CoV-2 infection from naso/oropharyngeal swabs and a large-scale shotgun metatranscriptomics platform (total-RNA-seq) for host, viral, and microbial profiling. We applied these methods to clinical specimens gathered from 669 patients in New York City during the first two months of the outbreak, yielding a broad molecular portrait of the emerging COVID-19 disease. We find significant enrichment of a NYC-distinctive clade of the virus (20C), as well as host responses in interferon, ACE, hematological, and olfaction pathways. In addition, we use 50,821 patient records to find that renin-angiotensin-aldosterone system inhibitors have a protective effect for severe COVID-19 outcomes, unlike similar drugs. Finally, spatial transcriptomic data from COVID-19 patient autopsy tissues reveal distinct ACE2 expression loci, with macrophage and neutrophil infiltration in the lungs. These findings can inform public health and may help develop and drive SARS-CoV-2 diagnostic, prevention, and treatment strategies.

Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation

Mental health concerns, such as suicidal thoughts, are frequently documented by providers in clinical notes, as opposed to structured coded data. In this study, we evaluated weakly supervised methods for detecting "current" suicidal ideation from unstructured clinical notes in electronic health record (EHR) systems. Weakly supervised machine learning methods leverage imperfect labels for training, alleviating the burden of creating a large manually annotated dataset. After identifying a cohort of 600 patients at risk for suicidal ideation, we used a rule-based natural language processing approach (NLP) approach to label the training and validation notes (n = 17,978). Using this large corpus of clinical notes, we trained several statistical machine learning models-logistic classifier, support vector machines (SVM), Naive Bayes classifier-and one deep learning model, namely a text classification convolutional neural network (CNN), to be evaluated on a manually-reviewed test set (n = 837). The CNN model outperformed all other methods, achieving an overall accuracy of 94% and a F1-score of 0.82 on documents with "current" suicidal ideation. This algorithm correctly identified an additional 42 encounters and 9 patients indicative of suicidal ideation but missing a structured diagnosis code. When applied to a random subset of 5,000 clinical notes, the algorithm classified 0.46% (n = 23) for "current" suicidal ideation, of which 87% were truly indicative via manual review. Implementation of this approach for large-scale document screening may play an important role in point-of-care clinical information systems for targeted suicide prevention interventions and improve research on the pathways from ideation to attempt.

Underserved populations with missing race ethnicity data differ significantly from those with structured race/ethnicity documentation

OBJECTIVE

We aimed to address deficiencies in structured electronic health record (EHR) data for race and ethnicity by identifying black and Hispanic patients from unstructured clinical notes and assessing differences between patients with or without structured race/ethnicity data.

MATERIALS AND METHODS

Using EHR notes for 16 665 patients with encounters at a primary care practice, we developed rule-based natural language processing (NLP) algorithms to classify patients as black/Hispanic. We evaluated performance of the method against an annotated gold standard, compared race and ethnicity between NLP-derived and structured EHR data, and compared characteristics of patients identified as black or Hispanic using only NLP vs patients identified as such only in structured EHR data.

RESULTS

For the sample of 16 665 patients, NLP identified 948 additional patients as black, a 26%increase, and 665 additional patients as Hispanic, a 20% increase. Compared with the patients identified as black or Hispanic in structured EHR data, patients identified as black or Hispanic via NLP only were older, more likely to be male, less likely to have commercial insurance, and more likely to have higher comorbidity.

DISCUSSION

Structured EHR data for race and ethnicity are subject to data quality issues. Supplementing structured EHR race data with NLP-derived race and ethnicity may allow researchers to better assess the demographic makeup of populations and draw more accurate conclusions about intergroup differences in health outcomes.

CONCLUSIONS

Black or Hispanic patients who are not documented as such in structured EHR race/ethnicity fields differ significantly from those who are. Relatively simple NLP can help address this limitation.

A Method to Improve Availability and Quality of Patient Race Data in an Electronic Health Record System

BACKGROUND

Although federal regulations mandate documentation of structured race data according to Office of Management and Budget (OMB) categories in electronic health record (EHR) systems, many institutions have reported gaps in EHR race data that hinder secondary use for population-level research focused on underserved populations. When evaluating race data available for research purposes, we found our institution's enterprise EHR contained structured race data for only 51% (1.6 million) of patients.

OBJECTIVES

We seek to improve the availability and quality of structured race data available to researchers by integrating values from multiple local sources.

METHODS

To address the deficiency in race data availability, we implemented a method to supplement OMB race values from four local sources-inpatient EHR, inpatient billing, natural language processing, and coded clinical observations. We evaluated this method by measuring race data availability and data quality with respect to completeness, concordance, and plausibility.

RESULTS

The supplementation method improved race data availability in the enterprise EHR up to 10% for some minority groups and 4% overall. We identified structured OMB race values for more than 142,000 patients, nearly a third of whom were from racial minority groups. Our data quality evaluation indicated that the supplemented race values improved completeness in the enterprise EHR, originated from sources in agreement with the enterprise EHR, and were unbiased to the enterprise EHR.

CONCLUSION

Implementation of this method can successfully increase OMB race data availability, potentially enhancing accrual of patients from underserved populations to research studies.

Brain Imaging of Patients with COVID-19: Findings at an Academic Institution during the Height of the Outbreak in New York City

BACKGROUND AND PURPOSE

A large spectrum of neurologic disease has been reported in patients with coronavirus disease 2019 (COVID-19) infection. Our aim was to investigate the yield of neuroimaging in patients with COVID-19 undergoing CT or MR imaging of the brain and to describe associated imaging findings.

MATERIALS AND METHODS

We performed a retrospective cohort study involving 2054 patients with laboratory-confirmed COVID-19 presenting to 2 hospitals in New York City between March 4 and May 9, 2020, of whom 278 (14%) underwent either CT or MR imaging of the brain. All images initially received a formal interpretation from a neuroradiologist within the institution and were subsequently reviewed by 2 neuroradiologists in consensus, with disputes resolved by a third neuroradiologist.

RESULTS

The median age of these patients was 64 years (interquartile range, 50-75 years), and 43% were women. Among imaged patients, 58 (21%) demonstrated acute or subacute neuroimaging findings, the most common including cerebral infarctions (11%), parenchymal hematomas (3.6%), and posterior reversible encephalopathy syndrome (1.1%). Among the 51 patients with MR imaging examinations, 26 (51%) demonstrated acute or subacute findings; notable findings included 6 cases of cranial nerve abnormalities (including 4 patients with olfactory bulb abnormalities) and 3 patients with a microhemorrhage pattern compatible with critical illness-associated microbleeds.

CONCLUSIONS

Our experience confirms the wide range of neurologic imaging findings in patients with COVID-19 and suggests the need for further studies to optimize management for these patients.

Extracting and classifying diagnosis dates from clinical notes: A case study

Myeloproliferative neoplasms (MPNs) are chronic hematologic malignancies that may progress over long disease courses. The original date of diagnosis is an important piece of information for patient care and research, but is not consistently documented. We describe an attempt to build a pipeline for extracting dates with natural language processing (NLP) tools and techniques and classifying them as relevant diagnoses or not. Inaccurate and incomplete date extraction and interpretation impacted the performance of the overall pipeline. Existing lightweight Python packages tended to have low specificity for identifying and interpreting partial and relative dates in clinical text. A rules-based regular expression (regex) approach achieved recall of 83.0% on dates manually annotated as diagnosis dates, and 77.4% on all annotated dates. With only 3.8% of annotated dates representing initial MPN diagnoses, additional methods of targeting candidate date instances may alleviate noise and class imbalance.

Risk of Ischemic Stroke in Patients With Coronavirus Disease 2019 (COVID-19) vs Patients With Influenza

IMPORTANCE

It is uncertain whether coronavirus disease 2019 (COVID-19) is associated with a higher risk of ischemic stroke than would be expected from a viral respiratory infection.

OBJECTIVE

To compare the rate of ischemic stroke between patients with COVID-19 and patients with influenza, a respiratory viral illness previously associated with stroke.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study was conducted at 2 academic hospitals in New York City, New York, and included adult patients with emergency department visits or hospitalizations with COVID-19 from March 4, 2020, through May 2, 2020. The comparison cohort included adults with emergency department visits or hospitalizations with influenza A/B from January 1, 2016, through May 31, 2018 (spanning moderate and severe influenza seasons).

EXPOSURES

COVID-19 infection confirmed by evidence of severe acute respiratory syndrome coronavirus 2 in the nasopharynx by polymerase chain reaction and laboratory-confirmed influenza A/B.

MAIN OUTCOMES AND MEASURES

A panel of neurologists adjudicated the primary outcome of acute ischemic stroke and its clinical characteristics, mechanisms, and outcomes. We used logistic regression to compare the proportion of patients with COVID-19 with ischemic stroke vs the proportion among patients with influenza.

RESULTS

Among 1916 patients with emergency department visits or hospitalizations with COVID-19, 31 (1.6%; 95% CI, 1.1%-2.3%) had an acute ischemic stroke. The median age of patients with stroke was 69 years (interquartile range, 66-78 years); 18 (58%) were men. Stroke was the reason for hospital presentation in 8 cases (26%). In comparison, 3 of 1486 patients with influenza (0.2%; 95% CI, 0.0%-0.6%) had an acute ischemic stroke. After adjustment for age, sex, and race, the likelihood of stroke was higher with COVID-19 infection than with influenza infection (odds ratio, 7.6; 95% CI, 2.3-25.2). The association persisted across sensitivity analyses adjusting for vascular risk factors, viral symptomatology, and intensive care unit admission.

CONCLUSIONS AND RELEVANCE

In this retrospective cohort study from 2 New York City academic hospitals, approximately 1.6% of adults with COVID-19 who visited the emergency department or were hospitalized experienced ischemic stroke, a higher rate of stroke compared with a cohort of patients with influenza. Additional studies are needed to confirm these findings and to investigate possible thrombotic mechanisms associated with COVID-19.

Understanding enterprise data warehouses to support clinical and translational research

OBJECTIVE

Among National Institutes of Health Clinical and Translational Science Award (CTSA) hubs, adoption of electronic data warehouses for research (EDW4R) containing data from electronic health record systems is nearly ubiquitous. Although benefits of EDW4R include more effective, efficient support of scientists, little is known about how CTSA hubs have implemented EDW4R services. The goal of this qualitative study was to understand the ways in which CTSA hubs have operationalized EDW4R to support clinical and translational researchers.

MATERIALS AND METHODS

After conducting semistructured interviews with informatics leaders from 20 CTSA hubs, we performed a directed content analysis of interview notes informed by naturalistic inquiry.

RESULTS

We identified 12 themes: organization and data; oversight and governance; data access request process; data access modalities; data access for users with different skill sets; engagement, communication, and literacy; service management coordinated with enterprise information technology; service management coordinated within a CTSA hub; service management coordinated between informatics and biostatistics; funding approaches; performance metrics; and future trends and current technology challenges.

DISCUSSION

This study is a step in developing an improved understanding and creating a common vocabulary about EDW4R operations across institutions. Findings indicate an opportunity for establishing best practices for EDW4R operations in academic medicine. Such guidance could reduce the costs associated with developing an EDW4R by establishing a clear roadmap and maturity path for institutions to follow.

CONCLUSIONS

CTSA hubs described varying approaches to EDW4R operations that may assist other institutions in better serving investigators with electronic patient data.

Replacing Paper Informed Consent with Electronic Informed Consent for Research in Academic Medical Centers: A Scoping Review

Although experts have identified benefits to replacing paper with electronic consent (eConsent) for research, a comprehensive understanding of strategies to overcome barriers to adoption is unknown. To address this gap, we performed a scoping review of the literature describing eConsent in academic medical centers. Of 69 studies that met inclusion criteria, 81% (n=56) addressed ethical, legal, and social issues; 67% (n=46) described user interface/user experience considerations; 39% (n=27) compared electronic versus paper approaches; 33% (n=23) discussed approaches to enterprise scalability; and 25% (n=17) described changes to consent elections. Findings indicate a lack of a leading commercial eConsent vendor, as articles described a myriad of homegrown systems and extensions of vendor EHR patient portals. Opportunities appear to exist for researchers and commercial software vendors to develop eConsent approaches that address the five critical areas identified in this review.

Characterizing Basic and Complex Usage of i2b2 at an Academic Medical Center

Developed to enable basic queries for cohort discovery, i2b2 has evolved to support complex queries. Little is known whether query sophistication - and the informatics resources required to support it - addresses researcher needs. In three years at our institution, 609 researchers ran 6,662 queries and requested re-identification of 80 patient cohorts to support specific studies. After characterizing all queries as "basic" or "complex" with respect to use of sophisticated query features, we found that the majority of all queries, and the majority of queries resulting in a request for cohort re-identification, did not use complex i2b2 features. Data domains that required extensive effort to implement saw relatively little use compared to common domains (e.g., diagnoses). These findings suggest that efforts to ensure the performance of basic queries using common data domains may better serve the needs of the research community than efforts to integrate novel domains or introduce complex new features.

Mapping Local Biospecimen Records to the OMOP Common Data Model

Research to support precision medicine for leukemia patients requires integration of biospecimen and clinical data. The Observational Medical Outcomes Partnership common data model (OMOP CDM) and its Specimen table presents a potential solution. Although researchers have described progress and challenges in mapping electronic health record (EHR) data to populate the OMOP CDM, to our knowledge no studies have described populating the OMOP CDM with biospecimen data. Using biobank data from our institution, we mapped 26% of biospecimen records to the OMOP Specimen table. Records failed mapping due to local codes for time point that were incompatible with the OMOP reference terminology. We recommend expanding allowable codes to encompass research data, adding foreign keys to leverage additional OMOP tables with data from other sources or to store additional specimen details, and considering a new table to represent processed samples and inventory.

Risk of Ischemic Stroke in Patients with Covid-19 versus Patients with Influenza

Importance:

Case series without control groups suggest that Covid-19 may cause ischemic stroke, but whether Covid-19 is associated with a higher risk of ischemic stroke than would be expected from a viral respiratory infection is uncertain.

Objective:

To compare the rate of ischemic stroke between patients with Covid-19 and patients with influenza, a respiratory viral illness previously linked to stroke.

Design:

A retrospective cohort study.

Setting:

Two academic hospitals in New York City.

Participants:

We included adult patients with emergency department visits or hospitalizations with Covid-19 from March 4, 2020 through May 2, 2020. Our comparison cohort included adult patients with emergency department visits or hospitalizations with influenza A or B from January 1, 2016 through May 31, 2018 (calendar years spanning moderate and severe influenza seasons).

Exposures:

Covid-19 infection confirmed by evidence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in the nasopharynx by polymerase chain reaction, and laboratory-confirmed influenza A or B.

Main Outcomes and Measures:

A panel of neurologists adjudicated the primary outcome of acute ischemic stroke and its clinical characteristics, etiological mechanisms, and outcomes. We used logistic regression to compare the proportion of Covid-19 patients with ischemic stroke versus the proportion among patients with influenza.

Results:

Among 2,132 patients with emergency department visits or hospitalizations with Covid-19, 31 patients (1.5%; 95% confidence interval [CI], 1.0%-2.1%) had an acute ischemic stroke. The median age of patients with stroke was 69 years (interquartile range, 66-78) and 58% were men. Stroke was the reason for hospital presentation in 8 (26%) cases. For our comparison cohort, we identified 1,516 patients with influenza, of whom 0.2% (95% CI, 0.0-0.6%) had an acute ischemic stroke. After adjustment for age, sex, and race, the likelihood of stroke was significantly higher with Covid-19 than with influenza infection (odds ratio, 7.5; 95% CI, 2.3-24.9).

Conclusions and Relevance:

Approximately 1.5% of patients with emergency department visits or hospitalizations with Covid-19 experienced ischemic stroke, a rate 7.5-fold higher than in patients with influenza. Future studies should investigate the thrombotic mechanisms in Covid-19 in order to determine optimal strategies to prevent disabling complications like ischemic stroke.

Evaluating the Portability of an NLP System for Processing Echocardiograms: A Retrospective, Multi-site Observational Study

While natural language processing (NLP) of unstructured clinical narratives holds the potential for patient care and clinical research, portability of NLP approaches across multiple sites remains a major challenge. This study investigated the portability of an NLP system developed initially at the Department of Veterans Affairs (VA) to extract 27 key cardiac concepts from free-text or semi-structured echocardiograms from three academic edical centers: Weill Cornell Medicine, Mayo Clinic and Northwestern Medicine. While the NLP system showed high precision and recall easurements for four target concepts (aortic valve regurgitation, left atrium size at end systole, mitral valve regurgitation, tricuspid valve regurgitation) across all sites, we found moderate or poor results for the remaining concepts and the NLP system performance varied between individual sites.

Relationship between left atrial volume and ischemic stroke subtype

OBJECTIVE

Atrial cardiopathy without atrial fibrillation (AF) may be a potential cardiac source of embolic strokes of undetermined source (ESUS). Atrial volume is a feature of atrial cardiopathy, but the relationship between atrial volume and ESUS remains unclear.

METHODS

We compared left atrial volume among ischemic stroke subtypes in the Cornell Acute Stroke Academic Registry (CAESAR), which includes all patients with acute ischemic stroke at our hospital since 2011. Stroke subtype was determined by neurologists per the TOAST classification and consensus ESUS definition. Left atrial volume index (LAVI) was obtained directly from our echocardiography image system (Xcelera, Philips Healthcare). We used t-tests and analysis of variance for unadjusted comparisons and targeted minimum loss-based estimation for comparisons adjusted for demographics and comorbidities.

RESULTS

Among 2116 patients in CAESAR from 2011 to 2016, 1293 had LAVI measurements. LAVI varied across subtypes (P < 0.001) from 48.8 (±30.0) mL/m² in cardioembolic strokes to 30.3 (±10.5) mL/m² in small-vessel strokes. LAVI was larger in ESUS (33.3 ± 13.6 mL/m² ) than in small- or large-vessel stroke (30.9 ± 10.7 mL/m² ) (P = 0.01). The association between LAVI and ESUS persisted after the adjustment for demographics and comorbidities: a 10 mL/m² increase in LAVI was associated with a 4.4% increase in ESUS probability (95% CI, 2.3%-6.4%). Results were similar after excluding patients with AF during post-discharge heart-rhythm monitoring.

INTERPRETATION

We found larger left atria among patients with ESUS versus non-cardioembolic stroke. There was significant overlap in left atrial size between ESUS and non-cardioembolic stroke, highlighting that many ESUS cases are not cardioembolic.

Evaluation of a REDCap-based Workflow for Supporting Federal Guidance for Electronic Informed Consent

Adoption of electronic informed consent (eConsent) for research remains low despite evidence of improved patient comprehension, usability, and workflow processes compared to paper. At our institution, we implemented an eConsent workflow using REDCap, a widely used electronic data capture system. The goal of this study was to evaluate the extent to which the REDCap eConsent solution adhered to federal guidance for eConsent. Of 29 requirements derived from sixteen recommendations from the United States Office for Human Research Protections (OHRP) and Food and Drug Administration (FDA), the REDCap eConsent solution supported 24 (86%). To the best of our knowledge, this is among the first studies to evaluate an eConsent approach's support for federal guidance. Findings suggest use of REDCap may help other institutions overcome barriers to eConsent adoption, and that OHRP and FDA expand guidance to recommend eConsent solutions integrate with enterprise clinical and research information systems.

A Method for Integrating Healthcare Provider Organization and Research Sponsor Systems and Workflows to Support Large-Scale Studies

Healthcare provider organizations (HPOs) increasingly participate in large-scale research efforts sponsored by external organizations that require use of consent management systems that may not integrate seamlessly with local workflows. The resulting inefficiency can hinder the ability of HPOs to participate in studies. To overcome this challenge, we developed a method using REDCap, a widely adopted electronic data capture system, and novel middleware that can potentially generalize to other settings. In this paper, we describe the method, illustrate its use to support the NIHAll of Us Research Program and PCORI ADAPTABLE studies at our HPO, and encourage other HPOs to test replicability of the method to facilitate similar research efforts. Code is available on GitHub at https://github.com/wcmc-research-informatics/.

Implementation of Informatics to Support the NIH All of Us Research Program in a Healthcare Provider Organization

The NIH All of Us Research Program, a national effort to collect biospecimens and health data for over one million participants from across the United States, requires participating healthcare provider organizations (HPOs) to use informatics tools maintained by the NIH to manage participant consent, biospecimen processing, physical measurements, and other workflows. HPOs also maintain distinct workflows for handling overlapping tasks within their individual aegis, which do not necessarily achieve seamless interoperability with NIH-maintained cloud-based systems. At our HPO, we implemented informatics to address gaps in enrollment workflows and hardware, clinical workflow integration, patient engagement, laboratory support, and study team reporting. In this case report we detail our approach to inform efforts at other institutions for the NIH All of Us Research Program and other studies.

Ascertaining Depression Severity by Extracting Patient Health Questionnaire-9 (PHQ-9) Scores from Clinical Notes

The Patient Health Questionnaire-9 (PHQ-9) is a validated instrument for assessing depression severity. While some electronic health record (EHR) systems capture PHQ-9 scores in a structured format, unstructured clinical notes remain the only source in many settings, which presents data retrieval challenges for research and clinical decision support. To address this gap, we extended the open-source Leo natural language processing (NLP) platform to extract PHQ-9 scores from clinical notes and evaluated performance using EHR data for n=123,703 patients who were prescribed antidepressants. Compared to a reference standard, the NLP method exhibited high accuracy (97%), sensitivity (98%), precision (97%), and F-score (97%). Furthermore, of patients with PHQ-9 scores identified by the NLP method, 31% (n=498) had at least one PHQ-9 score clinically indicative of major depressive disorder (MDD), but lacked a structured ICD-9/10 diagnosis code for MDD. This NLP technique may facilitate accurate identification and stratification of patients with depression.

Design and Implementation of a Secure Computing Environment for Analysis of Sensitive Data at an Academic Medical Center

Academic medical centers need to make sensitive data from electronic health records, payer claims, genomic pipelines, and other sources available for analytical and educational purposes while ensuring privacy and security. Although many studies have described warehouses for collecting biomedical data, few studies have described secure computing environments for analysis of sensitive data. This case report describes the Weill Cornell Medicine Data Core with respect to user access, data controls, hardware, software, audit, and financial considerations. In the 2.5 years since launch, the Data Core has supported more than 200 faculty, staff, and students across nearly 60 research and education projects. Other institutions may benefit from adopting elements of the approach, including tools available on Github, for balancing access with privacy and security.

A case study evaluating the portability of an executable computable phenotype algorithm across multiple institutions and electronic health record environments

Electronic health record (EHR) algorithms for defining patient cohorts are commonly shared as free-text descriptions that require human intervention both to interpret and implement. We developed the Phenotype Execution and Modeling Architecture (PhEMA, http://projectphema.org) to author and execute standardized computable phenotype algorithms. With PhEMA, we converted an algorithm for benign prostatic hyperplasia, developed for the electronic Medical Records and Genomics network (eMERGE), into a standards-based computable format. Eight sites (7 within eMERGE) received the computable algorithm, and 6 successfully executed it against local data warehouses and/or i2b2 instances. Blinded random chart review of cases selected by the computable algorithm shows PPV ≥90%, and 3 out of 5 sites had >90% overlap of selected cases when comparing the computable algorithm to their original eMERGE implementation. This case study demonstrates potential use of PhEMA computable representations to automate phenotyping across different EHR systems, but also highlights some ongoing challenges.

Lessons Learned in the Development of a Computable Phenotype for Response in Myeloproliferative Neoplasms

Determining response status in patients with myeloproliferative neoplasms is a complex problem requiring the integration of both structured and unstructured data elements from disparate information systems. By applying multiple techniques, a collaborative team of informatics professionals and research personnel were able to determine which elements were amenable to automated extraction and which required expert adjudication. With this knowledge in mind, we were able to build a system that joins together programmatically-derived and manually-abstracted data elements to facilitate response assessment - an important end point in clinical and translational research in this disease area.

A scalable method for supporting multiple patient cohort discovery projects using i2b2

Although i2b2, a popular platform for patient cohort discovery using electronic health record (EHR) data, can support multiple projects specific to individual disease areas or research interests, the standard approach for doing so duplicates data across projects, requiring additional disk space and processing time, which limits scalability. To address this deficiency, we developed a novel approach that stored data in a single i2b2 fact table and used structured query language (SQL) views to access data for specific projects. Compared to the standard approach, the view-based approach reduced required disk space by 59% and extract-transfer-load (ETL) time by 46%, without substantially impacting query performance. The view-based approach has enabled scalability of multiple i2b2 projects and generalized to another data model at our institution. Other institutions may benefit from this approach, code of which is available on GitHub (https://github.com/wcmc-research-informatics/super-i2b2).

From Sour Grapes to Low-Hanging Fruit: A Case Study Demonstrating a Practical Strategy for Natural Language Processing Portability

Natural Language Processing (NLP) holds potential for patient care and clinical research, but a gap exists between promise and reality. While some studies have demonstrated portability of NLP systems across multiple sites, challenges remain. Strategies to mitigate these challenges can strive for complex NLP problems using advanced methods (hard-to-reach fruit), or focus on simple NLP problems using practical methods (low-hanging fruit). This paper investigates a practical strategy for NLP portability using extraction of left ventricular ejection fraction (LVEF) as a use case. We used a tool developed at the Department of Veterans Affair (VA) to extract the LVEF values from free-text echocardiograms in the MIMIC-III database. The approach showed an accuracy of 98.4%, sensitivity of 99.4%, a positive predictive value of 98.7%, and F-score of 99.0%. This experience, in which a simple NLP solution proved highly portable with excellent performance, illustrates the point that simple NLP applications may be easier to disseminate and adapt, and in the short term may prove more useful, than complex applications.

Evaluating Generalizability of a Biospecimen Informatics Approach: Support for Local Requirements and Best Practices

To enable clinical and translational research, academic medical centers increasingly implement biospecimen information management systems. At our institution, one laboratory successfully implemented a multi-system solution that enabled collection and reporting of specimen- and aliquot-level data. The objective of this study was to assess the solution against the laboratory's requirements and with respect to support of best practices for biospecimen information management systems defined by the International Society for Biological and Environmental Repositories (ISBER). The solution supported the laboratory's reporting needs and 90% (n=26) of ISBER best practices. To the best of our knowledge, this is among the first studies to demonstrate the generalizability of a biospecimen informatics approach. Findings suggest that development and evaluation of biospecimen informatics approaches can potentially improve through closer collaboration of informatics and biorepository professional societies.

Secondary Use of Patients' Electronic Records (SUPER): An Approach for Meeting Specific Data Needs of Clinical and Translational Researchers

Academic medical centers commonly approach secondary use of electronic health record (EHR) data by implementing centralized clinical data warehouses (CDWs). However, CDWs require extensive resources to model data dimensions and harmonize clinical terminology, which can hinder effective support of the specific and varied data needs of investigators. We hypothesized that an approach that aggregates raw data from source systems, ignores initial modeling typical of CDWs, and transforms raw data for specific research purposes would meet investigator needs. The approach has successfully enabled multiple tools that provide utility to the institutional research enterprise. To our knowledge, this is the first complete description of a methodology for electronic patient data acquisition and provisioning that ignores data harmonization at the time of initial storage in favor of downstream transformation to address specific research questions and applications.

Evaluation of Computerized Free Text Sign-Out Notes: Baseline Understanding and Recommendations

BACKGROUND: Standardization of sign-out, the transfer of patient information and responsibility between inpatient providers at shift change, is a Joint Commission National Patient Safety Goal intended to improve communication and reduce risk of error. Computerized systems with free text data entry and limited structure allow clinicians to generate sign-out notes in a variety of ways. OBJECTIVES: The literature lacks a systematic exploration of the range of content generated by users of computerized sign-out systems. The goal of this study was to determine if and how clinicians record standardized sign-out information using a system with free text data entry and limited structure. METHODS: Using qualitative methods, we reviewed free text sign-out notes for 730 patient cases across 39 hospital units at an academic medical center. RESULTS: Two categories of information expression emerged from analysis: patient treatment-comprised of patient summaries, awareness items, and action items-and care team coordination-consisting of discharge information, contact information, and social concerns. A third category describing the format of sign-out note content, presentation of information, also emerged. Location and structure of information varied, but sign-out note content for some hospital units exhibited specific characteristics and was relatively standardized. CONCLUSIONS: Findings provide a baseline understanding of computerized free text sign-out note content. Sign-out notes contained a synthesis of data from disparate sources. We recommend formalizing existing unit-specific content standardization and system use patterns to reduce sign-out note variability and improve communication.

Generalizable Middleware to Support Use of REDCap Dynamic Data Pull for Integrating Clinical and Research Data

To support integration of clinical and research data, the makers of REDCap, a widely-used electronic data capture system, released the Dynamic Data Pull (DDP) module. Although DDP is a standard module in REDCap, institutions must develop custom middleware web services to exchange data between REDCap and local source systems. The lack of middleware is a barrier to institutional adoption and use by investigators. To overcome this gap, we developed a REDCap DDP web service middleware (accessible at https://github.com/wcmc-research-informatics/redcap-ddp) that minimizes developer effort, relies on configuration by non-developers, and can generalize to other settings. Early findings suggest the approach is successful.

Adoption of clinical data exchange in community settings: a comparison of two approaches

Adoption of electronic clinical data exchange (CDE) across disparate healthcare organizations remains low in community settings despite demonstrated benefits. To expand CDE in communities, New York State funded sixteen community-based organizations to implement point-to-point directed exchange (n=8) and multi-site query-based health information exchange (HIE) (n=8). We conducted a cross-sectional study to compare adoption of directed exchange versus query-based HIE. From 2008 to 2011, 66% (n=1,747) of providers targeted for directed exchange and 21% (n=5,427) of providers targeted for query-based HIE adopted CDE. Funding per provider adoptee was almost two times greater for directed exchange (median (interquartile range): $25,535 ($17,391-$42,240)) than query-based HIE ($14,649 ($9,897-$28,078)), although the difference was not statistically significant. Because its infrastructure can cover larger populations using similar levels of public funding, query-based HIE may scale more broadly than directed exchange. To our knowledge, this is among the first studies to compare directed exchange versus query-based HIE.