PheValuator: Development and evaluation of a phenotype algorithm evaluator

Adopting a Framework for Rapid Real-World Data Analyses in Safety Signal Assessment

The expanding availability of real-world data (RWD) has led to an increase in both the interest and possibilities for using this information in postmarketing safety analyses and signal management. While there is enormous potential value from the safety insights generated through RWD, the analysis preparation, execution, and communication required to reliably deliver the evidence can be time consuming. Since the safety signal assessment process is a regulated and timebound process, any supporting RWD analyses require a rapid turnaround of well-designed and informative results. To address this challenge, a TransCelerate BioPharma working group was formed and developed a framework to help teams responsible for safety signal assessment overcome the challenges of working with RWD rapidly to deliver analyses within regulatory timelines. Here, a previously performed safety assessment was evaluated within the context of the developed framework to illustrate how the framework may be adopted in practice.

Similar Risk of Kidney Failure among Patients with Blinding Diseases Who Receive Ranibizumab, Aflibercept, and Bevacizumab: An Observational Health Data Sciences and Informatics Network Study

PURPOSE

To characterize the incidence of kidney failure associated with intravitreal anti-VEGF exposure; and compare the risk of kidney failure in patients treated with ranibizumab, aflibercept, or bevacizumab.

DESIGN

Retrospective cohort study across 12 databases in the Observational Health Data Sciences and Informatics (OHDSI) network.

SUBJECTS

Subjects aged ≥ 18 years with ≥ 3 monthly intravitreal anti-VEGF medications for a blinding disease (diabetic retinopathy, diabetic macular edema, exudative age-related macular degeneration, or retinal vein occlusion).

METHODS

The standardized incidence proportions and rates of kidney failure while on treatment with anti-VEGF were calculated. For each comparison (e.g., aflibercept versus ranibizumab), patients from each group were matched 1:1 using propensity scores. Cox proportional hazards models were used to estimate the risk of kidney failure while on treatment. A random effects meta-analysis was performed to combine each database's hazard ratio (HR) estimate into a single network-wide estimate.

MAIN OUTCOME MEASURES

Incidence of kidney failure while on anti-VEGF treatment, and time from cohort entry to kidney failure.

RESULTS

Of the 6.1 million patients with blinding diseases, 37 189 who received ranibizumab, 39 447 aflibercept, and 163 611 bevacizumab were included; the total treatment exposure time was 161 724 person-years. The average standardized incidence proportion of kidney failure was 678 per 100 000 persons (range, 0-2389), and incidence rate 742 per 100 000 person-years (range, 0-2661). The meta-analysis HR of kidney failure comparing aflibercept with ranibizumab was 1.01 (95% confidence interval [CI], 0.70-1.47; P = 0.45), ranibizumab with bevacizumab 0.95 (95% CI, 0.68-1.32; P = 0.62), and aflibercept with bevacizumab 0.95 (95% CI, 0.65-1.39; P = 0.60).

CONCLUSIONS

There was no substantially different relative risk of kidney failure between those who received ranibizumab, bevacizumab, or aflibercept. Practicing ophthalmologists and nephrologists should be aware of the risk of kidney failure among patients receiving intravitreal anti-VEGF medications and that there is little empirical evidence to preferentially choose among the specific intravitreal anti-VEGF agents.

FINANCIAL DISCLOSURES

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Harnessing EHR data for health research

With the increasing availability of rich, longitudinal, real-world clinical data recorded in electronic health records (EHRs) for millions of patients, there is a growing interest in leveraging these records to improve the understanding of human health and disease and translate these insights into clinical applications. However, there is also a need to consider the limitations of these data due to various biases and to understand the impact of missing information. Recognizing and addressing these limitations can inform the design and interpretation of EHR-based informatics studies that avoid confusing or incorrect conclusions, particularly when applied to population or precision medicine. Here we discuss key considerations in the design, implementation and interpretation of EHR-based informatics studies, drawing from examples in the literature across hypothesis generation, hypothesis testing and machine learning applications. We outline the growing opportunities for EHR-based informatics studies, including association studies and predictive modeling, enabled by evolving AI capabilities-while addressing limitations and potential pitfalls to avoid.

Semi-supervised ROC analysis for reliable and streamlined evaluation of phenotyping algorithms

OBJECTIVE

High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (eg, sensitivity, specificity).

MATERIALS AND METHODS

ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC with synthetic, semi-synthetic, and EHR data from Mass General Brigham (MGB).

RESULTS

ssROC produced ROC parameter estimates with minimal bias and significantly lower variance than supROC in the simulated and semi-synthetic data. For the 5 PAs from MGB, the estimates from ssROC are 30% to 60% less variable than supROC on average.

DISCUSSION

ssROC enables precise evaluation of PA performance without demanding large volumes of labeled data. ssROC is also easily implementable in open-source R software.

CONCLUSION

When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.

Comparing broad and narrow phenotype algorithms: differences in performance characteristics and immortal time incurred

Introduction: When developing phenotype algorithms for observational research, there is usually a trade-off between definitions that are sensitive or specific. The objective of this study was to estimate the performance characteristics of phenotype algorithms designed for increasing specificity and to estimate the immortal time associated with each algorithm. Materials and methods: We examined algorithms for 11 chronic health conditions. The analyses were from data from five databases. For each health condition, we created five algorithms to examine performance (sensitivity and positive predictive value (PPV)) differences: one broad algorithm using a single code for the health condition and four narrow algorithms where a second diagnosis code was required 1-30 days, 1-90 days, 1-365 days, or 1- all days in a subject's continuous observation period after the first code. We also examined the proportion of immortal time relative to time-at-risk (TAR) for four outcomes. The TAR's were: 0-30 days after the first condition occurrence (the index date), 0-90 days post-index, 0-365 days post-index, and 0-1,095 days post-index. Performance of algorithms for chronic health conditions was estimated using PheValuator (V2.1.4) from the OHDSI toolstack. Immortal time was calculated as the time from the index date until the first of the following: 1) the outcome; 2) the end of the outcome TAR; 3) the occurrence of the second code for the chronic health condition. Results: In the first analysis, the narrow phenotype algorithms, i.e., those requiring a second condition code, produced higher estimates for PPV and lower estimates for sensitivity compared to the single code algorithm. In all conditions, increasing the time to the required second code increased the sensitivity of the algorithm. In the second analysis, the amount of immortal time increased as the window used to identify the second diagnosis code increased. The proportion of TAR that was immortal was highest in the 30 days TAR analyses compared to the 1,095 days TAR analyses. Conclusion: Attempting to increase the specificity of a health condition algorithm by adding a second code is a potentially valid approach to increase specificity, albeit at the cost of incurring immortal time.

Clinical evaluation of code-based algorithms to identify patients with pulmonary arterial hypertension in healthcare databases

Pulmonary arterial hypertension (PAH) is a rare subgroup of pulmonary hypertension (PH). Claims and administrative databases can be particularly important for research in rare diseases; however, there is a lack of validated algorithms to identify PAH patients using administrative codes. We aimed to measure the accuracy of code-based PAH algorithms against the true clinical diagnosis by right heart catheterization (RHC). This study evaluated algorithms in patients who were recorded in two linkable data assets: the Stanford Healthcare administrative electronic health record database and the Stanford Vera Moulton Wall Center clinical PH database (which records each patient's RHC diagnosis). We assessed the sensitivity and specificity achieved by 16 algorithms (six published). In total, 720 PH patients with linked data available were included and 558 (78%) of these were PAH patients. Algorithms consisting solely of a P(A)H-specific diagnostic code classed all or almost all PH patients as PAH (sensitivity >97%, specificity <12%) while multicomponent algorithms with well-defined temporal sequences of procedure, diagnosis and treatment codes achieved a better balance of sensitivity and specificity. Specificity increased and sensitivity decreased with increasing algorithm complexity. The best-performing algorithms, in terms of fewest misclassified patients, included multiple components (e.g., PH diagnosis, PAH treatment, continuous enrollment for ≥6 months before and ≥12 months following index date) and achieved sensitivities and specificities of around 95% and 38%, respectively. Our findings help researchers tailor their choice and design of code-based PAH algorithms to their research question and demonstrate the importance of including well-defined temporal components in the algorithms.

State-Level Hypertension Prevalence and Control Among Adults in the U.S

INTRODUCTION

Improving hypertension control is a national priority. Electronic health record data have the potential to augment traditional surveillance systems. This study aimed to assess hypertension prevalence and control at the state level using a previously established electronic health record-based phenotype for hypertension.

METHODS

Adult patients (N=11,031,368) were included from the IQVIA ambulatory electronic medical record-U.S. 2019 data set. IQVIA ambulatory electronic medical record comprises electronic health records from >100,000 providers and includes patients from every U.S. state and Washington DC. Authors compared hypertension prevalence and control estimates against those from the Behavioral Risk Factor Surveillance System 2019. Results were age-standardized and stratified by state and sociodemographic characteristics. Statistical analyses were conducted in 2022-2023.

RESULTS

IQVIA ambulatory electronic medical record-U.S. patients had a median age of 55 years, and 56.7% were women. Overall age-standardized hypertension prevalence was higher in IQVIA ambulatory electronic medical record-U.S. (35.0%) than in the Behavioral Risk Factor Surveillance System (29.7%), however, state-level geographic patterns were similar, with the highest burden in the South and Appalachia. Similar patterns were also observed by sociodemographic characteristics in both data sets: hypertension prevalence was higher in older age groups (than younger), men (than women), and Black patients (than other races). Hypertension control varied widely across states: among states with >1% data coverage, control rates were lowest in Nevada (51.1%), Washington DC (52.0%), and Mississippi (55.2%); highest in Kansas (73.4%), New Jersey (72.3%), and Iowa (71.9%).

CONCLUSIONS

This study provided the first-ever estimates of hypertension control for all states and Washington DC. Electronic health record-based surveillance could support hypertension prevention and control efforts at the state level.

Scalable and interpretable alternative to chart review for phenotype evaluation using standardized structured data from electronic health records

OBJECTIVES

Chart review as the current gold standard for phenotype evaluation cannot support observational research on electronic health records and claims data sources at scale. We aimed to evaluate the ability of structured data to support efficient and interpretable phenotype evaluation as an alternative to chart review.

MATERIALS AND METHODS

We developed Knowledge-Enhanced Electronic Profile Review (KEEPER) as a phenotype evaluation tool that extracts patient's structured data elements relevant to a phenotype and presents them in a standardized fashion following clinical reasoning principles. We evaluated its performance (interrater agreement, intermethod agreement, accuracy, and review time) compared to manual chart review for 4 conditions using randomized 2-period, 2-sequence crossover design.

RESULTS

Case ascertainment with KEEPER was twice as fast compared to manual chart review. 88.1% of the patients were classified concordantly using charts and KEEPER, but agreement varied depending on the condition. Missing data and differences in interpretation accounted for most of the discrepancies. Pairs of clinicians agreed in case ascertainment in 91.2% of the cases when using KEEPER compared to 76.3% when using charts. Patient classification aligned with the gold standard in 88.1% and 86.9% of the cases respectively.

CONCLUSION

Structured data can be used for efficient and interpretable phenotype evaluation if they are limited to relevant subset and organized according to the clinical reasoning principles. A system that implements these principles can achieve noninferior performance compared to chart review at a fraction of time.

Assessing Phenotype Definitions for Algorithmic Fairness

Phenotyping is a core, routine activity in observational health research. Cohorts impact downstream analyses, such as how a condition is characterized, how patient risk is defined, and what treatments are studied. It is thus critical to ensure that cohorts are representative of all patients, independently of their demographics or social determinants of health. In this paper, we propose a set of best practices to assess the fairness of phenotype definitions. We leverage established fairness metrics commonly used in predictive models and relate them to commonly used epidemiological metrics. We describe an empirical study for Crohn's disease and diabetes type 2, each with multiple phenotype definitions taken from the literature across gender and race. We show that the different phenotype definitions exhibit widely varying and disparate performance according to the different fairness metrics and subgroups. We hope that the proposed best practices can help in constructing fair and inclusive phenotype definitions.

Trends and opportunities in computable clinical phenotyping: A scoping review

Identifying patient cohorts meeting the criteria of specific phenotypes is essential in biomedicine and particularly timely in precision medicine. Many research groups deliver pipelines that automatically retrieve and analyze data elements from one or more sources to automate this task and deliver high-performing computable phenotypes. We applied a systematic approach based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to conduct a thorough scoping review on computable clinical phenotyping. Five databases were searched using a query that combined the concepts of automation, clinical context, and phenotyping. Subsequently, four reviewers screened 7960 records (after removing over 4000 duplicates) and selected 139 that satisfied the inclusion criteria. This dataset was analyzed to extract information on target use cases, data-related topics, phenotyping methodologies, evaluation strategies, and portability of developed solutions. Most studies supported patient cohort selection without discussing the application to specific use cases, such as precision medicine. Electronic Health Records were the primary source in 87.1 % (N = 121) of all studies, and International Classification of Diseases codes were heavily used in 55.4 % (N = 77) of all studies, however, only 25.9 % (N = 36) of the records described compliance with a common data model. In terms of the presented methods, traditional Machine Learning (ML) was the dominant method, often combined with natural language processing and other approaches, while external validation and portability of computable phenotypes were pursued in many cases. These findings revealed that defining target use cases precisely, moving away from sole ML strategies, and evaluating the proposed solutions in the real setting are essential opportunities for future work. There is also momentum and an emerging need for computable phenotyping to support clinical and epidemiological research and precision medicine.

Using a data-driven approach for the development and evaluation of phenotype algorithms for systemic lupus erythematosus

BACKGROUND

Systemic lupus erythematosus (SLE) is a chronic autoimmune disease of unknown origin. The objective of this research was to develop phenotype algorithms for SLE suitable for use in epidemiological studies using empirical evidence from observational databases.

METHODS

We used a process for empirically determining and evaluating phenotype algorithms for health conditions to be analyzed in observational research. The process started with a literature search to discover prior algorithms used for SLE. We then used a set of Observational Health Data Sciences and Informatics (OHDSI) open-source tools to refine and validate the algorithms. These included tools to discover codes for SLE that may have been missed in prior studies and to determine possible low specificity and index date misclassification in algorithms for correction.

RESULTS

We developed four algorithms using our process: two algorithms for prevalent SLE and two for incident SLE. The algorithms for both incident and prevalent cases are comprised of a more specific version and a more sensitive version. Each of the algorithms corrects for possible index date misclassification. After validation, we found the highest positive predictive value estimate for the prevalent, specific algorithm (89%). The highest sensitivity estimate was found for the sensitive, prevalent algorithm (77%).

CONCLUSION

We developed phenotype algorithms for SLE using a data-driven approach. The four final algorithms may be used directly in observational studies. The validation of these algorithms provides researchers an added measure of confidence that the algorithms are selecting subjects correctly and allows for the application of quantitative bias analysis.

Machine learning approaches for electronic health records phenotyping: a methodical review

OBJECTIVE

Accurate and rapid phenotyping is a prerequisite to leveraging electronic health records for biomedical research. While early phenotyping relied on rule-based algorithms curated by experts, machine learning (ML) approaches have emerged as an alternative to improve scalability across phenotypes and healthcare settings. This study evaluates ML-based phenotyping with respect to (1) the data sources used, (2) the phenotypes considered, (3) the methods applied, and (4) the reporting and evaluation methods used.

MATERIALS AND METHODS

We searched PubMed and Web of Science for articles published between 2018 and 2022. After screening 850 articles, we recorded 37 variables on 100 studies.

RESULTS

Most studies utilized data from a single institution and included information in clinical notes. Although chronic conditions were most commonly considered, ML also enabled the characterization of nuanced phenotypes such as social determinants of health. Supervised deep learning was the most popular ML paradigm, while semi-supervised and weakly supervised learning were applied to expedite algorithm development and unsupervised learning to facilitate phenotype discovery. ML approaches did not uniformly outperform rule-based algorithms, but deep learning offered a marginal improvement over traditional ML for many conditions.

DISCUSSION

Despite the progress in ML-based phenotyping, most articles focused on binary phenotypes and few articles evaluated external validity or used multi-institution data. Study settings were infrequently reported and analytic code was rarely released.

CONCLUSION

Continued research in ML-based phenotyping is warranted, with emphasis on characterizing nuanced phenotypes, establishing reporting and evaluation standards, and developing methods to accommodate misclassified phenotypes due to algorithm errors in downstream applications.

Time to diagnosis of pulmonary hypertension and diagnostic burden: A retrospective analysis of nationwide US healthcare data

The main aim of this analysis was to investigate time from symptom onset (chronic unexplained dyspnoea [CUD]) to diagnosis of Group 1 pulmonary hypertension (PH)-pulmonary arterial hypertension (PAH)-and to characterize healthcare resource utilization leading up to diagnosis using a nationwide US claims and an electronic health record (EHR) database from Optum©. Eligible patients were ≥18 years old at first CUD diagnosis (index event) and had a PAH diagnosis on or after index date. Based on administrative codes, PAH was defined as right heart catheterization (RHC), ≥ 2 PAH diagnoses (1 within a year of RHC), and ≥1 post-RHC prescription for PAH treatment. All values are median (1st quartile-3rd quartile) unless otherwise stated. Of 854,722 patients with CUD in the claims database, 582 (0.1%) had PAH. Time from CUD to PAH diagnosis was 2.26 (0.73-4.22) years. PAH patients experienced 3 (2-4) transthoracic echocardiograms (TTEs), 6 (3-12) specialist visits, and 2 (1-4) hospitalizations during the diagnostic interval. Almost one-third of patients (29%) waited 10 months or more to have a TTE. Findings from the EHR database were broadly similar. Resource utilization during the diagnostic interval was also analyzed in an overall PH cohort: findings were generally similar to the PAH cohort (2 [1-3] TTEs, 4 [2-9] specialist visits and 2 [1-4] hospitalizations). These data indicate a delay in the diagnostic pathway for PAH, and illustrate the burden associated with PAH diagnosis.

Towards an Interoperable Ecosystem of Research Cohort and Real-world Data Catalogues Enabling Multi-center Studies

OBJECTIVES

Existing individual-level human data cover large populations on many dimensions such as lifestyle, demography, laboratory measures, clinical parameters, etc. Recent years have seen large investments in data catalogues to FAIRify data descriptions to capitalise on this great promise, i.e. make catalogue contents more Findable, Accessible, Interoperable and Reusable. However, their valuable diversity also created heterogeneity, which poses challenges to optimally exploit their richness.

METHODS

In this opinion review, we analyse catalogues for human subject research ranging from cohort studies to surveillance, administrative and healthcare records.

RESULTS

We observe that while these catalogues are heterogeneous, have various scopes, and use different terminologies, still the underlying concepts seem potentially harmonizable. We propose a unified framework to enable catalogue data sharing, with catalogues of multi-center cohorts nested as a special case in catalogues of real-world data sources. Moreover, we list recommendations to create an integrated community of metadata catalogues and an open catalogue ecosystem to sustain these efforts and maximise impact.

CONCLUSIONS

We propose to embrace the autonomy of motivated catalogue teams and invest in their collaboration via minimal standardisation efforts such as clear data licensing, persistent identifiers for linking same records between catalogues, minimal metadata 'common data elements' using shared ontologies, symmetric architectures for data sharing (push/pull) with clear provenance tracks to process updates and acknowledge original contributors. And most importantly, we encourage the creation of environments for collaboration and resource sharing between catalogue developers, building on international networks such as OpenAIRE and research data alliance, as well as domain specific ESFRIs such as BBMRI and ELIXIR.

Phenotype Algorithms to Identify Hidradenitis Suppurativa Using Real-World Data: Development and Validation Study

BACKGROUND

Hidradenitis suppurativa (HS) is a potentially debilitating, chronic, recurring inflammatory disease. Observational databases provide opportunities to study the epidemiology of HS.

OBJECTIVE

This study's objective was to develop phenotype algorithms for HS suitable for epidemiological studies based on a network of observational databases.

METHODS

A data-driven approach was used to develop 4 HS algorithms. A literature search identified prior HS algorithms. Standardized databases from the Observational Medical Outcomes Partnership (n=9) were used to develop 2 incident and 2 prevalent HS phenotype algorithms. Two open-source diagnostic tools, CohortDiagnostics and PheValuator, were used to evaluate and generate phenotype performance metric estimates, including sensitivity, specificity, positive predictive value (PPV), and negative predictive value.

RESULTS

We developed 2 prevalent and 2 incident HS algorithms. Validation showed that PPV estimates were highest (mean 86%) for the prevalent HS algorithm requiring at least two HS diagnosis codes. Sensitivity estimates were highest (mean 58%) for the prevalent HS algorithm requiring at least one HS code.

CONCLUSIONS

This study illustrates the evaluation process and provides performance metrics for 2 incident and 2 prevalent HS algorithms across 9 observational databases. The use of a rigorous data-driven approach applied to a large number of databases provides confidence that the HS algorithms can correctly identify HS subjects.

PheValuator 2.0: Methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation

PURPOSE

Phenotype algorithms are central to performing analyses using observational data. These algorithms translate the clinical idea of a health condition into an executable set of rules allowing for queries of data elements from a database. PheValuator, a software package in the Observational Health Data Sciences and Informatics (OHDSI) tool stack, provides a method to assess the performance characteristics of these algorithms, namely, sensitivity, specificity, and positive and negative predictive value. It uses machine learning to develop predictive models for determining a probabilistic gold standard of subjects for assessment of cases and non-cases of health conditions. PheValuator was developed to complement or even replace the traditional approach of algorithm validation, i.e., by expert assessment of subject records through chart review. Results in our first PheValuator paper suggest a systematic underestimation of the PPV compared to previous results using chart review. In this paper we evaluate modifications made to the method designed to improve its performance.

METHODS

The major changes to PheValuator included allowing all diagnostic conditions, clinical observations, drug prescriptions, and laboratory measurements to be included as predictors within the modeling process whereas in the prior version there were significant restrictions on the included predictors. We also have allowed for the inclusion of the temporal relationships of the predictors in the model. To evaluate the performance of the new method, we compared the results from the new and original methods against results found from the literature using traditional validation of algorithms for 19 phenotypes. We performed these tests using data from five commercial databases.

RESULTS

In the assessment aggregating all phenotype algorithms, the median difference between the PheValuator estimate and the gold standard estimate for PPV was reduced from -21 (IQR -34, -3) in Version 1.0 to 4 (IQR -3, 15) using Version 2.0. We found a median difference in specificity of 3 (IQR 1, 4.25) for Version 1.0 and 3 (IQR 1, 4) for Version 2.0. The median difference between the two versions of PheValuator and the gold standard for estimates of sensitivity was reduced from -39 (-51, -20) to -16 (-34, -6).

CONCLUSION

PheValuator 2.0 produces estimates for the performance characteristics for phenotype algorithms that are significantly closer to estimates from traditional validation through chart review compared to version 1.0. With this tool in researcher's toolkits, methods, such as quantitative bias analysis, may now be used to improve the reliability and reproducibility of research studies using observational data.

Applying Machine Learning in Distributed Data Networks for Pharmacoepidemiologic and Pharmacovigilance Studies: Opportunities, Challenges, and Considerations

Increasing availability of electronic health databases capturing real-world experiences with medical products has garnered much interest in their use for pharmacoepidemiologic and pharmacovigilance studies. The traditional practice of having numerous groups use single databases to accomplish similar tasks and address common questions about medical products can be made more efficient through well-coordinated multi-database studies, greatly facilitated through distributed data network (DDN) architectures. Access to larger amounts of electronic health data within DDNs has created a growing interest in using data-adaptive machine learning (ML) techniques that can automatically model complex associations in high-dimensional data with minimal human guidance. However, the siloed storage and diverse nature of the databases in DDNs create unique challenges for using ML. In this paper, we discuss opportunities, challenges, and considerations for applying ML in DDNs for pharmacoepidemiologic and pharmacovigilance studies. We first discuss major types of activities performed by DDNs and how ML may be used. Next, we discuss practical data-related factors influencing how DDNs work in practice. We then combine these discussions and jointly consider how opportunities for ML are affected by practical data-related factors for DDNs, leading to several challenges. We present different approaches for addressing these challenges and highlight efforts that real-world DDNs have taken or are currently taking to help mitigate them. Despite these challenges, the time is ripe for the emerging interest to use ML in DDNs, and the utility of these data-adaptive modeling techniques in pharmacoepidemiologic and pharmacovigilance studies will likely continue to increase in the coming years.

JP. Feasibility of using real-world data in the evaluation of cardiac ablation catheters: a test-case of the National Evaluation System for Health Technology Coordinating Center

OBJECTIVES

To determine the feasibility of using real-world data to assess the safety and effectiveness of two cardiac ablation catheters for the treatment of persistent atrial fibrillation and ischaemic ventricular tachycardia.

DESIGN

Retrospective cohort.

SETTING

Three health systems in the USA.

PARTICIPANTS

Patients receiving ablation with the two ablation catheters of interest at any of the three health systems.

MAIN OUTCOME MEASURES

Feasibility of identifying the medical devices and participant populations of interest as well as the duration of follow-up and positive predictive values (PPVs) for serious safety (ischaemic stroke, acute heart failure and cardiac tamponade) and effectiveness (arrhythmia-related hospitalisation) clinical outcomes of interest compared with manual chart validation by clinicians.

RESULTS

Overall, the catheter of interest for treatment of persistent atrial fibrillation was used for 4280 ablations and the catheter of interest for ischaemic ventricular tachycardia was used 1516 times across the data available within the three health systems. The duration of patient follow-up in the three health systems ranged from 91% to 97% at ≥7 days, 89% to 96% at ≥30 days, 77% to 90% at ≥6 months and 66% to 84% at ≥1 year. PPVs were 63.4% for ischaemic stroke, 96.4% for acute heart failure, 100% at one health system for cardiac tamponade and 55.7% for arrhythmia-related hospitalisation.

CONCLUSIONS

It is feasible to use real-world health system data to evaluate the safety and effectiveness of cardiac ablation catheters, though evaluations must consider the implications of variation in follow-up and endpoint ascertainment among health systems.

Data Consult Service: Can we use observational data to address immediate clinical needs?

OBJECTIVE

A number of clinical decision support tools aim to use observational data to address immediate clinical needs, but few of them address challenges and biases inherent in such data. The goal of this article is to describe the experience of running a data consult service that generates clinical evidence in real time and characterize the challenges related to its use of observational data.

MATERIALS AND METHODS

In 2019, we launched the Data Consult Service pilot with clinicians affiliated with Columbia University Irving Medical Center. We created and implemented a pipeline (question gathering, data exploration, iterative patient phenotyping, study execution, and assessing validity of results) for generating new evidence in real time. We collected user feedback and assessed issues related to producing reliable evidence.

RESULTS

We collected 29 questions from 22 clinicians through clinical rounds, emails, and in-person communication. We used validated practices to ensure reliability of evidence and answered 24 of them. Questions differed depending on the collection method, with clinical rounds supporting proactive team involvement and gathering more patient characterization questions and questions related to a current patient. The main challenges we encountered included missing and incomplete data, underreported conditions, and nonspecific coding and accurate identification of drug regimens.

CONCLUSIONS

While the Data Consult Service has the potential to generate evidence and facilitate decision making, only a portion of questions can be answered in real time. Recognizing challenges in patient phenotyping and designing studies along with using validated practices for observational research are mandatory to produce reliable evidence.

Desiderata for the development of next-generation electronic health record phenotype libraries

BACKGROUND

High-quality phenotype definitions are desirable to enable the extraction of patient cohorts from large electronic health record repositories and are characterized by properties such as portability, reproducibility, and validity. Phenotype libraries, where definitions are stored, have the potential to contribute significantly to the quality of the definitions they host. In this work, we present a set of desiderata for the design of a next-generation phenotype library that is able to ensure the quality of hosted definitions by combining the functionality currently offered by disparate tooling.

METHODS

A group of researchers examined work to date on phenotype models, implementation, and validation, as well as contemporary phenotype libraries developed as a part of their own phenomics communities. Existing phenotype frameworks were also examined. This work was translated and refined by all the authors into a set of best practices.

RESULTS

We present 14 library desiderata that promote high-quality phenotype definitions, in the areas of modelling, logging, validation, and sharing and warehousing.

CONCLUSIONS

There are a number of choices to be made when constructing phenotype libraries. Our considerations distil the best practices in the field and include pointers towards their further development to support portable, reproducible, and clinically valid phenotype design. The provision of high-quality phenotype definitions enables electronic health record data to be more effectively used in medical domains.

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment

Objective

Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers.

Materials and Methods

The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics.

Results

Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access.

Conclusions

The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

Development and validation of phenotype classifiers across multiple sites in the observational health data sciences and informatics network

Objective

Accurate electronic phenotyping is essential to support collaborative observational research. Supervised machine learning methods can be used to train phenotype classifiers in a high-throughput manner using imperfectly labeled data. We developed 10 phenotype classifiers using this approach and evaluated performance across multiple sites within the Observational Health Data Sciences and Informatics (OHDSI) network.

Materials and Methods

We constructed classifiers using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) R-package, an open-source framework for learning phenotype classifiers using datasets in the Observational Medical Outcomes Partnership Common Data Model. We labeled training data based on the presence of multiple mentions of disease-specific codes. Performance was evaluated on cohorts derived using rule-based definitions and real-world disease prevalence. Classifiers were developed and evaluated across 3 medical centers, including 1 international site.

Results

Compared to the multiple mentions labeling heuristic, classifiers showed a mean recall boost of 0.43 with a mean precision loss of 0.17. Performance decreased slightly when classifiers were shared across medical centers, with mean recall and precision decreasing by 0.08 and 0.01, respectively, at a site within the USA, and by 0.18 and 0.10, respectively, at an international site.

Discussion and Conclusion

We demonstrate a high-throughput pipeline for constructing and sharing phenotype classifiers across sites within the OHDSI network using APHRODITE. Classifiers exhibit good portability between sites within the USA, however limited portability internationally, indicating that classifier generalizability may have geographic limitations, and, consequently, sharing the classifier-building recipe, rather than the pretrained classifiers, may be more useful for facilitating collaborative observational research.

Performance evaluation of case definitions of type 1 diabetes for health insurance claims data in Japan

Background

No case definition of Type 1 diabetes (T1D) for the claims data has been proposed in Japan yet. This study aimed to evaluate the performance of candidate case definitions for T1D using Electronic health care records (EHR) and claims data in a University Hospital in Japan.

Methods

The EHR and claims data for all the visiting patients in a University Hospital were used. As the candidate case definitions for claims data, we constructed 11 definitions by combinations of International Statistical Classification of Diseases and Related Health Problems, Tenth Revision. (ICD 10) code of T1D, the claims code of insulin needles for T1D patients, basal insulin, and syringe pump for continuous subcutaneous insulin infusion (CSII). We constructed a predictive model for T1D patients using disease names, medical practices, and medications as explanatory variables. The predictive model was applied to patients of test group (validation data), and performances of candidate case definitions were evaluated.

Results

As a result of performance evaluation, the sensitivity of the confirmed disease name of T1D was 32.9 (95% CI: 28.4, 37.2), and positive predictive value (PPV) was 33.3 (95% CI: 38.0, 38.4). By using the case definition of both the confirmed diagnosis of T1D and either of the claims code of the two insulin treatment methods (i.e., syringe pump for CSII and insulin needles), PPV improved to 90.2 (95% CI: 85.2, 94.4).

Conclusions

We have established a case definition with high PPV, and the case definition can be used for precisely detecting T1D patients from claims data in Japan.

Evaluation of code-based algorithms to identify pulmonary arterial hypertension and chronic thromboembolic pulmonary hypertension patients in large administrative databases

Large administrative healthcare (including insurance claims) databases are used for various retrospective real-world evidence studies. However, in pulmonary arterial hypertension and chronic thromboembolic pulmonary hypertension, identifying patients retrospectively based on administrative codes remains challenging, as it relies on code combinations (algorithms) and the accuracy for patient identification of most of them is unknown. This study aimed to assess the performance of various algorithms in correctly identifying patients with pulmonary arterial hypertension or chronic thromboembolic pulmonary hypertension in administrative databases. A systematic literature review was performed to find publications detailing code-based algorithms used to identify pulmonary arterial hypertension and chronic thromboembolic pulmonary hypertension patients. PheValuator, a diagnostic predictive modelling tool, was applied to three US claims databases, yielding models that estimated the probability of a patient having the disease. These models were used to evaluate the performance characteristics of selected pulmonary arterial hypertension and chronic thromboembolic pulmonary hypertension algorithms. With increasing algorithm complexity, average positive predictive value increased (pulmonary arterial hypertension: 13.4–66.0%; chronic thromboembolic pulmonary hypertension: 10.3–75.1%) and average sensitivity decreased (pulmonary arterial hypertension: 61.5–2.7%; chronic thromboembolic pulmonary hypertension: 20.7–0.2%). Specificities and negative predictive values were high (≥97.5%) for all algorithms. Several of the algorithms performed well overall when considering all of these four performance parameters, and all algorithms performed with similar accuracy across the three claims databases studied, even though most were designed for patient identification in a specific database. Therefore, it is the objective of a study that will determine which algorithm may be most suitable; one- or two-component algorithms are most inclusive and three- or four-component algorithms identify most precise pulmonary arterial hypertension or chronic thromboembolic pulmonary hypertension populations, respectively.