Potential bias and lack of generalizability in electronic health record data: reflections on health equity from the National Institutes of Health Pragmatic Trials Collaboratory

Applying Implementation Science to Advance Electronic Health Record-Driven Learning Health Systems: Case Studies, Challenges, and Recommendations

With the widespread implementation of electronic health records (EHRs), there has been significant progress in developing learning health systems (LHSs) aimed at improving health and health care delivery through rapid and continuous knowledge generation and translation. To support LHSs in achieving these goals, implementation science (IS) and its frameworks are increasingly being leveraged to ensure that LHSs are feasible, rapid, iterative, reliable, reproducible, equitable, and sustainable. However, 6 key challenges limit the application of IS to EHR-driven LHSs: barriers to team science, limited IS experience, data and technology limitations, time and resource constraints, the appropriateness of certain IS approaches, and equity considerations. Using 3 case studies from diverse health settings and 1 IS framework, we illustrate these challenges faced by LHSs and offer solutions to overcome the bottlenecks in applying IS and utilizing EHRs, which often stymie LHS progress. We discuss the lessons learned and provide recommendations for future research and practice, including the need for more guidance on the practical application of IS methods and a renewed emphasis on generating and accessing inclusive data.

Generative artificial intelligence in ophthalmology: current innovations, future applications and challenges

The rapid advancements in generative artificial intelligence are set to significantly influence the medical sector, particularly ophthalmology. Generative adversarial networks and diffusion models enable the creation of synthetic images, aiding the development of deep learning models tailored for specific imaging tasks. Additionally, the advent of multimodal foundational models, capable of generating images, text and videos, presents a broad spectrum of applications within ophthalmology. These range from enhancing diagnostic accuracy to improving patient education and training healthcare professionals. Despite the promising potential, this area of technology is still in its infancy, and there are several challenges to be addressed, including data bias, safety concerns and the practical implementation of these technologies in clinical settings.

Post-trial responsibilities in pragmatic clinical trials: Fulfilling the promise of research to drive real-world change

While considerable scholarship has explored responsibilities owed to research participants at the conclusion of explanatory clinical trials, no guidance exists regarding responsibilities owed at the conclusion of a pragmatic clinical trial (PCT). Yet post-trial responsibilities in PCTs present distinct considerations from those emphasized in existing guidance and prior scholarship. Among these considerations include the responsibilities of the healthcare delivery systems in which PCTs are embedded, and decisions about implementation for interventions that demonstrate meaningful benefit following their integration into usual care settings-or deimplementation for those that fail to do so. In this article, we present an overview of prior scholarship and guidance on post-trial responsibilities, and then identify challenges for post-trial responsibilities for PCTs. We argue that, given one of the key rationales for PCTs is that they can facilitate uptake of their results by relevant decision-makers, there should be a presumptive default that PCT study results be incorporated into future care delivery processes. Fulfilling this responsibility will require prospective planning by researchers, healthcare delivery system leaders, institutional review boards, and sponsors, so as to ensure that the knowledge gained from PCTs does, in fact, influence real-world practice.

A Machine Learning Application to Classify Patients at Differing Levels of Risk of Opioid Use Disorder: Clinician-Based Validation Study

Background

Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment.

Objective

This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians.

Methods

The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient's OUD risk level, which was then compared to the ML application's risk assignments.

Results

A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified.

Conclusions

A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients' different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.

Assessing Real-World Data From Electronic Health Records for Health Technology Assessment: The SUITABILITY Checklist: A Good Practices Report of an ISPOR Task Force

This ISPOR Good Practices report provides a framework for assessing the suitability of electronic health records data for use in health technology assessments (HTAs). Although electronic health record (EHR) data can fill evidence gaps and improve decisions, several important limitations can affect its validity and relevance. The ISPOR framework includes 2 components: data delineation and data fitness for purpose. Data delineation provides a complete understanding of the data and an assessment of its trustworthiness by describing (1) data characteristics; (2) data provenance; and (3) data governance. Fitness for purpose comprises (1) data reliability items, ie, how accurate and complete the estimates are for answering the question at hand and (2) data relevance items, which assess how well the data are suited to answer the particular question from a decision-making perspective. The report includes a checklist specific to EHR data reporting: the ISPOR SUITABILITY Checklist. It also provides recommendations for HTA agencies and policy makers to improve the use of EHR-derived data over time. The report concludes with a discussion of limitations and future directions in the field, including the potential impact from the substantial and rapid advances in the diffusion and capabilities of large language models and generative artificial intelligence. The report's immediate audiences are HTA evidence developers and users. We anticipate that it will also be useful to other stakeholders, particularly regulators and manufacturers, in the future.

Enhancing data practices for Whole Health: Strategies for a transformative future

We explored the challenges and solutions for managing data within the Whole Health System (WHS), which operates as a Learning Health System and a patient-centered healthcare approach that combines conventional and complementary approaches. Addressing these challenges is critical for enhancing patient care and improving outcomes within WHS. The proposed solutions include prioritizing interoperability for seamless data exchange, incorporating patient-centered comparative clinical effectiveness research and real-world data to personalize treatment plans and validate integrative approaches, and leveraging advanced data analytics tools to incorporate patient-reported outcomes, objective metrics, robust data platforms. Implementing these measures will enable WHS to fulfill its mission as a holistic and patient-centered healthcare model, promoting greater collaboration among providers, boosting the well-being of patients and providers, and improving patient outcomes.

The transition from genomics to phenomics in personalized population health

Modern health care faces several serious challenges, including an ageing population and its inherent burden of chronic diseases, rising costs and marginal quality metrics. By assessing and optimizing the health trajectory of each individual using a data-driven personalized approach that reflects their genetics, behaviour and environment, we can start to address these challenges. This assessment includes longitudinal phenome measures, such as the blood proteome and metabolome, gut microbiome composition and function, and lifestyle and behaviour through wearables and questionnaires. Here, we review ongoing large-scale genomics and longitudinal phenomics efforts and the powerful insights they provide into wellness. We describe our vision for the transformation of the current health care from disease-oriented to data-driven, wellness-oriented and personalized population health.

Making causal inferences from transactional data: A narrative review of opportunities and challenges when implementing the target trial framework

The target trial framework has emerged as a powerful tool for addressing causal questions in clinical practice and in public health. In the healthcare sector, where decision-making is increasingly data-driven, transactional databases, such as electronic health records (EHR) and insurance claims, present an untapped potential for answering complex causal questions. This narrative review explores the potential of the integration of the target trial framework with real-world data to enhance healthcare decision-making processes. We outline essential elements of the target trial framework, and identify pertinent challenges in data quality, privacy concerns, and methodological limitations, proposing solutions to overcome these obstacles and optimize the framework's application.

Visual Analytics for Data-Driven Understanding of the Substance Use Disorder Epidemic

The substance use disorder epidemic has emerged as a serious public health crisis, presenting complex challenges. Visual analytics offers a unique approach to address this complexity and facilitate effective interventions. This paper details the development of an innovative visual analytics dashboard, aimed at enhancing our understanding of the substance use disorder epidemic. By employing record linkage techniques, we integrate diverse data sources to provide a comprehensive view of the epidemic. Adherence to responsive, open, and user-centered design principles ensures the dashboard's usefulness and usability. Our approach to data and design encourages collaboration among various stakeholders, including researchers, politicians, and healthcare practitioners. Through illustrative outputs, we demonstrate how the dashboard can deepen our understanding of the epidemic, support intervention strategies, and evaluate the effectiveness of implemented measures. The paper concludes with a discussion of the dashboard's use cases and limitations.

Disparities in Diagnosis, Access to Specialist Care and Treatment for Inborn Errors of Immunity

Inborn errors of immunity represent a rapidly expanding group of genetic disorders of the immune system. Significant advances have been made in recent years in diagnosis, including using genetic testing and newborn screening; treatment, including precision therapies, gene therapy and hematopoietic stem cell transplant; and development of patient registries to inform prevalence, understand morbidity of these disorders and guide the development of clinical trials. However, significant disparities due to age, race, ethnicity, socioeconomic status, or geographic location exist in all aspects of care of patients with inborn errors of immunity, beginning with delays in diagnosis and further compounded by impaired access to specialist care and treatment, leading to a notable impact on outcomes including morbidity and mortality. Addressing and correcting these disparities will require coordinated, deliberate and prolonged effort. Proposed strategies to improve equity at different levels include public health measures such as implementing universal newborn screening, supporting expanded health insurance coverage for diagnostic testing and treatment, improving access to novel therapeutics in low and middle income countries and developing artificial intelligence / machine learning tools to reduce delays in diagnosis, particularly in rural or less developed areas where access to specialist care is limited.