Improving transparency to build trust in real-world secondary data studies for hypothesis testing-Why, what, and how: recommendations and a road map from the real-world evidence transparency initiative

Sharing Is Caring? International Society for Pharmacoepidemiology Review and Recommendations for Sharing Programming Code

PURPOSE

There is increasing recognition of the importance of transparency and reproducibility in scientific research. This study aimed to quantify the extent to which programming code is publicly shared in pharmacoepidemiology, and to develop a set of recommendations on this topic.

METHODS

We conducted a literature review identifying all studies published in Pharmacoepidemiology and Drug Safety (PDS) between 2017 and 2022. Data were extracted on the frequency and types of programming code shared, and other key open science practices (clinical codelist sharing, data sharing, study preregistration, and stated use of reporting guidelines and preprinting). We developed six recommendations for investigators who choose to share code and gathered feedback from members of the International Society for Pharmacoepidemiology (ISPE).

RESULTS

Programming code sharing by articles published in PDS ranged from 1.8% in 2017 to 9.5% in 2022. It was more prevalent among articles with a methodological focus, simulation studies, and papers which also shared record-level data.

CONCLUSION

Programming code sharing is rare but increasing in pharmacoepidemiology studies published in PDS. We recommend improved reporting of whether code is shared and how available code can be accessed. When sharing programming code, we recommend the use of permanent digital identifiers, appropriate licenses, and, where possible, adherence to good software practices around the provision of metadata and documentation, computational reproducibility, and data privacy.

Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data

PURPOSE

We describe constructs designed to protect the integrity of the results from comparative analyses using real-world data (RWD): staging and clean room.

METHODS

Staging involves performing sequential preliminary analyses and evaluating the population size available and potential bias before conducting comparative analyses. A clean room involves restricted access to data and preliminary results, policies governing exploratory analyses and protocol deviations, and audit trail. These constructs are intended to allow decisions about protocol deviations, such as changes to design or model specification, to be made without knowledge of how they might affect subsequent analyses. We describe an example for implementing staging with a clean room.

RESULTS

Stage 1 may involve selecting a data source, developing and registering a protocol, establishing a clean room, and applying inclusion/exclusion criteria. Stage 2 may involve attempting to achieve covariate balance, often through propensity score models. Stage 3 may involve evaluating the presence of residual confounding using negative control outcomes. After each stage, check points may be implemented when a team of statisticians, epidemiologists and clinicians masked to how their decisions may affect study outcomes, reviews the results. This review team may be tasked with making recommendations for protocol deviations to address study precision or bias. They may recommend proceeding to the next stage, conducting additional analyses to address bias, or terminating the study. Stage 4 may involve conducting the comparative analyses.

CONCLUSIONS

The staging and clean room constructs are intended to protect the integrity and enhance confidence in the results of analyses of RWD.

Biases in Electronic Health Records Data for Generating Real-World Evidence: An Overview

Electronic Health Records (EHR) are increasingly being perceived as a unique source of data for clinical research as they provide unprecedentedly large volumes of real-time data from real-world settings. In this review of the secondary uses of EHR, we identify the anticipated breadth of opportunities, pointing out the data deficiencies and potential biases that are likely to limit the search for true causal relationships. This paper provides a comprehensive overview of the types of biases that arise along the pathways that generate real-world evidence and the sources of these biases. We distinguish between two levels in the production of EHR data where biases are likely to arise: (i) at the healthcare system level, where the principal source of bias resides in access to, and provision of, medical care, and in the acquisition and documentation of medical and administrative data; and (ii) at the research level, where biases arise from the processes of extracting, analyzing, and interpreting these data. Due to the plethora of biases, mainly in the form of selection and information bias, we conclude with advising extreme caution about making causal inferences based on secondary uses of EHRs.

The FAIRification of research in real-world evidence: A practical introduction to reproducible analytic workflows using Git and R

Transparency and reproducibility are major prerequisites for conducting meaningful real-world evidence (RWE) studies that are fit for decision-making. Many advances have been made in the documentation and reporting of study protocols and results, but the principles for version control and sharing of analytic code in RWE are not yet as established as in other quantitative disciplines like computational biology and health informatics. In this practical tutorial, we aim to give an introduction to distributed version control systems (VCS) tailored toward the FAIR (Findable, Accessible, Interoperable, and Reproducible) implementation of RWE studies. To ease adoption, we provide detailed step-by-step instructions with practical examples on how the Git VCS and R programming language can be implemented into RWE study workflows to facilitate reproducible analyzes. We further discuss and showcase how these tools can be used to track changes, collaborate, disseminate, and archive RWE studies through dedicated project repositories that maintain a complete audit trail of all relevant study documents.

Real world evidence for public health decision-making on vaccination policies: perspectives from an expert roundtable

INTRODUCTION

Influenza causes significant morbidity and mortality, but influenza vaccine uptake remains below most countries' targets. Vaccine policy recommendations vary, as do procedures for reviewing and appraising the evidence.

AREAS COVERED

During a series of roundtable discussions, we reviewed procedures and methodologies used by health ministries in four European countries to inform vaccine recommendations. We review the type of evidence currently recommended by each health ministry and the range of approaches toward considering randomized controlled trials (RCTs) and real-world evidence (RWE) studies when setting influenza vaccine recommendations.

EXPERT OPINION

Influenza vaccine recommendations should be based on data from both RCTs and RWE studies of efficacy, effectiveness, and safety. Such data should be considered alongside health-economic, cost-effectiveness, and budgetary factors. Although RCT data are more robust and less prone to bias, well-designed RWE studies permit timely evaluation of vaccine benefits, effectiveness comparisons over multiple seasons in large populations, and detection of rare adverse events, under real-world conditions. Given the variability of vaccine effectiveness due to influenza virus mutations and increasing diversification of influenza vaccines, we argue that consideration of both RWE and RCT evidence is the best approach to more nuanced and timely updates of influenza vaccine recommendations.

The Evolve to Next-Gen ACT Network: An evolving open-access, real-world data resource primed for real-world evidence research across the Clinical and Translational Science Award Consortium

The ACT Network was funded by NIH to provide investigators from across the Clinical and Translational Science Award (CTSA) Consortium the ability to directly query national federated electronic health record (EHR) data for cohort discovery and feasibility assessment of multi-site studies. NIH refunded the program for expanded research application to become "Evolve to Next-Gen ACT" (ENACT). In parallel, the US Food and Drug Administration has been evaluating the use of real-world data (RWD), including EHR data, as sources of real-world evidence (RWE) for its regulatory decisions involving drug and biological products. Using insights from implementation science, six lessons learned from ACT for developing and sustaining RWD/RWE infrastructures and networks across the CTSA Consortium are presented in order to inform ENACT's development from the outset. Lessons include intentional institutional relationship management, end-user engagement, beta-testing, and customer-driven adaptation. The ENACT team is also conducting customer discovery interviews with CTSA hub and investigators using Innovation-Corps@NCATS (I-Corps™) methodology for biomedical entrepreneurs to uncover unmet RWD needs. Possible ENACT value proposition hypotheses are presented by stage of research. Developing evidence about methods for sustaining academically derived data infrastructures and support can advance the science of translation and support our nation's RWD/RWE research capacity.

Emulating Target Trials With Real-World Data to Inform Health Technology Assessment: Findings and Lessons From an Application to Emergency Surgery

OBJECTIVES

International health technology assessment (HTA) agencies recommend that real-world data (RWD) are used in some circumstances to add to the evidence base about the effectiveness and cost-effectiveness of health interventions. The target trial framework applies the design principles of randomized-controlled trials to RWD and can help alleviate inevitable concerns about bias and design flaws with nonrandomized studies. This article aimed to tackle the lack of guidance and exemplar applications on how this methodology can be applied to RWD to inform HTA decision making.

METHODS

We use Hospital Episode Statistics data from England on emergency hospital admissions from 2010 to 2019 to evaluate the cost-effectiveness of emergency surgery for 2 acute gastrointestinal conditions. We draw on the case study to describe the main challenges in applying the target trial framework alongside RWD and provide recommendations for how these can be addressed in practice.

RESULTS

The 4 main challenges when applying the target trial framework to RWD are (1) defining the study population, (2) defining the treatment strategies, (3) establishing time zero (baseline), and (4) adjusting for unmeasured confounding. The recommendations for how to address these challenges, mainly around the incorporation of expert judgment and use of appropriate methods for handling unmeasured confounding, are illustrated within the case study.

CONCLUSIONS

The recommendations outlined in this study could help future studies seeking to inform HTA decision processes. These recommendations can complement checklists for economic evaluations and design tools for estimating treatment effectiveness in nonrandomized studies.

Strategies to Address Current Challenges in Real-World Evidence Generation in Japan

The generation of real-world evidence (RWE), which describes patient characteristics or treatment patterns using real-world data (RWD), is rapidly growing more popular as a tool for decision-making in Japan. The aim of this review was to summarize challenges to RWE generation in Japan related to pharmacoepidemiology, and to propose strategies to address some of these challenges. We first focused on data-related issues, including the lack of transparency of RWD sources, linkage across different care settings, definitions of clinical outcomes, and the overall assessment framework of RWD when used for research purposes. Next the study reviewed methodology-related challenges. As lack of design transparency impairs study reproducibility, transparent reporting of study design is critical for stakeholders. For this review, we considered different sources of biases and time-varying confounding, along with potential study design and methodological solutions. Additionally, the implementation of robust assessment of definition uncertainty, misclassification, and unmeasured confounders would enhance RWE credibility in light of RWD source-related limitations, and is being strongly considered by task forces in Japan. Overall, the development of guidance for best practices on data source selection, design transparency, and analytical methods to address different sources of biases and robustness in the process of RWE generation will enhance credibility for stakeholders and local decision-makers.

Structure and Content of a Taxonomy to Support the Use of Real-World Evidence by Health Technology Assessment Practitioners and Healthcare Decision Makers

This is one of a series of articles that consider the barriers to optimal use of real-world evidence (RWE) in health technology assessment and how to overcome them. The work was performed as part of EUreccA 2025, in particular with the RWE workstream embodied within that collaboration. Elsewhere in this issue we described the reasoning and process that led us to develop practical tools to support RWE use, including this taxonomy and explained the methods used to do so. The taxonomy classifies questions that are typically addressed using real-world data in health technology assessment and the data sources typically used to address these questions. In this article, we describe the taxonomy itself. For as many of the pairings as possible, we have provided links to advice and methods on how to address the associated question using those data. We have also provided links to examples of RWE use in practical decision making to answer the questions posed. Our work is not complete, but we believe it is sufficient to demonstrate the value of such a taxonomy and information source if it is completed and curated as a "wiki" by the community that would use it.

Orphan drugs' clinical uncertainty and prices: Addressing allocative and technical inefficiencies in orphan drug reimbursement

Legislations incentivising orphan drug development and scientific advances have made orphan drugs pharma's high-end favourite for the past two decades. Currently, around 50% of new marketing authorizations are for orphan drugs. For third-party healthcare payers ("payers") the rise of orphan drugs presents new challenges, including a high degree of uncertainty around clinical benefits and harms, a moderate effect size (for many orphan drugs), and a high price tag. The association of high clinical uncertainty and moderate effect sizes is not surprising in small target populations but in combination with high prices creates the risk of allocative and technical inefficiencies for payers. We here discuss and illustrate these risks. A combination of policies is needed for mitigation of allocative inefficiency: while there may be a rationale for higher prices for orphan than non-orphan drugs, a focus of pricing and reimbursement negotiations should include considerations of product profitability and of the consequences of orphan drug costs on the distribution inequality of medication costs for individual insured persons, coupled to knowledge generation from reimbursement contracts covering high-price orphan drugs that would benefit the wider patient community. Performance-based managed entry agreements could help to de-risk the economic consequences of clinical uncertainty and to mitigate technical inefficiency.

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force

PROBLEM

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

WHAT WE DID

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Understanding variation in the results of real-world evidence studies that seem to address the same question

OBJECTIVES

Multiple database studies on the same question, conducted by different investigators using different approaches or different data sources, can be considered sensitivity analyses for the same causal treatment effect question. We evaluated the contribution of alternative study design parameters and analysis choices to variation in estimates of the risk of major bleeding with dabigatran compared with warfarin.

STUDY DESIGN AND SETTING

We followed a 7-step process: (1) identify published studies asking the same question, (2) independently reproduce selected studies in the same data sources as the original authors, (3) contact original authors, (4) evaluate validity, (5) document critical study parameter specifications, (6) implement a designed matrix of variations in study parameters based on the original studies, and (7) evaluate contributors to variation in results.

RESULTS

Most variation remained unexplained (60-88%). Of the explained variation, two-thirds were related to data and population differences, and one-third were related to the use of alternative study design and analysis parameters. Among these, the most prominent were differences in outcome algorithms and criteria used to define follow-up.

CONCLUSION

When making policy decisions based on database study findings, it is important to evaluate the validity, consistency, and robustness of results to alternative design and analysis decisions.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force

OBJECTIVES

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

METHODS

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Explanation and Elaboration: A Report of the ISPOR CHEERS II Good Practices Task Force

Health economic evaluations are comparative analyses of alternative courses of action in terms of their costs and consequences. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, published in 2013, was created to ensure health economic evaluations are identifiable, interpretable, and useful for decision making. It was intended as guidance to help authors report accurately which health interventions were being compared and in what context, how the evaluation was undertaken, what the findings were, and other details that may aid readers and reviewers in interpretation and use of the study. The new CHEERS 2022 statement replaces the previous CHEERS reporting guidance. It reflects the need for guidance that can be more easily applied to all types of health economic evaluation, new methods and developments in the field, and the increased role of stakeholder involvement including patients and the public. It is also broadly applicable to any form of intervention intended to improve the health of individuals or the population, whether simple or complex, and without regard to context (such as healthcare, public health, education, and social care). This Explanation and Elaboration Report presents the new CHEERS 2022 28-item checklist with recommendations and explanation and examples for each item. The CHEERS 2022 statement is primarily intended for researchers reporting economic evaluations for peer-reviewed journals and the peer reviewers and editors assessing them for publication. Nevertheless, we anticipate familiarity with reporting requirements will be useful for analysts when planning studies. It may also be useful for health technology assessment bodies seeking guidance on reporting, given that there is an increasing emphasis on transparency in decision making.

Lessons Learned Using Real-World Data to Emulate Randomized Trials: A Case Study of Treatment Effectiveness for Newly Diagnosed Immune Thrombocytopenia

Regulatory agencies are increasingly considering real-world evidence (RWE) to support label expansions of approved medicines. We conducted a comparative effectiveness study to emulate a proposed randomized trial of romiplostim vs. standard-of-care (SOC) therapy among patients with recently diagnosed (≤12 months) immune thrombocytopenia (ITP), that could support expansion of the romiplostim label. We discuss challenges that we encountered and solutions that were developed to address those challenges. Study size was a primary concern, particularly for romiplostim initiators, given the rarity of ITP and the stringent trial eligibility criteria. For this reason, we leveraged multiple data sources (Nordic Country Patient Registry for Romiplostim; chart review study of romiplostim initiators in Europe; Flatiron Health EMR linked with MarketScan claims). Additionally, unlike the strictly controlled clinical trial setting, platelet counts were not measured at regular intervals in the observational data sources, and therefore the end point of durable platelet response often used in trials could not be reliably measured. Instead, the median platelet count was chosen as the primary end point. Ultimately, while we observed a slightly higher median platelet count in the romiplostim group vs. SOC, precision was limited because of small study size (median difference was 11 × 10⁹ /L (95% CI: -59, 81)). We underscore the importance of conducting comprehensive feasibility assessments to identify fit-for-purpose data sources with sufficient sample size, data elements, and follow-up. Beyond technical challenges, we also discuss approaches to increase the credibility of RWE, including systematic incorporation of clinical expertise into study design decisions, and separation between decision makers and the data.