Evaluating the Use of Nonrandomized Real-World Data Analyses for Regulatory Decision Making

Contemporary Practice and Considerations for Real-World Data Source Identification and Feasibility Assessment

PURPOSE

There has been rapid growth in the variety and number of real-world data (RWD) sources, as well as the number of regulatory documents that provide guidance for assessing the suitability of RWD sources for pharmacoepidemiology studies. This study aims to assess differences in RWD guidance and variability in current practice for identifying and assessing RWD for studies with regulatory purpose.

METHODS

Key criteria for feasibility assessment were mapped against relevant regulatory guidance documents across US, EU, and Asia-Pacific regions. An online survey was designed and deployed to International Society for Pharmacoepidemiology members to understand current practice. Findings were summarized and used to inform key considerations and recommendations.

RESULTS

Eleven RWD guidance documents were identified and mapped against 14 RWD assessment criteria. Variability was seen across these documents in guidance for these criteria. Between December 2022 and January 2023, 37 survey respondents reported having used RWD for post-marketing commitments (34, 92%) and/or background epidemiology (28, 76%). RWD were mostly identified through literature (33, 89%) and data landscaping (26, 70%); guidance documents referenced included: Food and Drug Administration (20, 54%), European Network for Centres for Pharmacoepidemiology and Pharmacovigilance (17, 46%), European Medical Agency (16, 43%), and Structured Process to Identify Fit-For-Purpose Data (11, 30%). Challenges for conducting feasibility assessments included RWD accessibility, ability to complete validation, and RWD provider responsiveness.

CONCLUSIONS

Existing guidelines are used extensively by researchers, but key criteria for RWD identification and feasibility assessment are not reflected consistently and challenges remain. Recommendations have been made reflecting study findings.

Examining the Effect of Missing Data and Unmeasured Confounding on External Comparator Studies: Case Studies and Simulations

BACKGROUND AND OBJECTIVE

Missing data and unmeasured confounding are key challenges for external comparator studies. This work evaluates bias and other performance characteristics depending on missingness and unmeasured confounding by means of two case studies and simulations.

METHODS

Two case studies were constructed by taking the treatment arms from two randomised controlled trials and an external real-world data source that exhibited substantial missingness. The indications of the randomised controlled trials were multiple myeloma and metastatic hormone-sensitive prostate cancer. Overall survival was taken as the main endpoint. The effects of missing data and unmeasured confounding were assessed for the case studies by reporting estimated external comparator versus randomised controlled trial treatment effects. Based on the two case studies, simulations were performed broadening the settings by varying the underlying hazard ratio, the sample size, the sample size ratio between the experimental arm and the external comparator, the number of missing covariates and the percentage of missingness. Thereby, bias and other performance metrics could be quantified dependent on these factors.

RESULTS

For the multiple myeloma external comparator study, results were in line with the randomised controlled trial, despite missingness and potential unmeasured confounding, while for the metastatic hormone-sensitive prostate cancer case study missing data led to a low sample size, leading overall to inconclusive results. Furthermore, for the metastatic hormone-sensitive prostate cancer study, missing data in important eligibility criteria led to further limitations. Simulations were successfully applied to gain a quantitative understanding of the effects of missing data and unmeasured confounding.

CONCLUSIONS

This exploratory study confirmed external comparator strengths and limitations by quantifying the impact of missing data and unmeasured confounding using case studies and simulations. In particular, missing data in key eligibility criteria were seen to limit the ability to derive the external comparator target analysis population accurately, while simulations demonstrated the magnitude of bias to expect for various settings.

Exploring the discrepancies between clinical trials and real-world data: A small-cell lung cancer study

The potential of real-world data to inform clinical trial design and supplement control arms has gained much interest in recent years. The most common approach relies on reproducing control arm outcomes by matching real-world patient cohorts to clinical trial baseline populations. However, recent studies pointed out that there is a lack of replicability, generalisability, and consensus. In this article, we propose a novel approach that aims to explore and examine these discrepancies by concomitantly investigating the impact of selection criteria and operations on the measurements of outcomes from the patient data. We tested the approach on a dataset consisting of small-cell lung cancer patients receiving platinum-based chemotherapy regimens from a real-world data cohort (n = 223) and six clinical trial control arms (n = 1224). The results showed that the discrepancy between real-world and clinical trial data potentially depends on differences in both patient populations and operational conditions (e.g., frequency of assessments, and censoring), for which further investigation is required. Discovering and accounting for confounders, including hidden effects of differences in operations related to the treatment process and clinical trial study protocol, would potentially allow for improved translation between clinical trials and real-world data. Continued development of the method presented here to systematically explore and account for these differences could pave the way for transferring learning across clinical studies and developing mutual translation between the real-world and clinical trials to inform clinical study design.

The Use of Real-World Evidence for Regulatory Decisions in China

There is a growing demand for the use of high-quality real-world evidence (RWE) to support regulatory decision-making worldwide and in China, which highlights the need for conducting literature reviews to evaluate the available data and evidence. This study aims to review the use of RWE in Chinese regulatory decisions and to summarize relevant regulatory and methodological considerations to inform the future use of RWE in China. We identified policy documents, technical guidance documents, and cases on official Chinese government websites and extracted their contents separately. We consulted experts from the National Medical Products Administration (NMPA) and academic institutes and searched case-related articles for enrichment. We also searched and included articles related to the use of RWE/Real-world data in Chinese regulatory decisions. Six trial versions of technical guidance documents, 7 case studies, and 40 articles related to the Chinese regulatory decisions were included in this study. Based on the technical guidance, data quality, and appropriate study design and statistical analysis are the main concerns for RWE generation. The cases and articles related to regulatory decisions revealed 9 main concerns, including data sources and applicability, data quality, strength of existing evidence, appropriate study design and statistical analysis, regulated and transparent process for analysis and evidence generation, product safety and efficacy, product characteristics and clinical needs, ethical considerations and data security, and communicate adequately with regulatory authorities. Among these concerns, data issues are central. Preliminary attempts have been made by the NMPA to promote the use of RWE, but substantial challenges still remain.

A methodological review of the high-dimensional propensity score in comparative-effectiveness and safety-of-interventions research finds incomplete reporting relative to algorithm development and robustness

OBJECTIVES

The use of secondary databases has become popular for evaluating the effectiveness and safety of interventions in real-life settings. However, the absence of important confounders in these databases is challenging. To address this issue, the high-dimensional propensity score (hdPS) algorithm was developed in 2009. This algorithm uses proxy variables for mitigating confounding by combining information available across several healthcare dimensions. This study assessed the methodology and reporting of the hdPS in comparative effectiveness and safety research.

STUDY DESIGN AND SETTING

In this methodological review, we searched PubMed and Google Scholar from July 2009 to May 2022 for studies that used the hdPS for evaluating the effectiveness or safety of healthcare interventions. Two reviewers independently extracted study characteristics and assessed how the hdPS was applied and reported. Risk of bias was evaluated with the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool.

RESULTS

In total, 136 studies met the inclusion criteria; the median publication year was 2018 (Q1-Q3 2016-2020). The studies included 192 datasets, mostly North American databases (n = 132, 69%). The hdPS was used in primary analysis in 120 studies (88%). Dimensions were defined in 101 studies (74%), with a median of 5 (Q1-Q3 4-6) dimensions included. A median of 500 (Q1-Q3 200-500) empirically identified covariates were selected. Regarding hdPS reporting, only 11 studies (8%) reported all recommended items. Most studies (n = 81, 60%) had a moderate overall risk of bias.

CONCLUSION

There is room for improvement in the reporting of hdPS studies, especially regarding the transparency of methodological choices that underpin the construction of the hdPS.

Validation of alopecia coding in US claims data among women of childbearing age

BACKGROUND

Accurately identifying alopecia in claims data is important to study this rare medication side effect.

OBJECTIVES

To develop and validate a claims-based algorithm to identify alopecia in women of childbearing age.

METHODS

We linked electronic health records from a large healthcare system in Massachusetts (Mass General Brigham) with Medicaid claims data from 2016 through 2018 to identify all women aged 18 to 50 years with an ICD-10 code for alopecia, including alopecia areata, androgenic alopecia, non-scarring alopecia, or cicatricial alopecia, from a visit to the MGB system. Using eight predefined algorithms to identify alopecia in Medicaid claims data, we randomly selected 300 women for whom we reviewed their charts to validate the alopecia diagnosis. Positive predictive values (PPVs) were computed for the primary algorithm and seven algorithm variations, stratified by race.

RESULTS

Out of 300 patients with at least 1 ICD-10 code for alopecia in the Medicaid claims, 286 had chart-confirmed alopecia (PPV = 95.3%). The algorithm requiring two diagnosis codes plus one prescription claim for alopecia treatment identified 55 patients (PPV = 100%). The algorithm requiring 1 diagnosis code for alopecia plus 1 procedure claim for intralesional triamcinolone injection identified 35 patients (PPV = 100%). Across all 8 algorithms tested, the PPV varied between 95.3% and 100%. The PPV for alopecia ranged from 94% to 100% in White and 96%-100% in 48 non-White women. The exact date of alopecia onset was difficult to determine in charts.

CONCLUSION

At least one recorded ICD-10 code for alopecia in claims data identified alopecia in women of childbearing age with high accuracy.

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

OBJECTIVES

We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

DESIGN

Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

DATA SOURCES

Systematic search: Medline; Snowballing: starting from three articles (D'Andrea et al, Quigley et al and Faria et al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

DATA EXTRACTION AND SYNTHESIS

Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

RESULTS

Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

CONCLUSION

Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

OSF REGISTRATION NUMBER

OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data

Objective

To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology).

Design

Meta-analysis of RCT-DUPLICATE data.

Data sources

Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17.

Eligibility criteria for selecting studies

Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE.

Results

Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

Real-life study to assess effectiveness and safety of eculizumab in patients with neuromyelitis optica spectrum disorders in France: protocol for ECUP4, an observational study

Background

Eculizumab, a humanized monoclonal antibody targeting the C5 complement protein, has been approved for the treatment of neuromyelitis optica spectrum disorders (NMOSD) in adult patients who are anti-aquaporin-4 (AQP4) antibody positive (Ab+). The aim of this study is to evaluate the long-term effectiveness and safety of eculizumab in French adults with NMOSD and to describe patients' characteristics, disability, and quality of life using data collected in a real-world setting.

Methods

This is the protocol for ECUP4, an ongoing prospective, observational, non-comparative, multicenter study conducted in 32 reference centers in France. Eligible patients must also be enrolled in NOMADMUS, a nested cohort of the French national multiple sclerosis registry (OFSEP). The primary endpoint is the annualized relapse rate. Secondary endpoints include the long-term safety of eculizumab, as well as patients' characteristics, treatment outcomes, disability, pain, visual acuity, and quality of life. Visits and treatments follow routine clinical practice. The case report forms (CRF) comprise data recorded in the context of the NOMADMUS cohort, collected during routine visits. The inclusion period is planned for 3 years, with no limitation on the number of patients enrolled. The maximum follow-up duration will be 5.5 years.

Conclusion

The efficacy and safety of eculizumab in patients with AQP4+ NMOSD have been demonstrated in randomized clinical trials that showed a significant reduction in the risk of relapse, with a safety profile consistent with other indications. This study will provide clinical and patient-reported evidence of the benefits of eculizumab, using data from a real-world setting in France.

Trial registration number

This study is registered at the French public repertory Health data Hub, N° F20211228123801. All information can be accessed at: https://www.health-data-hub.fr/.

Overview of global real-world data sources for pediatric pharmacoepidemiologic research

PURPOSE

Given limited information available on real-world data (RWD) sources with pediatric populations, this study describes features of globally available RWD sources for pediatric pharmacoepidemiologic research.

METHODS

An online questionnaire about pediatric RWD sources and their attributes and capabilities was completed by members and affiliates of the International Society for Pharmacoepidemiology and representatives of nominated databases. All responses were verified by database representatives and summarized.

RESULTS

Of 93 RWD sources identified, 55 unique pediatric RWD sources were verified, including data from Europe (47%), United States (38%), multiregion (7%), Asia-Pacific (5%), and South America (2%). Most databases had nationwide coverage (82%), contained electronic health/medical records (47%) and/or administrative claims data (42%) and were linkable to other databases (65%). Most (71%) had limited outside access (e.g., by approval or through local collaborators); only 10 (18%) databases were publicly available. Six databases (11%) reported having >20 million pediatric observations. Most (91%) included children of all ages (birth until 18th birthday) and contained outpatient medication data (93%), while half (49%) contained inpatient medication data. Many databases captured vaccine information for children (71%), and one-third had regularly updated data on pediatric height (31%) and weight (33%). Other pediatric data attributes captured include diagnoses and comorbidities (89%), lab results (58%), vital signs (55%), devices (55%), imaging results (42%), narrative patient histories (35%), and genetic/biomarker data (22%).

CONCLUSIONS

This study provides an overview with key details about diverse databases that allow researchers to identify fit-for-purpose RWD sources suitable for pediatric pharmacoepidemiologic research.

Enhancement and external validation of algorithms using diagnosis codes to identify invasive Escherichia coli disease

OBJECTIVE

To assess the predictive accuracy of code-based algorithms for identifying invasive Escherichia coli (E. coli) disease (IED) among inpatient encounters in US hospitals.

METHODS

The PINC AI Healthcare Database (10/01/2015-03/31/2020) was used to assess the performance of six published code-based algorithms to identify IED cases among inpatient encounters. Case-confirmed IEDs were identified based on microbiological confirmation of E. coli in a normally sterile body site (Group 1) or in urine with signs of sepsis (Group 2). Code-based algorithm performance was assessed overall, and separately for Group 1 and Group 2 based on sensitivity, specificity, positive and negative predictive value (PPV and NPV) and F1 score. The improvement in performance of refinements to the best-performing algorithm was also assessed.

RESULTS

Among 2,595,983 encounters, 97,453 (3.8%) were case-confirmed IED (Group 1: 60.9%; Group 2: 39.1%). Across algorithms, specificity and NPV were excellent (>97%) for all but one algorithm, but there was a trade-off between sensitivity and PPV. The algorithm with the most balanced performance characteristics included diagnosis codes for: (1) infectious disease due to E. coli OR (2) sepsis/bacteremia/organ dysfunction combined with unspecified E. coli infection and no other concomitant non-E. coli invasive disease (sensitivity: 56.9%; PPV: 56.4%). Across subgroups, the algorithms achieved lower algorithm performance for Group 2 (sensitivity: 9.9%-61.1%; PPV: 3.8%-16.0%).

CONCLUSIONS

This study assessed code-based algorithms to identify IED during inpatient encounters in a large US hospital database. Such algorithms could be useful to identify IED in healthcare databases that lack information on microbiology data.

Effectively Leveraging RWD for External Controls: A Systematic Literature Review of Regulatory and HTA Decisions

Real-world data (RWD)-derived external controls can be used to contextualize efficacy findings for investigational therapies evaluated in uncontrolled trials. As the number of submissions to regulatory and health technology assessment (HTA) bodies using external controls rises, and in light of recent regulatory and HTA guidance on the appropriate use of RWD, there is a need to address the operational and methodological challenges impeding the quality of real-world evidence (RWE) generation and the consistency in evaluation of RWE across agencies. This systematic review summarizes publicly available information on the use of external controls to contextualize outcomes from uncontrolled trials for all indications from January 1, 2015, through August 20, 2021, that were submitted to the European Medicines Agency, the US Food and Drug Administration, and/or select major HTA bodies (National Institute for Health and Care Excellence (NICE), Haute Autorité de Santé (HAS), Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), and Gemeinsamer Bundesausschuss (G-BA)). By systematically reviewing submissions to regulatory and HTA bodies in the context of recent guidance, this study provides quantitative and qualitative insights into how external control design and analytic choices may be viewed by different agencies in practice. The primary operational and methodological aspects identified for discussion include, but are not limited to, engagement of regulators and HTA bodies, approaches to handling missing data (a component of data quality), and selection of real-world endpoints. Continued collaboration and guidance to address these and other aspects will inform and assist stakeholders attempting to generate evidence using external controls.

Tell me what you want, what you really really want: Estimands in observational pharmacoepidemiologic comparative effectiveness and safety studies

PURPOSE

Ideally, the objectives of a pharmacoepidemiologic comparative effectiveness or safety study should dictate its design and data analysis. This paper discusses how defining an estimand is instrumental to this process.

METHODS

We applied the ICH-E9 (Statistical Principles for Clinical Trials) R1 addendum on estimands - which originally focused on randomized trials - to three examples of observational pharmacoepidemiologic comparative effectiveness and safety studies. Five key elements specify the estimand: the population, contrasted treatments, endpoint, intercurrent events, and population-level summary measure.

RESULTS

Different estimands were defined for case studies representing three types of pharmacological treatments: (1) single-dose treatments using a case study about the effect of influenza vaccination versus no vaccination on mortality risk in an adult population of ≥60 years of age; (2) sustained-treatments using a case study about the effect of dipeptidyl peptidase 4 inhibitor versus glucagon-like peptide-1 agonist on hypoglycemia risk in treatment of uncontrolled diabetes; and (3) as needed treatments using a case study on the effect of nitroglycerin spray as-needed versus no nitroglycerin on syncope risk in treatment of stabile angina pectoris.

CONCLUSIONS

The case studies illustrated that a seemingly clear research question can still be open to multiple interpretations. Defining an estimand ensures that the study targets a treatment effect that aligns with the treatment decision the study aims to inform. Estimand definitions further help to inform choices regarding study design and data-analysis and clarify how to interpret study findings.

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS

Objectives

While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs.

Methods

We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences.

Results

Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Use of Real-World Data in Population Science to Improve the Prevention and Care of Diabetes-Related Outcomes

The past decade of population research for diabetes has seen a dramatic proliferation of the use of real-world data (RWD) and real-world evidence (RWE) generation from non-research settings, including both health and non-health sources, to influence decisions related to optimal diabetes care. A common attribute of these new data is that they were not collected for research purposes yet have the potential to enrich the information around the characteristics of individuals, risk factors, interventions, and health effects. This has expanded the role of subdisciplines like comparative effectiveness research and precision medicine, new quasi-experimental study designs, new research platforms like distributed data networks, and new analytic approaches for clinical prediction of prognosis or treatment response. The result of these developments is a greater potential to progress diabetes treatment and prevention through the increasing range of populations, interventions, outcomes, and settings that can be efficiently examined. However, this proliferation also carries an increased threat of bias and misleading findings. The level of evidence that may be derived from RWD is ultimately a function of the data quality and the rigorous application of study design and analysis. This report reviews the current landscape and applications of RWD in clinical effectiveness and population health research for diabetes and summarizes opportunities and best practices in the conduct, reporting, and dissemination of RWD to optimize its value and limit its drawbacks.

Engaging European society at the forefront of cancer research and care: How discussions at the 5th Gago Conference on European Science policy led to the Heidelberg Manifesto

European cancer research stakeholders met in October 2022 in Heidelberg, Germany, at the 5th Gago conference on European Cancer Policy, to discuss the current cancer research and cancer care policy landscape in Europe. Meeting participants highlighted gaps in the existing European programmes focusing on cancer research, including Europe's Beating Cancer Plan (EBCP), the Mission on Cancer (MoC), Understanding Cancer (UNCAN.eu), and the joint action CRANE, and put forward the next priorities, in the form of the Heidelberg Manifesto for cancer research. This meeting report presents all discussions that shed light on how infrastructures can be effectively shaped for translational, prevention, clinical and outcomes cancer research, with a focus on implementation and sustainability and while engaging patients and the public. In addition, we summarize recommendations on how to introduce frameworks for the digitalization of European cancer research. Finally, we discuss what structures, commitment, and resources are needed to establish a collaborative cancer research environment in Europe to achieve the scale required for innovation.

Strategies to Address Current Challenges in Real-World Evidence Generation in Japan

The generation of real-world evidence (RWE), which describes patient characteristics or treatment patterns using real-world data (RWD), is rapidly growing more popular as a tool for decision-making in Japan. The aim of this review was to summarize challenges to RWE generation in Japan related to pharmacoepidemiology, and to propose strategies to address some of these challenges. We first focused on data-related issues, including the lack of transparency of RWD sources, linkage across different care settings, definitions of clinical outcomes, and the overall assessment framework of RWD when used for research purposes. Next the study reviewed methodology-related challenges. As lack of design transparency impairs study reproducibility, transparent reporting of study design is critical for stakeholders. For this review, we considered different sources of biases and time-varying confounding, along with potential study design and methodological solutions. Additionally, the implementation of robust assessment of definition uncertainty, misclassification, and unmeasured confounders would enhance RWE credibility in light of RWD source-related limitations, and is being strongly considered by task forces in Japan. Overall, the development of guidance for best practices on data source selection, design transparency, and analytical methods to address different sources of biases and robustness in the process of RWE generation will enhance credibility for stakeholders and local decision-makers.

Confounding adjustment in the analysis of augmented randomized controlled trial with hybrid control arm

The augmented randomized controlled trial (RCT) with hybrid control arm includes a randomized treatment group (RT), a smaller randomized control group (RC), and a large synthetic control (SC) group from real-world data. This kind of trial is useful when there is logistics and ethics hurdle to conduct a fully powered RCT with equal allocation, or when it is necessary to increase the power of the RCT by incorporating real-world data. A difficulty in the analysis of augmented RCT is that the SC and RC may be systematically different in the distribution of observed and unmeasured confounding factors, causing bias when the two control groups are analyzed together as hybrid controls. We propose to use propensity score (PS) analysis to balance the observed confounders between SC and RC. The possible bias caused by unmeasured confounders can be estimated and tested by analyzing propensity score adjusted outcomes from SC and RC. We also propose a partial bias correction (PBC) procedure to reduce bias from unmeasured confounding. Extensive simulation studies show that the proposed PS + PBC procedures can improve the efficiency and statistical power by effectively incorporating the SC into the RCT data analysis, while still control the estimation bias and Type I error inflation that might arise from unmeasured confounding. We illustrate the proposed statistical procedures with data from an augmented RCT in oncology.

Effectiveness research in oncology with electronic health record data: A retrospective cohort study emulating the PALOMA-2 trial

PURPOSE

Oncology electronic health record (EHR) databases have increased in quality and availability over the past decade, yet it remains unclear whether these clinical practice data can be used to conduct reliable comparative effectiveness studies. We sought to emulate a clinical trial with EHR data in the advanced breast cancer population and compare our results against the trial.

METHODS

This cohort study used EHR data from US oncology practices. All elements of the study were defined to mimic the PALOMA-2 trial as closely as possible. Patients with hormone-positive, HER-2 negative metastatic breast cancer with no prior treatment for metastatic disease were included. Patients initiating palbociclib and letrozole on the same day following the earliest record of metastasis were compared to those initiating letrozole only. The primary associational measure was the conditional hazard ratio for time-to-next treatment (TTNT). TTNT is well-measured in our data source and amenable for calibration against the randomized study results of the PALOMA-2 trial. We used multiple imputation for several patient characteristics with missing values.

RESULTS

There were 3836 study-eligible women with advanced breast cancer. The hazard ratio for TTNT in the observational study (HR: 0.62; 95% CI: 0.56-0.68) was closely aligned with that of the randomized trial (HR: 0.64; 95% CI: 0.52-0.78).

CONCLUSIONS

Under our assumptions on missing data and comparability of the two study populations, results from our non-randomized study closely matched that of the randomized trial. Further studies are needed to determine whether EHR data can yield reliable conclusions on treatment effects in oncology.

Recording type 2 diabetes mellitus in a standardised central Saudi database: a retrospective validation study

OBJECTIVES

This study was conducted to assess the validity of recording (and the original diagnostic practice) of type 2 diabetes mellitus at a hospital whose records were integrated to a centralised database (the standardised common data model (CDM) of the Saudi National Pharmacoepidemiologic Database (NPED)).

DESIGN

A retrospective single-centre validation study.

SETTINGS

Data of the study participants were extracted from the CDM of the NPED (only records of one tertiary care hospital were integrated at the time of the study) between 1 January 2013 and 1 July 2018.

PARTICIPANTS

A random sample of patients with type 2 diabetes mellitus (≥18 years old and with a code of type 2 diabetes mellitus) matched with a control group (patients without diabetes) based on age and sex.

OUTCOME MEASURES

The standardised coding of type 2 diabetes in the CDM was validated by comparing the presence of diabetes in the CDM versus the original electronic records at the hospital, the recording in paper-based medical records, and the physician re-assessment of diabetes in the included cases and controls, respectively. Sensitivity, specificity, positive predictive value and negative predictive value were estimated for each pairwise comparison using RStudio V.1.4.1103.

RESULTS

A total of 437 random sample of patients with type 2 diabetes mellitus was identified and matched with 437 controls. Only 190 of 437 (43.0%) had paper-based medical records. All estimates were above 90% except for sensitivity and specificity of CDM versus paper-based records (54%; 95% CI 47% to 61% and 68%; 95% CI 62% to 73%, respectively).

CONCLUSIONS

This study provided an assessment to the extent of which only type 2 diabetes mellitus code can be used to identify patients with this disease at a Saudi centralised database. A future multi-centre study would help adding more emphasis to the study findings.

Transporting observational study results to a target population of interest using inverse odds of participation weighting

Inverse odds of participation weighting (IOPW) has been proposed to transport clinical trial findings to target populations of interest when the distribution of treatment effect modifiers differs between trial and target populations. We set out to apply IOPW to transport results from an observational study to a target population of interest. We demonstrated the feasibility of this idea with a real-world example using a nationwide electronic health record derived de-identified database from Flatiron Health. First, we conducted an observational study that carefully adjusted for confounding to estimate the treatment effect of fulvestrant plus palbociclib relative to letrozole plus palbociclib as a second-line therapy among estrogen receptor (ER)-positive, human epidermal growth factor receptor (HER2)-negative metastatic breast cancer patients. Second, we transported these findings to the broader cohort of patients who were eligible for a first-line therapy. The interpretation of the findings and validity of such studies, however, rely on the extent that causal inference assumptions are met.

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

OBJECTIVES

We developed guidance to inform decisions regarding the inclusion of nonrandomized studies of interventions (NRSIs) in systematic reviews (SRs) of the effects of interventions.

STUDY DESIGN AND SETTING

The guidance workgroup comprised SR experts and used an informal consensus generation method.

RESULTS

Instead of recommending NRSI inclusion only if randomized controlled trials (RCTs) are insufficient to address the SR key question, different topics may require different decisions regarding NRSI inclusion. We identified important considerations to inform such decisions from topic refinement through protocol development. During topic scoping and refinement, considerations were related to the clinical decisional dilemma, adequacy of RCTs to address the key questions, risk of bias in NRSIs, and the extent to which NRSIs are likely to complement RCTs. When NRSIs are included, during SR team formation, familiarity with topic-specific data sources and advanced analytic methods for NRSIs should be considered. During protocol development, the decision regarding NRSI inclusion or exclusion should be justified, and potential implications explained. When NRSIs are included, the protocol should describe the processes for synthesizing evidence from RCTs and NRSIs and determining the overall strength of evidence.

CONCLUSION

We identified specific considerations for decisions regarding NRSI inclusion in SRs and highlight the importance of flexibility and transparency.

Augmenting control arms with real-world data for cancer trials: Hybrid control arm methods and considerations

Background

Hybrid controlled trials with real-world data (RWD), where the control arm is composed of both trial and real-world patients, could facilitate research when the feasibility of randomized controlled trials (RCTs) is challenging and single-arm trials would provide insufficient information.

Methods

We propose a frequentist two-step borrowing method to construct hybrid control arms. We use parameters informed by a completed randomized trial in metastatic triple-negative breast cancer to simulate the operating characteristics of dynamic and static borrowing methods, highlighting key trade-offs and analytic decisions in the design of hybrid studies.

Results

Simulated data were generated under varying residual-bias assumptions (no bias: HR_RWD = 1) and experimental treatment effects (target trial scenario: HR_Exp = 0.78). Under the target scenario with no residual bias, all borrowing methods achieved the desired 88% power, an improvement over the reference model (74% power) that does not borrow information externally. The effective number of external events tended to decrease with higher bias between RWD and RCT (i.e. HR_RWD away from 1), and with weaker experimental treatment effects (i.e. HR_Exp closer to 1). All dynamic borrowing methods illustrated (but not the static power prior) cap the maximum Type 1 error over the residual-bias range considered. Our two-step model achieved comparable results for power, type 1 error, and effective number of external events borrowed compared to other borrowing methodologies.

Conclusion

By pairing high-quality external data with rigorous simulations, researchers have the potential to design hybrid controlled trials that better meet the needs of patients and drug development.

[Evidence-based health care with pharmaceuticals for rare diseases: the role of digitalisation]

Knowledge generation in the field of drug development for people with rare diseases (RDs) faces particular difficulties. This paper will show what improvements are expected from increasing digitalisation from the perspective of three healthcare institutions: the Federal Institute for Drugs and Medical Devices, the Institute for Quality and Efficiency in Health Care and the Federal Joint Committee.First, the potential of digitalisation to increase the efficiency of clinical development and regulatory decision-making through earlier collaboration of all stakeholders is proposed. Subsequently, it is argued that digitalisation should be used to reduce barriers to the implementation of care-associated randomised controlled trials, including those based on registries. High-quality registry studies should not only be started after approval but during the approval process, so that the evidence necessary for therapy decisions is available promptly after approval. Finally, it is stated that improving the evidence base through qualitative improvement of the data sources and their linkages directly benefits patients. Usable evidence that can be generated over a longer period of time - also beyond approval - and contribute to decisions within healthcare system ensures effective drug provision.The institutions agree that high-quality indication registries should be developed as product-independent, standing infrastructures so that high-quality data can be accessed early in the development of medicines for RD.

The Randomised Controlled Trial at the Intersection of Research Ethics and Innovation

The randomised controlled trial (RCT) has been considered for a long time as the gold standard for evidence generation to support regulatory decision making for medicines. The randomisation procedure involves an ethical dilemma since it means leaving the treatment choice to chance. Although currently contested, the ethical justification for the RCT that has gained widespread acceptance is the notion of 'clinical equipoise'. This state exists when "there is no consensus within the expert clinical community about the comparative merits of the alternatives to be tested"; it is argued that this confers the ethical grounds for the conduct of an RCT. The prominent position of the RCT is being challenged by new therapeutic modalities for which this study design may be unsuitable. Moreover, alternative approaches to evidence generation represent another area where innovation may have implications for the relevance of the RCT. Against the backdrop of the debate around the equipoise principle and some recent therapeutic and data analytical innovations, the aim of this article is to explore the current standing of the RCT from a regulatory perspective.

The challenges and opportunities in using real-world data to drive advances in healthcare in East Asia: expert panel recommendations

OBJECTIVE

To provide recommendations for overcoming the challenges associated with the generation and use of real-world evidence (RWE) in regulatory approvals, health technology assessments (HTAs), and reimbursement decision-making in East Asia.

METHODS

A panel of experts convened at the International Society for Pharmacoeconomics and Outcomes Research Asia Pacific 2020 congress to discuss the challenges limiting the use of RWE in healthcare decision-making and to provide insights into the perspectives of regulators, HTA agencies, the pharmaceutical industry, and physicians in China, Japan, and Taiwan. A nonsystematic literature review was conducted to expand on the themes addressed.

RESULTS

The use of RWE in regulatory approvals, HTAs, and reimbursement decision-making remains limited by legal/regulatory, technical, and attitudinal challenges in East Asia.

CONCLUSIONS

We recommend approaches and initiatives that aim to drive improvements in the utilization of RWE in healthcare decision-making in East Asia and other regions. We encourage large-scale collaborations that leverage the full range of skills offered by different stakeholders. Government agencies, hospitals, research organizations, patient groups, and the pharmaceutical industry must collaborate to ensure appropriate access to robust and reliable real-world data and seek alignment on how to address prioritized evidence needs. Increasingly, we believe that this work will be conducted by multidisciplinary teams with expertise in healthcare research and delivery, data science, and information technology. We hope this work will encourage further discussion among all stakeholders seeking to shape the RWE landscape in East Asia and other regions and drive next-generation healthcare.

Licensing of Orphan Medicinal Products—Use of Real-World Data and Other External Data on Efficacy Aspects in Marketing Authorization Applications Concluded at the European Medicines Agency Between 2019 and 2021

Background: Reference to so-called real-world data is more often made in marketing authorization applications for medicines intended to diagnose, prevent or treat rare diseases compared to more common diseases. We provide granularity on the type and aim of any external data on efficacy aspects from both real-world data sources and external trial data as discussed in regulatory submissions of orphan designated medicinal products in the EU. By quantifying the contribution of external data according to various regulatory characteristics, we aimed at identifying specific opportunities for external data in the field of orphan conditions.

Methods: Information on external data in regulatory documents covering 72 orphan designations was extracted. Our sample comprised public assessment reports for approved, refused, or withdrawn applications concluded from 2019–2021 at the European Medicines Agency. Products with an active orphan designation at the time of submission were scrutinized regarding the role of external data on efficacy aspects in the context of marketing authorization applications, or on the criterion of “significant benefit” for the confirmation of the orphan designation at the time of licensing. The reports allowed a broad distinction between clinical development, regulatory decision making, and intended post-approval data collection. We defined three categories of external data, administrative data, structured clinical data, and external trial data (from clinical trials not sponsored by the applicant), and noted whether external data concerned the therapeutic context of the disease or the product under review.

Results: While reference to external data with respect to efficacy aspects was included in 63% of the approved medicinal products in the field of rare diseases, 37% of marketing authorization applications were exclusively based on the dedicated clinical development plan for the product under review. Purely administrative data did not play any role in our sample of reports, but clinical data collected in a structured manner (from routine care or clinical research) were often used to inform on the trial design. Two additional recurrent themes for the use of external data were the contextualization of results, especially to confirm the orphan designation at the time of licensing, and reassurance of a large difference in treatment effect size or consistency of effects observed in clinical trials and practice. External data on the product under review were restricted to either active substances already belonging to the standard of care even before authorization or to compassionate use schemes. Furthermore, external data were considered pivotal for marketing authorization only exceptionally and only for active substances already in use within the specific therapeutic indication. Applications for the rarest conditions and those without authorized treatment alternatives were especially prominent with respect to the use of external data from real-world data sources both in the pre- and post-approval setting.

Conclusion: Specific opportunities for external data in the setting of marketing authorizations in the field of rare diseases were identified. Ongoing initiatives of fostering systematic data collection are promising steps for a more efficient medicinal product development in the field of rare diseases.

Real-World Evidence Prediction of a Phase IV Oncology Trial: Comparative Degarelix vs Leuprolide Safety

BACKGROUND

Medical and regulatory communities are increasingly interested in the utility of real-world evidence (RWE) for answering questions pertaining to drug safety and effectiveness but concerns about validity remain. A principled approach to conducting RWE studies may alleviate concerns and increase confidence in findings. This study sought to predict the findings from the PRONOUNCE trial using a principled approach to generating RWE.

METHODS

This propensity-score (PS) matched observational cohort study utilized 3 claims databases to compare the occurrence of major adverse cardiovascular events (MACE) among initiators of degarelix vs. leuprolide. Patients were included if they had history of prostate cancer and atherosclerotic cardiovascular disease. Subjects were excluded if they didn't have continuous database enrollment in the year prior to treatment initiation, were exposed to androgen deprivation therapy or experienced an acute cardiovascular event within 30 days prior to treatment initiation, or had a history or risk factors of QT prolongation.

RESULTS

There were 12,448 leuprolide and 1,969 degarelix study-eligible patients before matching, with 1,887 in each arm after PS-matching. The results for MACE comparing degarelix to leuprolide in the observational analysis (hazard ratio= 1.35; 95% confidence interval = 0.94-1.93) was consistent with the subsequently released PRONOUNCE result (hazard ratio = 1.28; 95% confidence interval = 0.59-2.79).

CONCLUSIONS

This study successfully predicted the result of a comparative cardiovascular safety trial in the oncology setting. Although the findings are encouraging, limitations of measuring cancer stage and tumor progression are representative of challenges in attempting to generalize whether claims-based RWE can be used as actionable evidence.

A Review of Causal Inference for External Comparator Arm Studies

Randomized controlled trials (RCTs) are the gold standard design to establish the efficacy of new drugs and to support regulatory decision making. However, a marked increase in the submission of single-arm trials (SATs) has been observed in recent years, especially in the field of oncology due to the trend towards precision medicine contributing to the rise of new therapeutic interventions for rare diseases. SATs lack results for control patients, and information from external sources can be compiled to provide context for better interpretability of study results. External comparator arm (ECA) studies are defined as a clinical trial (most commonly a SAT) and an ECA of a comparable cohort of patients-commonly derived from real-world settings including registries, natural history studies, or medical records of routine care. This publication aims to provide a methodological overview, to sketch emergent best practice recommendations and to identify future methodological research topics. Specifically, existing scientific and regulatory guidance for ECA studies is reviewed and appropriate causal inference methods are discussed. Further topics include sample size considerations, use of estimands, handling of different data sources regarding differential baseline covariate definitions, differential endpoint measurements and timings. In addition, unique features of ECA studies are highlighted, specifically the opportunity to address bias caused by unmeasured ECA covariates, which are available in the SAT.

Treatment effectiveness in a rare oncology indication: Lessons from an external control cohort study

Real-world data (RWD) reflecting patient treatment in routine clinical practice can be used to develop external control groups for single-arm trials. External controls can provide valuable benchmark results on potential comparator drug effectiveness, particularly in rare indications when randomized controlled trials are either infeasible or unethical. This paper describes lessons learned from a descriptive real-world external control cohort study conducted to provide benchmark data for a single-arm clinical trial in a rare oncology biomarker driven disease. Conducting external control cohort studies to evaluate treatment effectiveness in rare indications likely will present with data and analysis challenges as seen in the example study. However, there are mitigating measures that can be applied in the study design, identification of RWD sources, and data analysis. The lessons learned and reported here with a proposal of an external control study framework can provide guidance for future research in this area, and may be applicable as well in other rare indications. Taking these learnings into consideration, the use of real-world external controls to contextualize treatment effectiveness in rare indications is a valuable approach and warrants further application in the future.

Using observational study data as an external control group for a clinical trial: an empirical comparison of methods to account for longitudinal missing data

Background

Observational data are increasingly being used to conduct external comparisons to clinical trials. In this study, we empirically examined whether different methodological approaches to longitudinal missing data affected study conclusions in this setting.

Methods

We used data from one clinical trial and one prospective observational study, both Norwegian multicenter studies including patients with recently diagnosed rheumatoid arthritis and implementing similar treatment strategies, but with different stringency. A binary disease remission status was defined at 6, 12, and 24 months in both studies. After identifying patterns of longitudinal missing outcome data, we evaluated the following five approaches to handle missingness: analyses of patients with complete follow-up data, multiple imputation (MI), inverse probability of censoring weighting (IPCW), and two combinations of MI and IPCW.

Results

We found a complex non-monotone missing data pattern in the observational study (N = 328), while missing data in the trial (N = 188) was monotone due to drop-out. In the observational study, only 39.0% of patients had complete outcome data, compared to 89.9% in the trial. All approaches to missing data indicated favorable outcomes of the treatment strategy in the trial and resulted in similar study conclusions. Variations in results across approaches were mainly due to variations in estimated outcomes for the observational data.

Conclusions

Five different approaches to handle longitudinal missing data resulted in similar conclusions in our example. However, the extent and complexity of missing observational data affected estimated comparative outcomes across approaches, highlighting the need for careful consideration of methods to account for missingness in this setting. Based on this empirical examination, we recommend using a prespecified advanced missing data approach to account for longitudinal missing data, and to conduct alternative approaches in sensitivity analyses.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01639-0.

Agreement of treatment effects from observational studies and randomized controlled trials evaluating hydroxychloroquine, lopinavir-ritonavir, or dexamethasone for covid-19: meta-epidemiological study

OBJECTIVE

To systematically identify, match, and compare treatment effects and study demographics from individual or meta-analysed observational studies and randomized controlled trials (RCTs) evaluating the same covid-19 treatments, comparators, and outcomes.

DESIGN

Meta-epidemiological study.

DATA SOURCES

National Institutes of Health Covid-19 Treatment Guidelines, a living review and network meta-analysis published in The BMJ, a living systematic review with meta-analysis and trial sequential analysis in PLOS Medicine (The LIVING Project), and the Epistemonikos "Living OVerview of Evidence" (L·OVE) evidence database.

ELIGIBILITY CRITERIA FOR SELECTION OF STUDIES

RCTs in The BMJ's living review that directly compared any of the three most frequently studied therapeutic interventions for covid-19 across all data sources (that is, hydroxychloroquine, lopinavir-ritonavir, or dexamethasone) for any safety and efficacy outcomes. Observational studies that evaluated the same interventions, comparisons, and outcomes that were reported in The BMJ's living review.

DATA EXTRACTION AND SYNTHESIS

Safety and efficacy outcomes from observational studies were identified and treatment effects for dichotomous (odds ratios) or continuous (mean differences or ratios of means) outcomes were calculated and, when possible, meta-analyzed to match the treatment effects from individual RCTs or meta-analyses of RCTs reported in The BMJ's living review with the same interventions, comparisons, and outcomes (that is, matched pairs). The analysis compared the distribution of study demographics and the agreement between treatment effects from matched pairs. Matched pairs were in agreement if both observational and RCT treatment effects were significantly increasing or decreasing (P<0.05) or if both treatment effects were not significant (P≥0.05).

RESULTS

17 new, independent meta-analyses of observational studies were conducted that compared hydroxychloroquine, lopinavir-ritonavir, or dexamethasone with an active or placebo comparator for any safety or efficacy outcomes in covid-19 treatment. These studies were matched and compared with 17 meta-analyses of RCTs reported in The BMJ's living review. 10 additional matched pairs with only one observational study and/or one RCT were identified. Across all 27 matched pairs, 22 had adequate reporting of demographical and clinical data for all individual studies. All 22 matched pairs had studies with overlapping distributions of sex, age, and disease severity. Overall, 21 (78%) of the 27 matched pairs had treatment effects that were in agreement. Among the 17 matched pairs consisting of meta-analyses of observational studies and meta-analyses of RCTs, 14 (82%) were in agreement; seven (70%) of the 10 matched pairs consisting of at least one observational study or one RCT were in agreement. The 18 matched pairs with treatment effects for dichotomous outcomes had a higher proportion of agreement (n=16, 89%) than did the nine matched pairs with treatment effects for continuous outcomes (n=5, 56%).

CONCLUSIONS

Meta-analyses of observational studies and RCTs evaluating treatments for covid-19 have summary treatment effects that are generally in agreement. Although our evaluation is limited to three covid-19 treatments, these findings suggest that meta-analyzed evidence from observational studies might complement, but should not replace, evidence collected from RCTs.

Dabigatran versus vitamin K antagonists for atrial fibrillation in clinical practice: final outcomes from Phase III of the GLORIA-AF registry

BACKGROUND

Prospectively collected, routine clinical practice-based data on antithrombotic therapy in non-valvular atrial fibrillation (AF) patients are important for assessing real-world comparative outcomes. The objective was to compare the safety and effectiveness of dabigatran versus vitamin K antagonists (VKAs) in patients with newly diagnosed AF.

METHODS AND RESULTS

GLORIA-AF is a large, prospective, global registry program. Consecutive patients with newly diagnosed AF and CHA2DS2-VASc scores ≥ 1 were included and followed for 3 years. To control for differences in patient characteristics, the comparative analysis for dabigatran versus VKA was performed on a propensity score (PS)-matched patient set. Missing data were multiply imputed. Proportional-hazards regression was used to estimate hazard ratios (HRs) for outcomes of interest. Between 2014 and 2016, 21,300 eligible patients were included worldwide: 3839 patients were prescribed dabigatran and 4836 VKA with a median age of 71.0 and 72.0 years, respectively; > 85% in each group had a CHA2DS2-VASc-score ≥ 2. The PS-matched comparative analysis for dabigatran and VKA included on average 3326 pairs of matched initiators. For dabigatran versus VKAs, adjusted HRs (95% confidence intervals) were: stroke 0.89 (0.59-1.34), major bleeding 0.61 (0.42-0.88), all-cause death 0.78 (0.63-0.97), and myocardial infarction 0.89 (0.53-1.48). Further analyses stratified by PS and region provided similar results.

CONCLUSIONS

Dabigatran was associated with a 39% reduced risk of major bleeding and 22% reduced risk for all-cause death compared with VKA. Stroke and myocardial infarction risks were similar, confirming a more favorable benefit-risk profile for dabigatran compared with VKA in clinical practice. Clinical trial registration https://www.

CLINICALTRIALS

gov . NCT01468701, NCT01671007.

Regulatory Approval With Real-World Data From Regulatory Science Perspective in Japan

Recently, there has been a growing trend in clinical development to utilize real-world data (RWD) to improve the efficiency of drug/medical device development. Especially, the use of RWD to generate real-world evidence (RWE) in regulatory approval is currently undergoing a period of great change with an increasing degree of active discussion. In Japan, RWE has been used in the control arms of clinical trials, observational studies, post-marketing surveillance, and public knowledge-based applications for regulatory approval. However, the exclusive use of RWE applications has still not been applied. In this paper, we summarize the history and the current situation of RWE and focus on the utilization for the purpose of regulatory approval. In addition, we will discuss the issues and perspectives for registry research in the utilization for regulatory approval in Japan.

Applying the OMOP Common Data Model to Facilitate Benefit-Risk Assessments of Medicinal Products Using Real-World Data from Singapore and South Korea

OBJECTIVES

The aim of this study was to characterize the benefits of converting Electronic Medical Records (EMRs) to a common data model (CDM) and to assess the potential of CDM-converted data to rapidly generate insights for benefit-risk assessments in post-market regulatory evaluation and decisions.

METHODS

EMRs from January 2013 to December 2016 were mapped onto the Observational Medical Outcomes Partnership-CDM (OMOP-CDM) schema. Vocabulary mappings were applied to convert source data values into OMOP-CDM-endorsed terminologies. Existing analytic codes used in a prior OMOP-CDM drug utilization study were modified to conduct an illustrative analysis of oral anticoagulants used for atrial fibrillation in Singapore and South Korea, resembling a typical benefit-risk assessment. A novel visualization is proposed to represent the comparative effectiveness, safety and utilization of the drugs.

RESULTS

Over 90% of records were mapped onto the OMOP-CDM. The CDM data structures and analytic code templates simplified the querying of data for the analysis. In total, 2,419 patients from Singapore and South Korea fulfilled the study criteria, the majority of whom were warfarin users. After 3 months of follow-up, differences in cumulative incidence of bleeding and thromboembolic events were observable via the proposed visualization, surfacing insights as to the agent of preference in a given clinical setting, which may meaningfully inform regulatory decision-making.

CONCLUSIONS

While the structure of the OMOP-CDM and its accessory tools facilitate real-world data analysis, extending them to fulfil regulatory analytic purposes in the post-market setting, such as benefit-risk assessments, may require layering on additional analytic tools and visualization techniques.

Linking the Paul Coverdell National Acute Stroke Program to commercial claims to establish a framework for real-world longitudinal stroke research

BACKGROUND

Non-interventional large-scale research on real-world patients who had a stroke requires the use of multiple data sources ensuring access to longitudinal data from large populations with clinically-detailed information. We sought to establish a framework for longitudinal research on patients hospitalised with stroke by linking information-rich, deidentified inpatient data from the Paul Coverdell National Acute Stroke Program (PCNASP) to commercial and Medicare Advantage longitudinal claims data.

METHODS

All stroke admissions in PCNASP between 2008 and 2015 were evaluated for linkage to longitudinal claims from a commercial insurer using an algorithm based on six available common data fields (patient age, gender, admission date, discharge date, discharge diagnosis and state) and a hospital match. We evaluated the linkage quality (via the percentage of unique records in the linked dataset) and the representativeness of the linked population. We also described medical history, stroke severity and patterns of medication use among the PCNASP-claims linked cohort.

RESULTS

The linkage produced uniqueness equal to 99.1%. We identified 5644 linked and 98 896 unlinked patients who had an ischaemic stroke hospitalisation in claims data. Linked patients were younger than unlinked (69.7 vs 72.5 years), but otherwise similar by medical history, prestroke medication use or lab values. Stroke severity was mild and most patients were discharged home. Prestroke and discharge use of antihypertensive and statins in the PCNASP were greater than their use as measured by filled prescriptions in claims.

CONCLUSIONS

High-quality linkage between the PCNASP and commercial claims data is feasible. This linkage identified differences between reported or recommended versus actual out-of-hospital medication utilisation, highlighting the importance of longitudinal data availability for research aimed to improve the care of patients who had a stroke.

Safety Profile of Antipsychotic Drugs: Analysis Based on a Provincial Spontaneous Reporting Systems Database

Introduction: Antipsychotic drugs are the main therapy for schizophrenia and have been widely used in mental disorder fields. However, the research on the safety of antipsychotic drugs in the real-world is rare. The purpose of this research is to evaluate the safety of antipsychotic drugs based on real-world data. Methods: ADR reports collected by the Henan Adverse Drug Reaction Monitoring Center from 2016 to 2020 were analyzed. We described the safety of antipsychotic drugs by descriptive analysis and four signal mining methods. Meanwhile, the risk factors for serious adverse reactions of antipsychotics were identified. Results: A total of 3363 ADR reports related to antipsychotics were included. We found that the number of adverse drug reaction reports and the proportion of serious adverse reactions have increased year by year from 2016 to 2020. Most adverse drug reactions occurred within 3 months after taking the medicine. The symptoms caused by typical antipsychotics and atypical antipsychotics were different and dyskinesia was more common in typical antipsychotics. Most patients improved or recovered after treatment or intervention while only one patient had sequelae. Low-level hospitals, psychiatric hospitals, youth, and old age could increase the risk of serious adverse reactions. Four off-label signals were found through signal mining, including amisulpride-pollakiuria, ziprasidone-dyspnoea, quetiapine-urinary incontinence, olanzapine-hepatic function abnormal. Conclusion: We found that most ADRs occurred within 3 months after taking the medicine, so close observation was required for patients during the first 3 months of treatment. The ADRs of antipsychotics involved multiple organ-system damages but were not serious. It might be recommended to take alternative drugs after a serious ADR occurred. The symptoms caused by typical APDs and atypical APDs were different. For patients with typical APDs, dyskinesia was more common and should be given special attention. Statistics showed that low-level hospitals, psychiatric hospitals, youth, and old age were risk factors for serious ADRs. The four off-label signals obtained by signal mining should be paid special attention, including amisulpride-pollakiuria, ziprasidone-dyspnoea, quetiapine-urinary incontinence, and olanzapine-hepatic function abnormal.

Solving the Evidence Interpretability Crisis in Health Technology Assessment: A Role for Mechanistic Models?

Health technology assessment (HTA) aims to be a systematic, transparent, unbiased synthesis of clinical efficacy, safety, and value of medical products (MPs) to help policymakers, payers, clinicians, and industry to make informed decisions. The evidence available for HTA has gaps-impeding timely prediction of the individual long-term effect in real clinical practice. Also, appraisal of an MP needs cross-stakeholder communication and engagement. Both aspects may benefit from extended use of modeling and simulation. Modeling is used in HTA for data-synthesis and health-economic projections. In parallel, regulatory consideration of model informed drug development (MIDD) has brought attention to mechanistic modeling techniques that could in fact be relevant for HTA. The ability to extrapolate and generate personalized predictions renders the mechanistic MIDD approaches suitable to support translation between clinical trial data into real-world evidence. In this perspective, we therefore discuss concrete examples of how mechanistic models could address HTA-related questions. We shed light on different stakeholder's contributions and needs in the appraisal phase and suggest how mechanistic modeling strategies and reporting can contribute to this effort. There are still barriers dissecting the HTA space and the clinical development space with regard to modeling: lack of an adapted model validation framework for decision-making process, inconsistent and unclear support by stakeholders, limited generalizable use cases, and absence of appropriate incentives. To address this challenge, we suggest to intensify the collaboration between competent authorities, drug developers and modelers with the aim to implement mechanistic models central in the evidence generation, synthesis, and appraisal of HTA so that the totality of mechanistic and clinical evidence can be leveraged by all relevant stakeholders.

Real-world data in drug development strategies for orphan drugs: tafasitamab in B cell lymphoma, a case study for approval based on a single-arm combination trial

Tafasitamab (TAF) plus lenalidomide (LEN) is a novel treatment option for patients with relapsed/refractory diffuse large B cell lymphoma (rrDLBCL) who are not eligible for autologous stem cell transplantation. The initial US/EU approvals for TAF represent precedents because this is the first time that approval of a novel combination therapy was granted based on a pivotal single-arm trial (SAT). Matching real world-data (RWD) helped to disentangle the contribution of individual agents. In this review, we present the TAF development strategy, the prospective incorporation of RWD within the clinical development plan, the corresponding regulatory hurdles of this strategy, and the prior regulatory actions for other cancer drugs that previously incorporated RWD and propensity score matching in EU and US regulatory submissions. We also outline how RWD could further advance and impact orphan drug development.

Integrative Analysis of Randomized Clinical Trial and Observational Study Data to Inform Post-marketing Safety Decision-Making

Safety evaluation is a continual and iterative process throughout the drug development life cycle and requires long time horizons and large amounts of data to fully understand the safety profile of a medical product. Although randomized clinical trials (RCT) provide high-quality data for an initial assessment of safety signals, the safety signals may not all have been known at the time of approval because safety data collected from RCT only involve a relatively small number of subjects during a relatively short follow-up period. The increased accumulation of post-marketing real-world data (RWD) presents an opportunity to utilize them for safety decision-making; these include identifying new safety signals, further characterization of safety concerns that are raised in pre-marketing RCT, and further generalization of RCT findings to the broader patient populations not previously studied in RCT. In this paper, we use cardiovascular safety outcome trial for antidiabetic therapies as an illustrative example and discuss how integrative analysis of RCT and observational study data can answer regulatory concerns about cardiovascular risk in a post-marketing setting. A novel statistical analysis strategy is proposed to combine both sources of safety data in a data fusion approach. The proposed approach includes three stages: (1) feasibility analysis that uses an RCT to validate an observational study, applying estimand framework and emulating RCT with RWD; (2) integrative analysis that combines evidence from the RCT and observational study data cooperatively; and (3) sensitivity analysis that examines the consistency of the previous analyses. Two potential utilities of the proposed integrative analysis for the cardiovascular safety outcome trial are discussed.

Real-world effectiveness and safety of golimumab in rheumatoid arthritis treatment: A two-center study in Taiwan

BACKGROUND

The real-world outcomes of golimumab (GLM) use have been rarely studied in Asian patients with rheumatoid arthritis (RA). This study assessed the real-world effectiveness and safety of GLM in a Taiwanese cohort.

METHODS

One hundred and eight GLM-treated RA patients were enrolled. Predictors of a good European League Against Rheumatism (EULAR) response at 24 months and drug retention were identified through multivariate analyses.

RESULTS

After 24 months of GLM treatment, the mean Disease Activity Score using 28 joint counts with the erythrocyte sedimentation rate (DAS28-ESR) decreased from 6.7 to 3.1 (p < 0.001). Up to 58.9% of patients achieved a good EULAR response at 24 months. Multivariate logistic regression analysis revealed that after adjustment for other variables, a higher baseline C-reactive protein was an independent negative predictor of good EULAR responses (odds ratio, 0.82; 95% confidence interval [CI], 0.67-0.99; p = 0.043). During the mean follow-up period of 38.3 months, 15 (13.9%) patients discontinued GLM due to treatment failure. In multivariate analysis, high baseline ESR level, high DAS28-ESR, and the experience of biologic therapy were independent risk factors for GLM discontinuation (adjusted hazard ratio [HR], 1.03; 95% CI, 1.01-1.05; p = 0.003; adjusted HR, 2.93; 95% CI, 1.42-6.08; p = 0.004; and adjusted HR, 5.00; 95% CI, 1.75-14.26; p = 0.003, respectively). In receiver operator characteristic curve analysis, the optimal cutoff values of baseline ESR and DAS28-ESR for predicting drug survival were 52 mm/h (sensitivity: 60.0% and specificity: 77.4%) and 7.7 (sensitivity: 46.7% and specificity: 94.3%), respectively. During the follow-up period, 22 patients (20.4%) developed adverse events. The safety profile of GLM in this study was comparable with that in previous clinical trials.

CONCLUSION

GLM was effective and safe for the real-life management of Taiwanese RA patients and showed a high retention rate in biologic-naive patients compared with biologic-experienced patients.

Overall survival in the real-world and clinical trials: a case study validating external controls in advanced melanoma

Aims: We assessed the suitability of real-world data (RWD) as an external control for analysis of overall survival (OS) compared with clinical trial data (CTD) in advanced melanoma. Methods: OS among adults receiving ipilimumab for advanced melanoma was compared between trials (CTD group) and the Flatiron Health database (RWD group) using Cox models. Adjusted analyses accounted for differences in baseline factors; missing data were addressed through multiple imputation. Results: After adjusting for baseline factors and accounting for missingness, OS was similar in the CTD (n = 241) versus RWD groups (n = 816) (hazard ratio: 0.98; 95% CI: 0.75-1.26). Conclusion: Flatiron Health data is suitable to construct external control groups for OS in advanced melanoma trials after adjusting for baseline factors and missing data.

Application of Real-World Data to External Control Groups in Oncology Clinical Trial Drug Development

Randomized controlled trials (RCTs) that assess overall survival are considered the "gold standard" when evaluating the efficacy and safety of a new oncology intervention. However, single-arm trials that use surrogate endpoints (e.g., objective response rate or duration of response) to evaluate clinical benefit have become the basis for accelerated or breakthrough regulatory approval of precision oncology drugs for cases where the target and research populations are relatively small. Interpretation of efficacy in single-arm trials can be challenging because such studies lack a standard-of-care comparator arm. Although an external control group can be based on data from other clinical trials, using an external control group based on data collected outside of a trial may not only offer an alternative to both RCTs and uncontrolled single-arm trials, but it may also help improve decision-making by study sponsors or regulatory authorities. Hence, leveraging real-world data (RWD) to construct external control arms in clinical trials that investigate the efficacy and safety of drug interventions in oncology has become a topic of interest. Herein, we review the benefits and challenges associated with the use of RWD to construct external control groups, and the relevance of RWD to early oncology drug development.

When Can We Rely on Real-World Evidence to Evaluate New Medical Treatments?

Concerns regarding both the limited generalizability and the slow pace of traditional randomized trials have led to calls for greater use of real-world evidence (RWE) in the evaluation of new treatments or products. The RWE label has been used to refer to a variety of departures from the methods of traditional randomized controlled trials. Recognizing this complexity and potential confusion, the National Academies of Science, Engineering, and Medicine convened a series of workshops to clarify and address questions regarding the use of RWE to evaluate new medical treatments. Those workshops identified three specific dimensions in which RWE studies might differ from traditional clinical trials: use of real-world data (data extracted from health system records or data captured by mobile devices), delivery of real-world treatment (open-label treatments delivered in community settings by community practitioners), and real-world treatment assignment (including nonrandomized comparisons and variations on random assignment such as before-after or stepped-wedge designs). For any RWE study, decisions regarding each of these dimensions depends on the specific research question, characteristics of the potential study settings, and characteristics of the settings where study results would be applied.

Cytarabine dose reduction in patients with low-risk acute myeloid leukemia: A report from the Children's Oncology Group

BACKGROUND

The optimal number of chemotherapy courses for low-risk (LR) pediatric acute myeloid leukemia (AML) is not known.

OBJECTIVE

To compare outcomes for four (21.6 g/m² cytarabine) versus five (45.6 g/m² cytarabine) chemotherapy courses for LR-AML using data from Children's Oncology Group (COG) AAML0531 and AAML1031.

METHODS

We compared relapse risk (RR), disease-free survival (DFS), and overall survival (OS), and the differential impact in LR subgroups for patients receiving four versus five chemotherapy courses. Cox (OS and DFS) and risk (RR) regressions were used to estimate hazard ratios (HR) to compare outcomes.

RESULTS

A total of 923 LR-AML patients were included; 21% received five courses. Overall, LR-AML patients who received four courses had higher RR (40.9% vs. 31.4%; HR = 1.40, 95% confidence interval [CI]: 1.06-1.85), and worse DFS (56.0% vs. 67.0%; HR = 1.45, 95% CI: 1.10-1.91). There was a similar decrement in OS though it was not statistically significant (77.0% vs. 83.5%; HR = 1.45, 95% CI: 0.97-2.17). Stratified analyses revealed the detrimental effects of cytarabine dose de-escalation to be most pronounced in the LR-AML subgroup with uninformative cytogenetic/molecular features who were minimal residual disease (MRD) negative after the first induction course (EOI1). The absolute decrease in DFS with four courses for patients with favorable cytogenetic/molecular features and positive MRD was similar to that observed for patients with uninformative cytogenetic/molecular features and negative MRD at EOI1, though not statistically significant.

CONCLUSIONS

Our results support de-escalation of cytarabine exposure through the elimination of a fifth chemotherapy course only for LR-AML patients who have both favorable cytogenetic/molecular features and negative MRD after the first induction cycle.

Unicompartmental compared with total knee replacement for patients with multimorbidities: a cohort study using propensity score stratification and inverse probability weighting

BACKGROUND

Although routine NHS data potentially include all patients, confounding limits their use for causal inference. Methods to minimise confounding in observational studies of implantable devices are required to enable the evaluation of patients with severe systemic morbidity who are excluded from many randomised controlled trials.

OBJECTIVES

Stage 1 - replicate the Total or Partial Knee Arthroplasty Trial (TOPKAT), a surgical randomised controlled trial comparing unicompartmental knee replacement with total knee replacement using propensity score and instrumental variable methods. Stage 2 - compare the risk benefits and cost-effectiveness of unicompartmental knee replacement with total knee replacement surgery in patients with severe systemic morbidity who would have been ineligible for TOPKAT using the validated methods from stage 1.

DESIGN

This was a cohort study.

SETTING

Data were obtained from the National Joint Registry database and linked to hospital inpatient (Hospital Episode Statistics) and patient-reported outcome data.

PARTICIPANTS

Stage 1 - people undergoing unicompartmental knee replacement surgery or total knee replacement surgery who met the TOPKAT eligibility criteria. Stage 2 - participants with an American Society of Anesthesiologists grade of ≥ 3.

INTERVENTION

The patients were exposed to either unicompartmental knee replacement surgery or total knee replacement surgery.

MAIN OUTCOME MEASURES

The primary outcome measure was the postoperative Oxford Knee Score. The secondary outcome measures were 90-day postoperative complications (venous thromboembolism, myocardial infarction and prosthetic joint infection) and 5-year revision risk and mortality. The main outcome measures for the health economic analysis were health-related quality of life (EuroQol-5 Dimensions) and NHS hospital costs.

RESULTS

In stage 1, propensity score stratification and inverse probability weighting replicated the results of TOPKAT. Propensity score adjustment, propensity score matching and instrumental variables did not. Stage 2 included 2256 unicompartmental knee replacement patients and 57,682 total knee replacement patients who had severe comorbidities, of whom 145 and 23,344 had linked Oxford Knee Scores, respectively. A statistically significant but clinically irrelevant difference favouring unicompartmental knee replacement was observed, with a mean postoperative Oxford Knee Score difference of < 2 points using propensity score stratification; no significant difference was observed using inverse probability weighting. Unicompartmental knee replacement more than halved the risk of venous thromboembolism [relative risk 0.33 (95% confidence interval 0.15 to 0.74) using propensity score stratification; relative risk 0.39 (95% confidence interval 0.16 to 0.96) using inverse probability weighting]. Unicompartmental knee replacement was not associated with myocardial infarction or prosthetic joint infection using either method. In the long term, unicompartmental knee replacement had double the revision risk of total knee replacement [hazard ratio 2.70 (95% confidence interval 2.15 to 3.38) using propensity score stratification; hazard ratio 2.60 (95% confidence interval 1.94 to 3.47) using inverse probability weighting], but half of the mortality [hazard ratio 0.52 (95% confidence interval 0.36 to 0.74) using propensity score stratification; insignificant effect using inverse probability weighting]. Unicompartmental knee replacement had lower costs and higher quality-adjusted life-year gains than total knee replacement for stage 2 participants.

LIMITATIONS

Although some propensity score methods successfully replicated TOPKAT, unresolved confounding may have affected stage 2. Missing Oxford Knee Scores may have led to information bias.

CONCLUSIONS

Propensity score stratification and inverse probability weighting successfully replicated TOPKAT, implying that some (but not all) propensity score methods can be used to evaluate surgical innovations and implantable medical devices using routine NHS data. Unicompartmental knee replacement was safer and more cost-effective than total knee replacement for patients with severe comorbidity and should be considered the first option for suitable patients.

FUTURE WORK

Further research is required to understand the performance of propensity score methods for evaluating surgical innovations and implantable devices.

TRIAL REGISTRATION

This trial is registered as EUPAS17435.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 66. See the NIHR Journals Library website for further project information.

Use of Real-World Data and Evidence in Drug Development of Medicinal Products Centrally Authorized in Europe in 2018-2019

Real‐world data/real‐world evidence (RWD/RWE) are considered to have a great potential to complement, in some cases, replace the evidence generated through randomized controlled trials. By tradition, use of RWD/RWE in the postauthorization phase is well‐known, whereas published evidence of use in the pre‐authorization phase of medicines development is lacking. The primary aim of this study was to identify and quantify the role of potential use of RWD/RWE (RWE signatures) during the pre‐authorization phase, as presented in the initial marketing authorization applications of new medicines centrally evaluated with a positive opinion in 2018–2019 (n = 111) by the European Medicines Agency (EMA). Data for the study was retrieved from the evaluation overviews of the European Public Assessment Reports (EPARs), which reflect the scientific conclusions of the assessment process and are accessible through the EMA website. RWE signatures were extracted into an RWE Data Matrix, including 11 categories divided over 5 stages of the drug development lifecycle. Nearly all EPARs included RWE signatures for the discovery (98.2%) and life‐cycle management (100.0%). Half of them included RWE signatures for the full development phase (48.6%) and for supporting regulatory decisions at the registration (46.8%), whereas over a third (35.1%) included RWE signatures for the early development. RWE signatures were more often seen for orphan and conditionally approved medicines. Oncology, hematology, and anti‐infectives stood out as therapeutic areas with most RWE signatures in their full development phase. The findings bring unprecedented insights about the vast use of RWD/RWE in drug development supporting the regulatory decision making.

Marketing Authorization Applications Made to the European Medicines Agency in 2018-2019: What was the Contribution of Real-World Evidence?

Information derived from routinely collected real‐world data has for a long time been used to support regulatory decision making on the safety of drugs and has more recently been used to support marketing authorization submissions to regulators. There is a lack of detailed information on the use and types of this real‐world evidence (RWE) as submitted to regulators. We used resources held by the European Medicines Agency (EMA) to describe the characteristics of RWE included in new marketing authorization applications (MAAs) and extensions of indication (EOIs) for already authorized products submitted to the EMA in 2018 and 2019. For MAAs, 63 of 158 products (39.9%) contained RWE with a total of 117 studies. For 31.7% of these products, the RWE submitted was derived from data collected before the planned authorization. The most common data sources were registries (60.3%) followed by hospital data (31.7%). RWE was mainly included to support safety (87.3%) and efficacy (49.2%) with cohort studies being the most frequently used study design (88.9%). For EOIs, 28 of 153 products (18.3%) contained RWE with a total of 36 studies. For 57.1% of these products, studies were conducted prior to the EOIs. RWE sources were mainly registries (35.6%) and hospital data (27.0%). RWE was typically used to support safety (82.1%) and efficacy (53.6%). Cohort studies were the most commonly used study design (87.6%). We conclude that there is widespread use of RWE to support evaluation of MAAs and EOIs submitted to the EMA and identify areas where further research is required.