Improving Transparency to Build Trust in Real-World Secondary Data Studies for Hypothesis Testing-Why, What, and How: Recommendations and a Road Map from the Real-World Evidence Transparency Initiative

Generating real-world evidence on diabetes technology using the CareLink Personal data management system

AIM

To establish trust in real-world evidence (RWE) derived from CareLink Personal (CP), Medtronic's data management system for MiniMed system users, we show that this database and its analyses strictly adhere to the principles of RWE.

METHODS

The methodology is applicable to all MiniMed iterations. We described every step from raw data to predefined outcomes. In addition, we showed CP's fitness-for-research by the below metrics (using last year's MiniMed 780G system data as a case study): representative population, relevant endpoints, appropriate granularity, high data completeness, high data representativity and consistency in results.

RESULTS

The process from raw data to outcomes has been validated, and metrics/logics adhere to established definitions. Over 95% of users have a CP account; with 96% providing consent, this allows the use of >91% of the census population. There is no rationale for an over-representation of a specific phenotype among users not included. CP includes >50 endpoints, including 'International Consensus on Time in Range' based metrics. Data are recorded at 5-min intervals (maximum 288 per day), and on average there were 263 data points per person per day. Ninety-nine per cent of uploads were automated. For the last year, only 1 in 6 users had a data gap >1 day, and 1 in 50 had a gap >1 week. The time in range from in-silico studies was similar to that of real-world studies from different geographies and with ever growing populations.

CONCLUSION

RWE from CP adheres to the principles of RWE and can serve as robust evidence on the performance and safety of MiniMed systems.

FOUNTAIN: a modular research platform for integrated real-world evidence generation

BACKGROUND

Real-world evidence (RWE) plays a key role in regulatory and healthcare decision-making, but the potentially fragmentated nature of generated evidence may limit its utility for clinical decision-making. Heterogeneity and a lack of reproducibility in RWE resulting from inconsistent application of methodologies across data sources should be minimized through harmonization.

METHODS

This paper's aim is to describe and reflect upon a multidisciplinary research platform (FOUNTAIN; FinerenOne mUlti-database NeTwork for evidence generAtIoN) with coordinated studies using diverse RWE generation approaches and explore the platform's strengths and limitations. With guidance from an executive advisory committee of multidisciplinary experts and patient representatives, the goal of the FOUNTAIN platform is to harmonize RWE generation across a portfolio of research projects, including research partner collaborations and a common data model (CDM)-based program. FOUNTAIN's overarching objectives as a research platform are to establish long-term collaborations among pharmacoepidemiology research partners and experts and to integrate diverse approaches for RWE generation, including global protocol execution by research partners in local data sources and common protocol execution in multiple data sources through federated data networks, while ensuring harmonization of medical definitions, methodology, and reproducible artifacts across all studies. Specifically, the aim of the multiple studies run within the frame of FOUNTAIN is to provide insight into the real-world utilization, effectiveness, and safety of finerenone across its life-cycle.

RESULTS

Currently, the FOUNTAIN platform includes 9 research partner collaborations and 8 CDM-mapped data sources from 7 countries (United States, United Kingdom, China, Japan, The Netherlands, Spain, and Denmark). These databases and research partners were selected after a feasibility fit-for-purpose evaluation. Six multicountry, multidatabase, cohort studies are ongoing to describe patient populations, current standard of care, comorbidity profiles, healthcare resource use, and treatment effectiveness and safety in different patient populations with chronic kidney disease and type 2 diabetes. Strengths and potential limitations of FOUNTAIN are described in the context of valid RWE generation.

CONCLUSION

The establishment of the FOUNTAIN platform has allowed harmonized execution of multiple studies, promoting consistency both within individual studies that employ multiple data sources and across all studies run within the platform's framework. FOUNTAIN presents a proposal to efficiently improve the consistency and generalizability of RWE on finerenone.

A scalable approach to assess the safety of recently marketed systemic treatments for atopic dermatitis in clinical practice

Targeted systemic immune-modulating drugs (IMDs) to treat atopic dermatitis (AD) were highly efficacious in randomized trials. Trials with limited number of subjects leave questions about their safety. We describe a data and analytics structure for the production of timely, high-quality evidence on the comparative safety of recently approved IMDs in patients with AD in clinical practice. We established a series of sequential propensity score (PS)-balanced cohorts that grow in size with each annual data refresh. Nine health outcomes of interest plus conjunctivitis as a positive tracer outcome were identified. The initial treatment comparison was dupilumab, an interleukin-4/13 inhibitor, or tralokinumab, an interleukin-13 inhibitor, versus abrocitinib/upadacitinib, both JAK inhibitors. The first analysis cycle (December 2021-February 2023) compared 269 patients initiating JAK inhibitors and 2,650 initiating IL-4/IL-13 inhibitors. Patient characteristics were well balanced after PS-matching. Outpatient infections within 180 days occurred in 18% of JAK-1 inhibitor initiators versus 12% of dupilumab/ tralokinumab initiators (RR=1.50; 0.96 to 2.33) whereas acne risks were 7% vs. 3%, respectively (RR=2.29, 0.96 to 5.46). This sequential monitoring system will produce essential knowledge on the safety of IMDs to treat AD based on its growing study size of patients observed in clinical practice.

The interdependence of targets for continuous glucose monitoring outcomes in type 1 diabetes with automated insulin delivery

AIM

The aim was to determine the interdependence of targets for glucose management indicator (GMI), time within the ranges of 70-180 mg/dL (TIR) and 70-140 mg/dL (time in tight glucose range [TITR]), time above 180 mg/dL (TA180) and 250 mg/dL (TA250) and time below 70 mg/dL (TB70) and 54 mg/dL (TB54) and its implications for setting targets in automated insulin delivery (AID).

MATERIALS AND METHODS

Real-world data from individuals with type 1 diabetes using the 780G system were used to calculate the receiver operating characteristic curves and establish interdependent targets for time in ranges based on several GMI benchmarks. Correlation, regression and principal component analysis were used to determine their association and dimensionality.

RESULTS

In individuals aged >15 years (n = 41 692), a GMI <6.5% required targets of >81%, >58%, <15% and <1.9% for TIR, TITR, TA180 and TA250, respectively, with high sensitivity, specificity and accuracy (>90%), whereas these values were poor for time in hypoglycaemia and GMI, which had a modest correlation (-0.21 to -0.43). Two dimensions emerged: (1) GMI, TIR, TITR, TA180 and TA250, and (2) TB70 and TB54, explaining 95% of total variability. GMI (or TIR) and TB70 explained >81% of the variability in the remaining continuous glucose monitoring (CGM) metrics, providing accurate predictions. Individuals aged ≤15 years (n = 14 459) showed similar results.

CONCLUSION

We developed a methodology to establish interdependent CGM targets for therapies with CGM data outputs. In AID with the 780G system, a GMI <7% requires time in ranges close to consensus targets. Targets for GMI, TIR, TITR, TA180 and TA250 could be reduced to targets for GMI or TIR, whereas targets for time in hypoglycaemia are not inherently tied to GMI/TIR targets.

The maze of real-world evidence frameworks: from a desert to a jungle! An environmental scan and comparison across regulatory and health technology assessment agencies

Aim: Regulatory and health technology assessment (HTA) agencies have increasingly published frameworks, guidelines, and recommendations for the use of real-world evidence (RWE) in healthcare decision-making. Variations in the scope and content of these documents, with updates running in parallel, may create challenges for their implementation especially during the market authorization and reimbursement phases of a medicine's life cycle. This environmental scan aimed to comprehensively identify and summarize the guidance documents for RWE developed by most well-established regulatory and reimbursement agencies, as well as other organizations focused on healthcare decision-making, and present their similarities and differences. Methods: RWE guidance documents, including white papers from regulatory and HTA agencies, were reviewed in March 2024. Data on scope and recommendations from each body were extracted by two reviewers and similarities and differences were summarized across four topics: study planning, choosing fit-for-purpose data, study conduct, and reporting. Post-authorization or non-pharmacological guidance was excluded. Results: Forty-six documents were identified across multiple agencies; US FDA produced the most RWE-related guidance. All agencies addressed specific and often similar methodological issues related to study design, data fitness-for-purpose, reliability, and reproducibility, although inconsistency in terminologies on these topics was noted. Two HTA bodies (National Institute for Health and Care Excellence [NICE] and Canada's Drug Agency) each centralized all related RWE guidance under a unified framework. RWE quality tools and checklists were not consistently named and some differences in preferences were noted. European Medicines Agency, NICE, Haute Autorité de Santé, and the Institute for Quality and Efficiency in Health Care included specific recommendations on the use of analytical approaches to address RWE complexities and increase trust in its findings. Conclusion: Similarities in agencies' expectations on RWE studies design, quality elements, and reporting will facilitate evidence generation strategy and activities for manufacturers facing multiple, including global, regulatory and reimbursement submissions and re-submissions. A strong preference by decision-making bodies for local real-world data generation may hinder opportunities for data sharing and outputs from international federated data networks. Closer collaboration between decision-making agencies towards a harmonized RWE roadmap, which can be centrally preserved in a living mode, will provide manufacturers and researchers clarity on minimum acceptance requirements and expectations, especially as novel methodologies for RWE generation are rapidly emerging.

A Comprehensive Framework for Evaluating the Value Created by Real-World Evidence for Diverse Stakeholders: The Case for Coordinated Registry Networks

OBJECTIVES

This manuscript presents a comprehensive framework for the assessment of the value of real-world evidence (RWE) in healthcare decision-making. While RWE has been proposed to overcome some limitations of traditional, one-off studies, no systematic framework exists to measure if RWE actually lowers the burden. This framework aims to fill that gap by providing conceptual approaches for evaluating the time and cost efficiencies of RWE, thus guiding strategic investments in RWE infrastructure.

METHODS

The framework consists of four components: (114th Congress. 21st Century Cures Act.; 2015. https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf .) identification of stakeholders using and producing RWE, (National Health Council. Glossary of Patient Engagement Terms. Published 2019. Accessed May 18. 2021. https://nationalhealthcouncil.org/glossary-of-patient-engagement-terms/ .) understanding value propositions on how RWE can benefit stakeholders, (Center for Drug Evaluation and Research. CDER Patient-Focused Drug Development. U.S. Food & Drug Administration.) defining key performance indicators (KPIs), and (U.S. Department of Health and Human Services - Food and Drug Administration: Center for Devices and Radiological Health and Center for Biologics Evaluation and Research. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices - Guidance for Industry and Food and Drug Administration Staff. 2017. http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guida .) establishing metrics and case studies to assess value. KPIs are categorized as 'better, faster, or cheaper" as an indicator of value: better focusing on high-quality actionable evidence; 'faster,' denoting time-saving in evidence generation, and 'cheaper,' emphasizing cost-efficiency decision compared to methodologies that do not involve data routinely collected in clinical practice. Metrics and relevant case studies are tailored based on stakeholder value propositions and selected KPIs that can be used to assess what value has been created by using RWE compared to traditional evidence-generation approaches and comparing different RWE sources.

RESULTS

Operationalized through metrics and case studies drawn from the literature, the value of RWE is documented as improving treatment effect heterogeneity evaluation, expanding medical product labels, and expediting post-market compliance. RWE is also shown to reduce the cost and time required to produce evidence compared to traditional one-off approaches. An original example of a metric that measures the time saved by RWE methods to detect a signal of a product failure was presented based on analysis of the National Cardiovascular Disease Registry.

CONCLUSIONS

The framework presented in this manuscript offers a comprehensive approach for evaluating the value of RWE, applicable to all stakeholders engaged in leveraging RWE for healthcare decision-making. Through the proposed metrics and illustrated case studies, valuable insights are provided into the heightened efficiency, cost-effectiveness, and improved decision-making within clinical and regulatory domains facilitated by RWE. While this framework is primarily focused on medical devices, it could potentially inform the determination of RWE value in other medical products. By discerning the variations in cost, time, and data utility among various evidence-generation methods, stakeholders are empowered to invest strategically in RWE infrastructure and shape future research endeavors.

Generalizability in real-world trials

Real-world evidence (RWE) trials have a key advantage over conventional randomized controlled trials (RCTs) due to their potentially better generalizability. High generalizability of study results facilitates new biological insights and enables targeted therapeutic strategies. Random sampling of RWE trial participants is regarded as the gold standard for generalizability. Additionally, the use of sample correction procedures can increase the generalizability of trial results, even when using nonrandomly sampled real-world data (RWD). This study presents descriptive evidence on the extent to which the design of currently planned or already conducted RWE trials takes sampling into account. It also examines whether random sampling or procedures for correcting nonrandom samples are considered. Based on text mining of publicly available metadata provided during registrations of RWE trials on clinicaltrials.gov, EU-PAS, and the OSF-RWE registry, it is shown that the share of RWE trial registrations with information on sampling increased from 65.27% in 2002 to 97.43% in 2022, with a corresponding increase from 14.79% to 28.30% for trials with random samples. For RWE trials with nonrandom samples, there is an increase from 0.00% to 0.95% of trials in which sample correction procedures are used. We conclude that the potential benefits of RWD in terms of generalizing trial results are not yet being fully realized.

Factors Affecting on Environmental Strategy in SMEs manufacturing Firms of a developing country: Mediating role of Green Employee Behavior

Background

Sustainability in small and medium-sized enterprises (SMEs) which engage heavily in manufacturing provides them with an essential platform for implementing environmental strategy, which is aimed at reducing environmental damage and promoting environmental protection. This study aims to examine the impact of the reduction in resource use, the greening of processes and products on environmental strategy and the mediating role of green employee behaviour.

Methods

Primary data gathered with the help of questionnaires from employees of SMEs manufacturing in developing countries, Pakistan. The research gathered data from 211 participants by applying a pre-tested instrumental questionnaire and structural equation modelling (SEM) for data analysis.

Results

The results confirmed all direct effects, including a reduction in resource use, the greening of processes and products and green employee behaviour of environmental strategies in small and medium manufacturing firms. Also, confirmed partial mediation effect of green employee behavior between greening of processes and products and environmental strategy.

Conclusion

The current study clearly showed the need for programs and infrastructures that support resource reduction and greening businesses in SMEs. Companies' implementation of green processes and technologies allows SMEs to take an environmentally responsible stand and become part of the movement toward sustainability. The research stresses the importance of green employee conduct as a mediator, the central point being the individual employee being engaged and aware of the subject and willing to participate in environmentally friendly practices. Therefore, small and medium enterprises should enlighten their staff to practice green behaviour, give them what they need and empower them to be change drivers that will trigger sustainability throughout the organisations.

Streamlining intersectoral provision of real-world health data: a service platform for improved clinical research and patient care

Introduction

Obtaining real-world data from routine clinical care is of growing interest for scientific research and personalized medicine. Despite the abundance of medical data across various facilities - including hospitals, outpatient clinics, and physician practices - the intersectoral exchange of information remains largely hindered due to differences in data structure, content, and adherence to data protection regulations. In response to this challenge, the Medical Informatics Initiative (MII) was launched in Germany, focusing initially on university hospitals to foster the exchange and utilization of real-world data through the development of standardized methods and tools, including the creation of a common core dataset. Our aim, as part of the Medical Informatics Research Hub in Saxony (MiHUBx), is to extend the MII concepts to non-university healthcare providers in a more seamless manner to enable the exchange of real-world data among intersectoral medical sites.

Methods

We investigated what services are needed to facilitate the provision of harmonized real-world data for cross-site research. On this basis, we designed a Service Platform Prototype that hosts services for data harmonization, adhering to the globally recognized Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) international standard communication format and the Observational Medical Outcomes Partnership (OMOP) common data model (CDM). Leveraging these standards, we implemented additional services facilitating data utilization, exchange and analysis. Throughout the development phase, we collaborated with an interdisciplinary team of experts from the fields of system administration, software engineering and technology acceptance to ensure that the solution is sustainable and reusable in the long term.

Results

We have developed the pre-built packages "ResearchData-to-FHIR," "FHIR-to-OMOP," and "Addons," which provide the services for data harmonization and provision of project-related real-world data in both the FHIR MII Core dataset format (CDS) and the OMOP CDM format as well as utilization and a Service Platform Prototype to streamline data management and use.

Conclusion

Our development shows a possible approach to extend the MII concepts to non-university healthcare providers to enable cross-site research on real-world data. Our Service Platform Prototype can thus pave the way for intersectoral data sharing, federated analysis, and provision of SMART-on-FHIR applications to support clinical decision making.

OpenSAFELY: A platform for analysing electronic health records designed for reproducible research

Electronic health records (EHRs) and other administrative health data are increasingly used in research to generate evidence on the effectiveness, safety, and utilisation of medical products and services, and to inform public health guidance and policy. Reproducibility is a fundamental step for research credibility and promotes trust in evidence generated from EHRs. At present, ensuring research using EHRs is reproducible can be challenging for researchers. Research software platforms can provide technical solutions to enhance the reproducibility of research conducted using EHRs. In response to the COVID-19 pandemic, we developed the secure, transparent, analytic open-source software platform OpenSAFELY designed with reproducible research in mind. OpenSAFELY mitigates common barriers to reproducible research by: standardising key workflows around data preparation; removing barriers to code-sharing in secure analysis environments; enforcing public sharing of programming code and codelists; ensuring the same computational environment is used everywhere; integrating new and existing tools that encourage and enable the use of reproducible working practices; and providing an audit trail for all code that is run against the real data to increase transparency. This paper describes OpenSAFELY's reproducibility-by-design approach in detail.

When Will Real-World Data Fulfill Its Promise to Provide Timely Insights in Oncology?

Randomized trials provide high-quality, internally consistent data on selected clinical questions, but lack generalizability for the aging population who are most often diagnosed with cancer and have comorbid conditions that may affect the interpretation of treatment benefit. The need for high-quality, relevant, and timely data is greater than ever. Promising solutions lie in the collection and analysis of real-world data (RWD), which can potentially provide timely insights about the patient's course during and after initial treatment and the outcomes of important subgroups such as the elderly, rural populations, children, and patients with greater social health needs. However, to inform practice and policy, real-world evidence must be created from trustworthy and comprehensive sources of RWD; these may include pragmatic clinical trials, registries, prospective observational studies, electronic health records (EHRs), administrative claims, and digital technologies. There are unique challenges in oncology since key parameters (eg, cancer stage, biomarker status, genomic assays, imaging response, side effects, quality of life) are not recorded, siloed in inaccessible documents, or available only as free text or unstructured reports in the EHR. Advances in analytics, such as artificial intelligence, may greatly enhance the ability to obtain more granular information from EHRs and support integrated diagnostics; however, they will need to be validated purpose by purpose. We recommend a commitment to standardizing data across sources and building infrastructures that can produce fit-for-purpose RWD that will provide timely understanding of the effectiveness of individual interventions.

Real-World Evidence to Reinforce Clinical Trial Evidence in Health Technology Assessment: A Critical Review of Real-World Evidence Requirements from Seven Countries and Recommendations to Improve Acceptance

BACKGROUND

Real-world evidence (RWE) can reinforce clinical trial evidence in health technology assessment (HTA).

OBJECTIVES

Review HTA bodies' (HTAbs) requirements for RWE, real uses, and acceptance across seven countries (Brazil, Canada, France, Germany, Italy, Spain, and the United Kingdom) and outline recommendations that may improve acceptance of RWE in efficacy/effectiveness assessments and appraisals processes.

METHODS

RWE requirements were summarized based on HTAbs' guidelines. Acceptance by HTAbs was evaluated based on industry experience and case studies.

RESULTS

As of June 2022, RWE methodological guidelines were in place in three of the seven countries. HTAbs typically requested analyses based on local data sources, but the preferred study design and data sources differed. HTAbs had individual submission, assessment, and appraisal processes; some allowed early meetings for the protocol and/or results validation, though few involved external experts or medical societies to provide input to assessment and appraisal. The extent of submission, assessment, and appraisal requirements did not necessarily reflect the degree of acceptance.

CONCLUSION

All the countries reviewed face common challenges regarding the use of RWE. Our proposals address the need to facilitate collaboration and communication with industry and regulatory agencies and the need for specific guidelines describing RWE design and criteria of acceptance throughout the assessment and appraisal processes.

Describing diversity of real world data sources in pharmacoepidemiologic studies: The DIVERSE scoping review

PURPOSE

Real-world evidence (RWE) is increasingly used for medical regulatory decisions, yet concerns persist regarding its reproducibility and hence validity. This study addresses reproducibility challenges associated with diversity across real-world data sources (RWDS) repurposed for secondary use in pharmacoepidemiologic studies. Our aims were to identify, describe and characterize practices, recommendations and tools for collecting and reporting diversity across RWDSs, and explore how leveraging diversity could improve the quality of evidence.

METHODS

In a preliminary phase, keywords for a literature search and selection tool were designed using a set of documents considered to be key by the coauthors. Next, a systematic search was conducted up to December 2021. The resulting documents were screened based on titles and abstracts, then based on full texts using the selection tool. Selected documents were reviewed to extract information on topics related to collecting and reporting RWDS diversity. A content analysis of the topics identified explicit and latent themes.

RESULTS

Across the 91 selected documents, 12 topics were identified: 9 dimensions used to describe RWDS (organization accessing the data source, data originator, prompt, inclusion of population, content, data dictionary, time span, healthcare system and culture, and data quality), tools to summarize such dimensions, challenges, and opportunities arising from diversity. Thirty-six themes were identified within the dimensions. Opportunities arising from data diversity included multiple imputation and standardization.

CONCLUSIONS

The dimensions identified across a large number of publications lay the foundation for formal guidance on reporting diversity of data sources to facilitate interpretation and enhance replicability and validity of RWE.

From “exceptional ethics” to public health ethics. Patients and proxies facing COVID-19. Ethics and public health during COVID-19

The study aimed to elicit the perception and ethical considerations of patients and proxies with respect both to the individual medical decisions and public health decisions made during the COVID-19 crisis. It used a qualitative, multi-center study based on semi-directive interviews, conducted by an interdisciplinary team. The analysis was conducted using a thematic analysis approach and an ethical framework. Three themes emerged from the analysis: 1) patients, unlike proxies, did not complain about their diminished role in the decision-making process. Both highlighted the importance of “basic care” as opposed to a technical approach to treatment; 2) despite the transparency of the information process, a deep “crisis of trust” has developed between citizens and public authorities; 3) although both patients and proxies accepted the limitations of personal liberties imposed in the name of public health, they argued that these limitations should respect certain boundaries, both temporal and spacial. Above all, they should not affect basic affective human relationships, even if such boundaries are a factor in an increased risk of infection. The study showed that there is a need to reconsider the definition and the main principles of public health ethics, namely transparency and proportionality.

Generating and using real-world data: A worthwhile uphill battle

The precision oncology paradigm challenges the feasibility and data generalizability of traditional clinical trials. Consequently, an unmet need exists for practical approaches to test many subgroups, evaluate real-world drug value, and gather comprehensive, accessible datasets to validate novel biomarkers. Real-world data (RWD) are increasingly recognized to have the potential to fill this gap in research methodology. Established applications of RWD include informing disease epidemiology, pharmacovigilance, and healthcare quality assessment. Currently, concerns regarding RWD quality and comprehensiveness, privacy, and biases hamper their broader application. Nonetheless, RWD may play a pivotal role in supplementing clinical trials, enabling conditional reimbursement and accelerated drug access, and innovating trial conduct. Moreover, purpose-built RWD repositories may support the extension or refinement of drug indications and facilitate the discovery and validation of new biomarkers. This perspective explores the potential of leveraging RWD to advance oncology, highlights its benefits and challenges, and suggests a path forward in this evolving field.

Celebrating the Data from 100,000 Real-World Users of the MiniMed™ 780G System in Europe, Middle East, and Africa Collected Over 3 Years: From Data to Clinical Evidence

Introduction: The present report celebrates the benchmarking of 100,000 MiniMed™ 780G system users in Europe, Middle East, and Africa (EMEA) and summarizes the major insights into the usability and outcomes of this system. Methods: Carelink Personal data (August 2020-August 2023) of users living in EMEA were analyzed. Continuous glucose monitoring-based endpoints were aggregated for (1) the full cohort and (2) a 12-month longitudinal cohort. Subanalyses were done for users on optimal settings (those spending ≥95% of time with glucose target of 100 mg/dL, and ≥95% of time with active insulin time of 2 h), for self-reported age groups (≤15 and ≥56 years) and for various countries/regions. Results: Data from 101,629 users (34 countries) were analyzed. Mean time in range (TIR) was 72.3%, glucose management indicator (GMI) was 7%, time below 70 mg/dL (TBR70) was 2.0% and time below 54 mg/dL (TBR54) was 0.4%. In terms of international targets, 59.6% of users achieved a GMI <7%, 62.5% a TIR >70%, 88.4% a TBR70 < 4%, and 90.0% a TBR54 < 1%. Data improved impressively in optimal setting users (TIR = 78.8%, and users reaching TIR >70% = 86.3%) while safety remained (TBR70 = 2.2% and TBR54 = 0.4%). Data showed consistency across self-reported age groups and geographies. In the longitudinal cohort, TIR reached 75.5% in the first month and remained 73.3% or higher over the 12-month period. Conclusion: Over 100,000 users of the MiniMed™ 780G system have demonstrated consistency in achieving target control of glycemia.

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

OBJECTIVES

We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

DESIGN

Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

DATA SOURCES

Systematic search: Medline; Snowballing: starting from three articles (D'Andrea et al, Quigley et al and Faria et al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

DATA EXTRACTION AND SYNTHESIS

Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

RESULTS

Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

CONCLUSION

Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

OSF REGISTRATION NUMBER

OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).

A patient-centred and multi-stakeholder co-designed observational prospective study protocol: Example of the adolescent experience of treatment for X-linked hypophosphataemia (XLH)

The importance of patient centricity and keeping the patient at the heart of research design is now well recognised within the healthcare community. The involvement of patient, caregiver and clinician representatives in the study design process may help researchers to achieve this goal and to ensure robust and meaningful data generation. Real-world data collection allows for a more flexible and patient-centred research approach for gaining important insights into the experience of disease and treatments, which is acutely relevant for rare diseases where knowledge about the disease is more likely to be limited. Here, we describe a practical example of a patient-centric, multi-stakeholder approach that led to the co-design of a prospective observational study investigating the lived experience of adolescents with the rare disease, X-linked hypophosphataemia. Specifically, we describe how the knowledge and expertise of a diverse research team, which included expert physicians, research and technology specialists, patients and caregivers, were applied in order to identify the relevant research questions and to ensure the robustness of the study design and its appropriateness to the population of interest within the context of the current clinical landscape. We also demonstrate how a structured patient engagement exercise was key to informing the selection of appropriate outcome measures, data sources, timing of data collection, and to assessing the feasibility and acceptability of the proposed data collection approach.

Building trust in real world evidence (RWE): moving transparency in RWE towards the randomized controlled trial standard

The increasing use of RWE in regulatory and reimbursement decision-making indicates the significant progress that has been made in building trust in RWE through greater transparency. This review of the published literature and key online sources was conducted to provide an update on progress towards greater transparency in RWE, based on four key barriers to trust identified in a 2016 paper and applying learnings from transparency initiatives established for RCTs, such as the US FDA Amendments Act (FDAAA) 2007 Final Rule. Multiple initiatives from the US FDA, EMA and organizations such as the ISPOR-ISPE Joint Task Force have yielded new guidance documents and tools that enable greater transparency in RWE study methodology (STaRT-RWE and HARPER templates), data source selection and quality (SPIFD2 framework, REQueST tool), strategy (the Center for Open Science RWE Study Registry), and will therefore improve transparency in RWE study reporting. Programs such as the REPEAT Initiative and RWE DUPLICATE are investigating reproducibility of RWE studies and improving understanding of the circumstances when valid inference on treatment effects can be obtained from RWE studies. Further work is needed to embed and to implement the new tools and guidance that are available, and thus raise standards for RWE transparency towards the levels expected for RCTs.

Principles for Good Practice in the Conduct of Non-interventional Studies: The View of Industry Researchers

This reflection paper presents a consolidated view of EFPIA on the need for principles for good practice in the generation and use of non-interventional studies (NIS), including overarching principles such as the registration of hypothesis evaluating treatment effect (HETE) studies. We first define NIS and the important adjacencies to clinical trials and relationship with real-world evidence (RWE). We then outline the principles for good practice with respect to appropriate research design, study protocol, fit-for-purpose variables and data quality, analytical methods, bias reduction, transparency in conduct and use, privacy management and ethics review. We conclude with recommendations for action for the research community to promote trust and credibility in the use of NIS.

A Road Map for Peer Review of Real-World Evidence Studies on Safety and Effectiveness of Treatments

The growing acceptance of real-world evidence (RWE) in clinical and regulatory decision-making, coupled with increasing availability of health care data and advances in automated analytic approaches, has contributed to a marked expansion of RWE studies of diabetes and other diseases. However, a recent spate of high-profile retractions highlights the need for improvements in the conduct of RWE research as well as in the associated peer review and editorial processes. We review best pharmacoepidemiologic practices and common pitfalls regarding design, measurement, analysis, data validity, appropriateness, and generalizability of RWE studies. To enhance RWE study assessments, we propose that journal editors require 1) study authors to complete RECORD-PE, a reporting guideline for pharmacoepidemiological studies on routinely collected data, 2) availability of predetermined study protocols and analysis plans, 3) inclusion of pharmacoepidemiologists on the peer review team, and 4) provision of detail on data provenance, characterization, and custodianship to facilitate assessment of the data source. We recognize that none of these steps guarantees a high-quality research study. Collectively, however, they permit an informed assessment of whether the study was adequately designed and conducted and whether the data source used was fit for purpose.

Effectively Leveraging RWD for External Controls: A Systematic Literature Review of Regulatory and HTA Decisions

Real-world data (RWD)-derived external controls can be used to contextualize efficacy findings for investigational therapies evaluated in uncontrolled trials. As the number of submissions to regulatory and health technology assessment (HTA) bodies using external controls rises, and in light of recent regulatory and HTA guidance on the appropriate use of RWD, there is a need to address the operational and methodological challenges impeding the quality of real-world evidence (RWE) generation and the consistency in evaluation of RWE across agencies. This systematic review summarizes publicly available information on the use of external controls to contextualize outcomes from uncontrolled trials for all indications from January 1, 2015, through August 20, 2021, that were submitted to the European Medicines Agency, the US Food and Drug Administration, and/or select major HTA bodies (National Institute for Health and Care Excellence (NICE), Haute Autorité de Santé (HAS), Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), and Gemeinsamer Bundesausschuss (G-BA)). By systematically reviewing submissions to regulatory and HTA bodies in the context of recent guidance, this study provides quantitative and qualitative insights into how external control design and analytic choices may be viewed by different agencies in practice. The primary operational and methodological aspects identified for discussion include, but are not limited to, engagement of regulators and HTA bodies, approaches to handling missing data (a component of data quality), and selection of real-world endpoints. Continued collaboration and guidance to address these and other aspects will inform and assist stakeholders attempting to generate evidence using external controls.

Integration of real-world evidence from different data sources in health technology assessment

Real-world evidence (RWE) is being increasingly used by a wide range of stakeholders involved in the therapeutic product lifecycle but remains underutilized in the health technology assessment (HTA) process. RWE aims to fill the current evidence gaps, reduce the uncertainty around the benefits of medical technologies, and better understand the long-term impact of health technologies in real-world conditions. Despite the minimal use of RWE in some elements of HTA, there has been a larger push to further utilize RWE in the HTA processes. HTA bodies, as other stakeholders, work towards developing more robust means to leverage RWE from various data sources in the HTA processes. However, these agencies need to overcome important challenges before the broader incorporation of RWE into their routine practice. This paper aims to explore the extensive integration of RWE utilizing diverse sources of RWD. We discuss the utilization of RWE in HTA processes, considering aspects such as when, where, and how RWE can be effectively applied. Additionally, we seek the potential challenges and barriers associated with the utilization of different data sources.

Using Real-World Data in the Health Technology Assessment of Pharmaceuticals: Strengths, Difficulties, and a Pragmatic Way Forward

In the past decade, there have been increasing calls for greater use of real-world evidence (RWE) and data (RWD), with the explicit goal of enabling faster provision of effective medicines to patients in need. The push for decision makers to accept RWE is especially noticeable in the pursuit of regulatory approval, but RWE, particularly when used to estimate the relative effectiveness of interventions, is not always readily accepted by agencies responsible for reimbursement and pricing of new pharmaceuticals and, to a varying degree, is not accepted across jurisdictions. This lack of trust hampers the use of RWE despite a very large and growing literature base on the principles of how RWE should be used. In this article, we suggest an important part of the explanation of why this situation has arisen and make suggestions for its alleviation. Given that problems commonly arise that are particular to the question being asked and the data sources being used, general guidance on the principles of how to use RWD cannot cover all eventualities. Therefore, we are suggesting the creation of an archive, or repository, to record uses of RWD in support of decisions by funding bodies or their advisors. This article introduces a proposed, structured classification of decision types using RWE, around which evidence can be assembled in a curated source (RWD/RWE taxonomy) and thus facilitate judgments on when evidence is "good enough." This article is part of a series in a special issue of this journal that looks at the barriers to optimal use of RWE in health technology assessment and how to overcome them. We begin significantly to populate our "taxonomy" with examples in an accompanying article. We also propose recommendations for international standards of evaluating the acceptability of RWD governance practices.

Guidance on using real-world evidence from Western Europe in Central and Eastern European health policy decision making

Aim: Real-world data and real-world evidence (RWE) are becoming more important for healthcare decision making and health technology assessment. We aimed to propose solutions to overcome barriers preventing Central and Eastern European (CEE) countries from using RWE generated in Western Europe. Materials & methods: To achieve this, following a scoping review and a webinar, the most important barriers were selected through a survey. A workshop was held with CEE experts to discuss proposed solutions. Results: Based on survey results, we selected the nine most important barriers. Multiple solutions were proposed, for example, the need for a European consensus, and building trust in using RWE. Conclusion: Through collaboration with regional stakeholders, we proposed a list of solutions to overcome barriers on transferring RWE from Western Europe to CEE countries.

Cost-utility analysis of the UPRIGHT intervention promoting resilience in adolescents

BACKGROUND

As mental health in adulthood is related to mental status during adolescence, school-based interventions have been proposed to improve resilience. The objective of this study was to build a simulation model representing the natural history of mental disorders in childhood, adolescence and youth to estimate the cost-effectiveness of the UPRIGHT school-based intervention in promoting resilience and mental health in adolescence.

METHODS

We built a discrete event simulation model fed with real-world data (cumulative incidence disaggregated into eight clusters) from the Basque Health Service database (609,381 individuals) to calculate utilities (quality-adjusted life years [QALYs]) and costs for the general population in two scenarios (base case and intervention). The model translated changes in the wellbeing of adolescents into different risks of mental illnesses for a time horizon of 30 years.

RESULTS

The number of cases of anxiety was estimated to fall by 5,125 or 9,592 and those of depression by 1,269 and 2,165 if the effect of the intervention lasted 2 or 5 years respectively. From a healthcare system perspective, the intervention was cost-effective for all cases considered with incremental cost-utility ratios always lower than €10,000/QALY and dominant for some subgroups. The intervention was always dominant when including indirect and non-medical costs (societal perspective).

CONCLUSIONS

Although the primary analysis of the trial did not did not detect significant differences, the UPRIGHT intervention promoting positive mental health was dominant in the economic evaluation from the societal perspective. Promoting resilience was more cost-effective in the most deprived group. Despite a lack of information about the spillover effect in some sectors, the economic evaluation framework developed principally for pharmacoeconomics can be applied to interventions to promote resilience in adolescents. As prevention of mental health disorders is even more necessary in the post-coronavirus disease-19 era, such evaluation is essential to assess whether investment in mental health promotion would be good value for money by avoiding costs for healthcare providers and other stakeholders.

What, Where, and How to Collect Real-World Data and Generate Real-World Evidence to Support Drug Reimbursement Decision-Making in Asia: A reflection Into the Past and A Way Forward

BACKGROUND

Globally, there is increasing interest in the use of real-world data (RWD) and real-world evidence (RWE) to inform health technology assessment (HTA) and reimbursement decision-making. Using current practices and case studies shared by eleven health systems in Asia, a non-binding guidance that seeks to align practices for generating and using RWD/RWE for decision-making in Asia was developed by the REAL World Data In ASia for HEalth Technology Assessment in Reimbursement (REALISE) Working Group, addressing a current gap and needs among HTA users and generators.

METHODS

The guidance document was developed over two face-to-face workshops, in addition to an online survey, a face-to-face interview and pragmatic search of literature. The specific focus was on what, where and how to collect RWD/ RWE.

RESULTS

All 11 REALISE member jurisdictions participated in the online survey and the first in-person workshop, 10 participated in the second in-person workshop, and 8 participated in the in-depth face-to-face interviews. The guidance document was iteratively reviewed by all working group members and the International Advisory Panel. There was substantial variation in: (a) sources and types of RWD being used in HTA, and (b) the relative importance and prioritization of RWE being used for policy-making. A list of national-level databases and other sources of RWD available in each country was compiled. A list of useful guidance on data collection, quality assurance and study design were also compiled.

CONCLUSION

The REALISE guidance document serves to align the collection of better quality RWD and generation of reliable RWE to ultimately inform HTA in Asia.

Follicle-stimulating hormone effectiveness in male idiopathic infertility: What happens in daily practice?

OBJECTIVES

To assess the effectiveness of follicle-stimulating hormone (FSH) administration in male idiopathic infertility in a clinical setting.

METHODS

A retrospective real-world study was carried out, including all consecutive FSH-treated infertile men attending the Andrology Unit of Modena (Italy) from June 2015 to May 2022. Medical history, physical and andrological examinations, hormonal and seminal parameters, therapeutic management and pregnancy data were collected. The primary endpoint was the number of pregnancies obtained after FSH administration, whereas semen parameters change was the secondary outcome.

RESULTS

A total of 194 of 362 (53.6%) infertile men, eligible according to the Italian Health System regulations, were treated with FSH (mean age 37.9 ± 6.1 years). Following FSH administration (mean therapy duration 9.1 ± 7.1 months), 43 pregnancies were recorded (27.6%), of which 22 occurred naturally and 21 after assisted reproduction. A significant increase in sperm concentration (9.9 ± 12.2 vs. 18.9 ± 38.9 million/mL, p = 0.045) was detected after treatment, together with a significant increase in normozoospermia (from 1.0% to 5.1%, p = .044) and a reduction in azoospermia rate (from 9.8% to 7%, p = 0.044). Dividing the cohort in FSH-responders and non-responders, in terms of pregnancy achieved, higher sperm concentrations (15.7 ± 26.6 vs. 22.2 ± 25.7 million/mL, p = 0.033) and progressive sperm motility (18.0 ± 18.2 vs. 27.3 ± 11.3, p = 0.044) were found in pregnancy group.

CONCLUSION

Our experience suggests that FSH, empirically administered to men with idiopathic infertility, leads to pregnancy in one out of four patients and increases sperm concentration. Although the expected limits because of a real-world data study, the number of FSH-treated patients required to achieve one pregnancy seems to be lower in clinical setting if compared to previously published data.

Worldwide Trends in Registering Real-World Studies at ClinicalTrials.gov: A Cross-Sectional Analysis

Objective

The purpose of this study was to characterize real-world studies (RWSs) registered at ClinicalTrials.gov to help investigators better conduct relevant research in clinical practice.

Methods

A retrospective analysis of 944 studies was performed on February 28, 2023.

Results

A total of 944 studies were included. The included studies involved a total of 48 countries. China was the leading country in terms of the total number of registered studies (37.9%, 358), followed by the United States (19.7%, 186). Regarding intervention type, 42.4% (400) of the studies involved drugs, and only 9.1% (86) of the studies involved devices. Only 8.5% (80) of the studies mentioned both the detailed study design type and data source in the "Brief Summary". A total of 49.4% (466) of studies had a sample size of 500 participants and above. Overall, 63% (595) of the studies were single-center studies. A total of 213 conditions were covered in the included studies. One-third of the studies (32.7%, 309) involved neoplasms (or tumors). China and the United States were very different regarding the study of different conditions.

Conclusion

Although the pandemic has provided new opportunities for RWSs, the rigor of scientific research still needs to be emphasized. Special attention needs to be given to the correct and comprehensive description of the study design in the Brief Summary of registered studies, thereby promoting communication and understanding. In addition, deficiencies in ClinicalTrials.gov registration data remain prominent.

Clinical trial data-sharing policies among journals, funding agencies, foundations, and other professional organizations: a scoping review

BACKGROUND AND OBJECTIVES

To identify the similarities and differences in data-sharing policies for clinical trial data that are endorsed by biomedical journals, funding agencies, and other professional organizations. Additionally, to determine the beliefs, and opinions regarding data-sharing policies for clinical trials discussed in articles published in biomedical journals.

METHODS

Two searches were conducted, a bibliographic search for published articles that present beliefs, opinions, similarities, and differences regarding policies governing the sharing of clinical trial data. The second search analyzed the gray literature (non-peer-reviewed publications) to identify important data-sharing policies in selected biomedical journals, foundations, funding agencies, and other professional organizations.

RESULTS

A total of 471 articles were included after database search and screening, with 45 from the bibliographic search and 426 from the gray literature search. A total of 424 data-sharing policies were included. Fourteen of the 45 published articles from the bibliographic search (31.1%) discussed only advantages specific to data-sharing policies, 27 (27/45; 60%) discussed both advantages and disadvantages, and 4 (4/45; 8.9%) discussed only disadvantages specific. A total of 216 journals (of 270; 80%) specified a data-sharing policy provided by the journal itself. One hundred industry data-sharing policies were included, and 32 (32%) referenced a data-sharing policy on their website. One hundred and thirty-six (42%) organizations (of 327) specified a data-sharing policy.

CONCLUSION

We found many similarities listed as advantages to data-sharing and fewer disadvantages were discussed within the literature. Additionally, we found a wide variety of commonalities and differences-such as the lack of standardization between policies, and inadequately addressed details regarding the accessibility of research data-that exist in data-sharing policies endorsed by biomedical journals, funding agencies, and other professional organizations. Our study may not include information on all data sharing policies and our data is limited to the entities' descriptions of each policy.

Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment

Background

Artificial intelligence (AI) has attracted much attention because of its enormous potential in healthcare, but uptake has been slow. There are substantial barriers that challenge health technology assessment (HTA) professionals to use AI-generated evidence for decision-making from large real-world databases (e.g., based on claims data). As part of the European Commission-funded HTx H2020 (Next Generation Health Technology Assessment) project, we aimed to put forward recommendations to support healthcare decision-makers in integrating AI into the HTA processes. The barriers, addressed by the paper, are particularly focusing on Central and Eastern European (CEE) countries, where the implementation of HTA and access to health databases lag behind Western European countries.

Methods

We constructed a survey to rank the barriers to using AI for HTA purposes, completed by respondents from CEE jurisdictions with expertise in HTA. Using the results, two members of the HTx consortium from CEE developed recommendations on the most critical barriers. Then these recommendations were discussed in a workshop by a wider group of experts, including HTA and reimbursement decision-makers from both CEE countries and Western European countries, and summarized in a consensus report.

Results

Recommendations have been developed to address the top 15 barriers in areas of (1) human factor-related barriers, focusing on educating HTA doers and users, establishing collaborations and best practice sharing; (2) regulatory and policy-related barriers, proposing increasing awareness and political commitment and improving the management of sensitive information for AI use; (3) data-related barriers, suggesting enhancing standardization and collaboration with data networks, managing missing and unstructured data, using analytical and statistical approaches to address bias, using quality assessment tools and quality standards, improving reporting, and developing better conditions for the use of data; and (4) technological barriers, suggesting sustainable development of AI infrastructure.

Conclusion

In the field of HTA, the great potential of AI to support evidence generation and evaluation has not yet been sufficiently explored and realized. Raising awareness of the intended and unintended consequences of AI-based methods and encouraging political commitment from policymakers is necessary to upgrade the regulatory and infrastructural environment and knowledge base required to integrate AI into HTA-based decision-making processes better.

Developing Patient-Centered Real-World Evidence: Emerging Methods Recommendations From a Consensus Process

OBJECTIVES

The Joint ISPOR-ISPE Special Task Force on Real-World Evidence included patient/stakeholder engagement as a recommended good procedural practice when designing, conducting, and disseminating real-world evidence (RWE). However, there are no guidelines describing how patient experience data (PED) can be applied when designing real-world data (RWD) studies. This article describes development of consensus recommendations to guide researchers in applying PED to develop patient-centered RWE.

METHODS

A multidisciplinary advisory board, identified through recommendations of collaborators, was established to guide development of recommendations. Semistructured interviews were conducted to identify how experienced RWD researchers (n = 15) would apply PED when designing a hypothetical RWD study. Transcripts were analyzed and emerging themes developed into preliminary methods recommendations. An eDelphi survey (n = 26) was conducted to refine/develop consensus on the draft recommendations.

RESULTS

We identified 13 recommendations for incorporating PED throughout the design, conduct, and translation of RWE. The recommendations encompass themes related to the development of a patient-centered research question, designing a study, disseminating RWE, and general considerations. For example, consider how patient input can inform population/subgroups, comparators, and study period. Researchers can leverage existing information describing PED and may be able to apply those insights to studies relying on traditional RWD sources and/or patient registries.

CONCLUSIONS

Applying these emerging recommendations may improve the patient centricity of RWE through improved relevance of RWE to patient communities of interest and foster greater multidisciplinary participation and transparency in RWD research. As researchers gather experience by applying the methods recommendations, further refinement of these consensus recommendations may lead to "best practices."

Which sperm parameter limits could really guide the clinical decision in assisted reproduction?

BACKGROUND

The predictive role of sperm motility and morphology was recently detected in a large sample of more than 20000 assisted reproductive technology (ART) fresh cycles. However, the complete ART procedure consisted of both fresh and frozen-embryos transfers and only a comprehensive evaluation of the entire process could really confirm if these parameters really predict the ART success. The aim of the study was to evaluate which sperm parameter could predict the success of ART.

METHODS

A retrospective, real-world data analysis was performed, enrolling all couples attending ART from 2008 to 2021, including both fresh and frozen cycles and both in vitro fertilization (IVF) and intra-cytoplasmic sperm injection (ICSI) procedures.

RESULTS

Fresh cycles success (considering live birth rate) was predicted by female age (1.04 [1.02-1.06]), injected oocytes (0.96 [0.93-0.99]), embryo number (0.79 [0.75-0.83]) and progressive sperm motility (0.98 [0.97-0.99]). On the contrary, frozen cycle outcomes were predicted only by sperm motility (0.97 [0.95-0.99]). This prediction was confirmed in IVF but not in ICSI cycles.

CONCLUSION

Both female and male parameters predicted the ART success considering the entire path. However, frozen cycle success was predicted only by progressive sperm motility in IVF cycles, suggesting that the potential amelioration of this male parameter is relevant to improve ART success. Those couples expected to obtain the highest embryos after fertilization (low female age and better semen parameters) will have more attempts with frozen cycles and thus would benefit from a potential treatment focused to improve sperm parameters.

Real-world evidence analysis of the follicle-stimulating hormone use in male idiopathic infertility

Male idiopathic infertility remains a therapeutic challenge in the couple infertility management. In this setting, an empirical treatment with follicle-stimulating hormone (FSH) is allowed, although not recommended. Twenty-one clinical trials and four meta-analyses highlighted an overall increased pregnancy rate in case of FSH administration, but the indiscriminate FSH prescription is still unsupported by clinical evidence in idiopathic infertility. This context could represent an example in which real-world data (RWD) could add useful information. From a nationwide clinical practice survey performed in Italy, emerged the clinicians' attitude to prescribe FSH in the case of impaired semen with a significant improvement of semen parameters, identifying FSH treatment as a therapeutic card in the real-life management. Although more robust data are still needed to optimize FSH treatment in male idiopathic infertility, RWD should be included in the body of evidence considered in healthcare decision-making.

Real-world data: a brief review of the methods, applications, challenges and opportunities

Background

The increased adoption of the internet, social media, wearable devices, e-health services, and other technology-driven services in medicine and healthcare has led to the rapid generation of various types of digital data, providing a valuable data source beyond the confines of traditional clinical trials, epidemiological studies, and lab-based experiments.

Methods

We provide a brief overview on the type and sources of real-world data and the common models and approaches to utilize and analyze real-world data. We discuss the challenges and opportunities of using real-world data for evidence-based decision making This review does not aim to be comprehensive or cover all aspects of the intriguing topic on RWD (from both the research and practical perspectives) but serves as a primer and provides useful sources for readers who interested in this topic.

Results and Conclusions

Real-world hold great potential for generating real-world evidence for designing and conducting confirmatory trials and answering questions that may not be addressed otherwise. The voluminosity and complexity of real-world data also call for development of more appropriate, sophisticated, and innovative data processing and analysis techniques while maintaining scientific rigor in research findings, and attentions to data ethics to harness the power of real-world data.

Visualizations throughout pharmacoepidemiology study planning, implementation, and reporting

Transparency is increasingly promoted to instill trust in nonrandomized studies using real-world data. Graphics and data visualizations support transparency by aiding communication and understanding, and can inform study design and analysis decisions. However, other than graphical representation of a study design and flow diagrams (e.g., a Consolidated Standards of Reporting Trials [CONSORT] like diagram), specific standards on how to maximize validity and transparency with visualization are needed. This paper provides guidance on how to use visualizations throughout the life cycle of a pharmacoepidemiology study-from initial study design to final report-to facilitate rationalized and transparent decision-making about study design and implementation, and clear communication of study findings. Our intent is to help researchers align their practices with current consensus statements on transparency.

Value-based evidence across health care sectors: a push for transparent real-world studies, data, and evidence dissemination

There is currently a heightened need for transparency in pharmaceutical sectors. The inclusion of real-world (RW) evidence, in addition to clinical trial evidence, in decision-making processes, was an important step forward toward a more inclusive established value proposition. This advance has introduced new transparency challenges. Increasing transparency is a critical step toward accelerating improvement in type, quality, and access to data, regardless of whether these originate from clinical trials or from RW studies. However, so far, advances in transparency have been relatively restricted to clinical trials, and there remains a lack of similar expectations or standards of transparency concerning the generation and reporting of RW data. This perspective paper aims to highlight the need for transparency concerning RW studies, data, and evidence across health care sectors, to identify areas for improvement, and provide concrete recommendations and practices for the future. Specific issues are discussed from different stakeholder perspectives, culminating in recommended actions, from individual stakeholder perspectives, for improved RW study, data, and evidence transparency. Furthermore, a list of potential guidelines for consideration by stakeholders is proposed. While recommendations from different stakeholder perspectives are made, true transparency in the processes involved in the generation, reporting, and use of RW evidence will require a concerted effort from all stakeholders across health care sectors.

The challenges and opportunities in using real-world data to drive advances in healthcare in East Asia: expert panel recommendations

OBJECTIVE

To provide recommendations for overcoming the challenges associated with the generation and use of real-world evidence (RWE) in regulatory approvals, health technology assessments (HTAs), and reimbursement decision-making in East Asia.

METHODS

A panel of experts convened at the International Society for Pharmacoeconomics and Outcomes Research Asia Pacific 2020 congress to discuss the challenges limiting the use of RWE in healthcare decision-making and to provide insights into the perspectives of regulators, HTA agencies, the pharmaceutical industry, and physicians in China, Japan, and Taiwan. A nonsystematic literature review was conducted to expand on the themes addressed.

RESULTS

The use of RWE in regulatory approvals, HTAs, and reimbursement decision-making remains limited by legal/regulatory, technical, and attitudinal challenges in East Asia.

CONCLUSIONS

We recommend approaches and initiatives that aim to drive improvements in the utilization of RWE in healthcare decision-making in East Asia and other regions. We encourage large-scale collaborations that leverage the full range of skills offered by different stakeholders. Government agencies, hospitals, research organizations, patient groups, and the pharmaceutical industry must collaborate to ensure appropriate access to robust and reliable real-world data and seek alignment on how to address prioritized evidence needs. Increasingly, we believe that this work will be conducted by multidisciplinary teams with expertise in healthcare research and delivery, data science, and information technology. We hope this work will encourage further discussion among all stakeholders seeking to shape the RWE landscape in East Asia and other regions and drive next-generation healthcare.

Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions

Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson's correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.

Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers

As part of the HTx (Next Generation Health Technology Assessment) project, this study was aimed at identifying the main barriers for application of real-world evidence (RWE) for the purposes of health technology assessment in the Central and Eastern European countries. A mixed methods approach was employed to identify the main barriers: a scoping review of the literature and a series of discussions with stakeholders. Based on the applied approaches, we attempted to summarize the main barriers and challenges related to transferability of RWE in five main groups: technical, regulatory, clinical, scientific and perceptional barriers. Further research should pursue the development of detailed, consensus-based guidelines to improve the harmonization and standardization of RWE.

Baseline characteristics and secondary medication adherence among Medicare patients diagnosed with transthyretin amyloid cardiomyopathy and/or receiving tafamidis prescriptions: A retrospective analysis of a Medicare cohort

BACKGROUND: Transthyretin amyloid cardiomyopathy (ATTR-CM) is an underdiagnosed, life-threatening condition that mostly affects older persons. In May 2019, regulatory approval of tafamidis provided the first pharmacologic treatment of ATTR-CM. In the pivotal phase 3 Transthyretin Amyloidosis Cardiomyopathy Clinical Trial (ATTR-ACT), 97.2% of patients were classified as adherent (defined as taking ≥ 80% of scheduled doses). Given its recent approval, there is limited real-world evidence examining patient adherence to tafamidis. OBJECTIVE: To evaluate adherence patterns, demographics, and clinical characteristics of patients in the United States receiving tafamidis prescriptions through Medicare. Secondarily, we aimed to evaluate concomitant medications filled by this patient population. METHODS: We conducted a retrospective cohort study of US Medicare claims data, limited by the Health Insurance Portability and Accountability Act of 1996, in adult patients with an adjudicated pharmacy claim for tafamidis (tafamidis free acid 61-mg capsule once daily or tafamidis meglumine four 20-mg capsules once daily) between May 1, 2019, and June 30, 2021. Gaps in therapy were measured using day gaps between prescription refills and continuous measure of medication gaps. Implementation adherence was assessed through modified medication possession ratio (MPRm), medication refill adherence (MRA), and proportion of days covered (PDC). Patients were grouped based on Medicare coverage. Patients were analyzed by subgroups based on age and at the zip code level, via distressed communities index quartiles and rural-urban tiers. RESULTS: A total of 3,558 patients who received a prescription fill of a tafamidis formulation were identified using Medicare Fee-for-Service (FFS) and Medicare Advantage (MA) claims data from May 1, 2019, to June 30, 2021. The characteristics of this patient population were consistent with published literature, as 98.6% were older than 65 years, 53.4% were between 75 years and 84 years, and 81.5% were male. In the patient population receiving tafamidis refills, adherence was high across all 3 measures, with mean MPRm greater than 90% and mean MRA greater than 80%, across all age groups. Mean PDC adherence rates were 79% or more across all age groups. Concomitant medications were generally indicated for heart failure and thrombosis. Among monotherapy groups with similar demographic makeup, adherence was significantly higher among users of tafamidis free acid vs tafamidis meglumine (P < 0.0001 across all mean adherence measures). CONCLUSIONS: Our results demonstrate that real-world adherence to tafamidis in the Medicare population is high, regardless of age, zip code-level socioeconomic quartile, or geography. Adherence was higher among patients receiving tafamidis free acid, suggesting that the enhanced convenience of a single capsule once daily may positively contribute to adherence among patients with ATTR-CM. DISCLOSURES: Darrin Benjumea is an employee of Genesis Research who has been contracted by Pfizer, Inc., for involvement in this study. Andrew Peterson is an employee of University of the Sciences who has been contracted by Pfizer, Inc., for involvement in this study. Zach Bredl is an employee of Care Journey who has been contracted by Pfizer, Inc., for involvement in this study. Anuja Roy, Nick Marchant, Jose Alvir, Rahul Bhambri, Jason Kemner, and Bhash Parasuraman are employees of Pfizer, Inc., and own stock and/or stock options. This study was supported by Pfizer, Inc.

Machine Learning Methods in Health Economics and Outcomes Research-The PALISADE Checklist: A Good Practices Report of an ISPOR Task Force

Advances in machine learning (ML) and artificial intelligence offer tremendous potential benefits to patients. Predictive analytics using ML are already widely used in healthcare operations and care delivery, but how can ML be used for health economics and outcomes research (HEOR)? To answer this question, ISPOR established an emerging good practices task force for the application of ML in HEOR. The task force identified 5 methodological areas where ML could enhance HEOR: (1) cohort selection, identifying samples with greater specificity with respect to inclusion criteria; (2) identification of independent predictors and covariates of health outcomes; (3) predictive analytics of health outcomes, including those that are high cost or life threatening; (4) causal inference through methods, such as targeted maximum likelihood estimation or double-debiased estimation-helping to produce reliable evidence more quickly; and (5) application of ML to the development of economic models to reduce structural, parameter, and sampling uncertainty in cost-effectiveness analysis. Overall, ML facilitates HEOR through the meaningful and efficient analysis of big data. Nevertheless, a lack of transparency on how ML methods deliver solutions to feature selection and predictive analytics, especially in unsupervised circumstances, increases risk to providers and other decision makers in using ML results. To examine whether ML offers a useful and transparent solution to healthcare analytics, the task force developed the PALISADE Checklist. It is a guide for balancing the many potential applications of ML with the need for transparency in methods development and findings.

Methodological considerations on estimating medication adherence from self-report, electronic monitoring, and electronic healthcare databases using the TEOS framework

AIM

Measuring adherence to medication is complex due to the diversity of contexts in which medications are prescribed, dispensed, and used. The Timelines-Events-Objectives-Sources (TEOS) framework outlined a process to operationalize adherence. We aimed to develop practical recommendations for quantification of medication adherence using self-report (SR), electronic monitoring (EM), and electronic healthcare databases (EHD) consistent with the TEOS framework for adherence operationalization.

METHODS

An adherence methodology working group of the International Society for Medication Adherence (ESPACOMP) analysed implications of the process of medication adherence for all data sources and discussed considerations specific to SR, ED, and EHD regarding the information available on the prescribing, dispensing, recommended and actual use timelines, the four events relevant for distinguishing the adherence phases, the study objectives commonly addressed with each type of data, and the potential sources of measurement error and quality criteria applicable.

RESULTS

Four key implications for medication adherence measurement are common to all data sources: adherence is a comparison between two series of events (recommended and actual use); it refers to one or more specific medication(s); it applies to regular repeated events coinciding with known recommended dosing; and it requires separate measurement of the three adherence phases for a complete picture of patients' adherence. We propose recommendations deriving from these statements, and aspects to be considered in study design when measuring adherence with SR, EM and EHD using the TEOS framework.

CONCLUSION

The quality of medication adherence estimates is the result of several design choices that may optimize the data available.

Personalized Reimbursement Model (PRM) program: A real-world data platform of cancer drugs use to improve and personalize drug pricing and reimbursement in France

Objective

This article describes the Personalized Reimbursement Model (PRM) program methodology, limitations, achievement and perspectives in using real-world data of cancer drugs use to improve and personalize drug pricing and reimbursement in France.

Materials and methods

PRM platform aggregates Electronic Pharmacy Records (EPR) data from French medical centers (PRM centers) to build retrospective cohorts of patients treated with injectable cancer drugs in a hospital setting. Data extracted on January 1^st, 2020, from breast cancer (BC) patients who received trastuzumab, trastuzumab emtansin or pertuzumab since January 1st, 2011, and from lung cancer (LC) patients who received bevacizumab or atezolizumab since January 1^st, 2015, enabled recovering their injectable cancer drugs history from diagnosis date until December 30^th, 2019, and served as dataset for assessment.

Results

123 PRM centers provided data from 30,730 patients (25,660 BC and 5,070 LC patients respectively). Overall, 20,942 (82%) of BC and 4,716 (93%) of LC patients were analyzed. Completion rate was above 98% for patients characteristics, diagnostic and treatment related data. PRM centers cover 48% and 33% of BC and LC patients in-hospital therapeutic management in France, respectively. Distribution of BC and LC patients therapeutic management, by medical center category and geographic location, was similar in PRM centers to all French medical centers, ensuring the representativeness of the PRM platform.

Conclusion

PRM Platform enabled building a national database generating on demand Real-World Evidence based on EPR. This enabled the first performance-based risk-sharing arrangements based on PRM data, between the CEPS and Roche, for atezolizumab cancer immunotherapy in metastatic non-small cell lung cancer indication.

Evaluating fitness-for-use of electronic health records in pragmatic clinical trials: reported practices and recommendations

OBJECTIVE

To empirically explore how pragmatic clinical trials (PCTs) that used real-world data (RWD) assessed study-specific fitness-for-use.

METHODS

We conducted interviews and surveys with PCT teams who used electronic health record (EHR) data to ascertain endpoints. The survey cataloged key concerns about RWD, activities used to assess data fitness-for-use, and related barriers encountered by study teams. Patterns and commonalities across trials were used to develop recommendations for study-specific fitness-for-use assessments.

RESULTS

Of 15 invited trial teams, 7 interviews were conducted. Of 31 invited trials, 15 responded to the survey. Most respondents had prior experience using RWD (93%). Major concerns about EHR data were data reliability, missingness or incompleteness of EHR elements, variation in data quality across study sites, and presence of implausible or incorrect values. Although many PCTs conducted fitness-for-use activities (eg, data quality assessments, 11/14, 79%), less than a quarter did so before choosing a data source. Fitness-for-use activities, findings, and resulting study design changes were not often publically documented. Overall costs and personnel costs were barriers to fitness-for-use assessments.

DISCUSSION

These results support three recommendations for PCTs that use EHR data for endpoint ascertainment. Trials should detail the rationale and plan for study-specific fitness-for-use activities, conduct study-specific fitness-for-use assessments early in the prestudy phase to inform study design changes before the trial begins, and share results of fitness-for-use assessments and description of relevant challenges and facilitators.

CONCLUSION

These recommendations can help researchers and end-users of real-world evidence improve characterization of RWD reliability and relevance in the PCT-specific context.

A review of stakeholder recommendations for defining fit-for-purpose real-world evidence algorithms

Aim: The credibility and value of real-world evidence (RWE) are either supported or undermined by the algorithms (i.e., operational definitions) used. Methods: We conducted a targeted evidence review of key RWE decision makers' published recommendations on RWE algorithms through April 2021. Stakeholders were regulatory bodies, other governmental agencies and payer organizations. Results: Our review identified recommended criteria: relevance, validity, reliability, responsiveness, transparency and replicability, safety, feasibility and quality process. Stakeholders routinely recommended accuracy measures, subgroups evaluation and specific considerations for assessing exposures and covariates and the underlying real-world data (RWD) quality. Conclusion: The importance of stakeholder guidance on fit-for-purpose RWE algorithms is growing. We highlight gaps that future guidance and stakeholder recommendations could address.

The value of anticancer drugs - a regulatory view

The high prices of new anticancer drugs and the marginal added benefit perceived by some stakeholders have fuelled a debate on the value of anticancer drugs in the European Union, even though an agreed definition of what constitutes a drug's value does not exist. In this Perspective, we discuss the value of drugs from different viewpoints and objectives of decision makers: for regulators, assessment of the benefit-risk balance of a drug is a cornerstone for approval; payers rely on cost-effectiveness analyses carried out by health technology assessment agencies for reimbursement decisions; for patients, treatment choices are based on personal preferences and attitudes to risk; and clinicians can use several scales (such as the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS)) that have been developed as an attempt to measure value objectively. Although a unique definition that fully captures the concept of value is unlikely to emerge, herein we discuss the importance of understanding different perspectives, and how regulators can help to inform different decision makers.