Disproportionality analysis for signal detection of implantable cardioverter-defibrillator-related adverse events in the Food and Drug Administration Medical Device Reporting System

A Comprehensive Framework for Evaluating the Value Created by Real-World Evidence for Diverse Stakeholders: The Case for Coordinated Registry Networks

OBJECTIVES

This manuscript presents a comprehensive framework for the assessment of the value of real-world evidence (RWE) in healthcare decision-making. While RWE has been proposed to overcome some limitations of traditional, one-off studies, no systematic framework exists to measure if RWE actually lowers the burden. This framework aims to fill that gap by providing conceptual approaches for evaluating the time and cost efficiencies of RWE, thus guiding strategic investments in RWE infrastructure.

METHODS

The framework consists of four components: (114th Congress. 21st Century Cures Act.; 2015. https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf .) identification of stakeholders using and producing RWE, (National Health Council. Glossary of Patient Engagement Terms. Published 2019. Accessed May 18. 2021. https://nationalhealthcouncil.org/glossary-of-patient-engagement-terms/ .) understanding value propositions on how RWE can benefit stakeholders, (Center for Drug Evaluation and Research. CDER Patient-Focused Drug Development. U.S. Food & Drug Administration.) defining key performance indicators (KPIs), and (U.S. Department of Health and Human Services - Food and Drug Administration: Center for Devices and Radiological Health and Center for Biologics Evaluation and Research. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices - Guidance for Industry and Food and Drug Administration Staff. 2017. http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guida .) establishing metrics and case studies to assess value. KPIs are categorized as 'better, faster, or cheaper" as an indicator of value: better focusing on high-quality actionable evidence; 'faster,' denoting time-saving in evidence generation, and 'cheaper,' emphasizing cost-efficiency decision compared to methodologies that do not involve data routinely collected in clinical practice. Metrics and relevant case studies are tailored based on stakeholder value propositions and selected KPIs that can be used to assess what value has been created by using RWE compared to traditional evidence-generation approaches and comparing different RWE sources.

RESULTS

Operationalized through metrics and case studies drawn from the literature, the value of RWE is documented as improving treatment effect heterogeneity evaluation, expanding medical product labels, and expediting post-market compliance. RWE is also shown to reduce the cost and time required to produce evidence compared to traditional one-off approaches. An original example of a metric that measures the time saved by RWE methods to detect a signal of a product failure was presented based on analysis of the National Cardiovascular Disease Registry.

CONCLUSIONS

The framework presented in this manuscript offers a comprehensive approach for evaluating the value of RWE, applicable to all stakeholders engaged in leveraging RWE for healthcare decision-making. Through the proposed metrics and illustrated case studies, valuable insights are provided into the heightened efficiency, cost-effectiveness, and improved decision-making within clinical and regulatory domains facilitated by RWE. While this framework is primarily focused on medical devices, it could potentially inform the determination of RWE value in other medical products. By discerning the variations in cost, time, and data utility among various evidence-generation methods, stakeholders are empowered to invest strategically in RWE infrastructure and shape future research endeavors.

Assessing the Reproducibility of Research Based on the Food and Drug Administration Manufacturer and User Facility Device Experience Data

OBJECTIVE

This article aims to assess the reproducibility of Manufacturer and User Facility Device Experience (MAUDE) data-driven studies by analyzing the data queries used in their research processes.

METHODS

Studies using MAUDE data were sourced from PubMed by searching for "MAUDE" or "Manufacturer and User Facility Device Experience" in titles or abstracts. We manually chose articles with executable queries. The reproducibility of each query was assessed by replicating it in the MAUDE Application Programming Interface. The reproducibility of a query is determined by a reproducibility coefficient that ranges from 0.95 to 1.05. This coefficient is calculated by comparing the number of medical device reports (MDRs) returned by the reproduced queries to the number of reported MDRs in the original studies. We also computed the reproducibility ratio, which is the fraction of reproducible queries in subgroups divided by the query complexity, the device category, and the presence of a data processing flow.

RESULTS

As of August 8, 2022, we identified 523 articles from which 336 contained queries, and 60 of these were executable. Among these, 14 queries were reproducible. Queries using a single field like product code, product class, or brand name showed higher reproducibility (50%, 33.3%, 31.3%) compared with other fields (8.3%, P = 0.037). Single-category device queries exhibited a higher reproducibility ratio than multicategory ones, but without statistical significance (27.1% versus 8.3%, P = 0.321). Studies including a data processing flow had a higher reproducibility ratio than those without, although this difference was not statistically significant (42.9% versus 17.4%, P = 0.107).

CONCLUSIONS

Our findings indicate that the reproducibility of queries in MAUDE data-driven studies is limited. Enhancing this requires the development of more effective MAUDE data query strategies and improved application programming interfaces.

Validation of CORE-MD PMS Support Tool: A Novel Strategy for Aggregating Information from Notices of Failures to Support Medical Devices' Post-Market Surveillance

INTRODUCTION

The EU Medical Device Regulation 2017/745 defines new rules for the certification and post-market surveillance of medical devices (MD), including an additional review by Expert Panels of clinical evaluation data for high-risk MD if reports and alerts suggest possibly associated increased risks. Within the EU-funded CORE-MD project, our aim was to develop a tool to support such process in which web-accessible safety notices (SN) are automatically retrieved and aggregated based on their specific MD categories and the European Medical Device Nomenclature (EMDN) classification by applying an Entity Resolution (ER) approach to enrich data integrating different sources. The performance of such approach was tested through a pilot study on the Italian data.

METHODS

Information relevant to 7622 SN from 2009 to 2021 was retrieved from the Italian Ministry of Health website by Web scraping. For incomplete EMDN data (68%), the MD best match was searched within a list of about 1.5 M MD on the Italian market, using Natural Language Processing techniques and pairwise ER. The performance of this approach was tested on the 2440 SN (32%) already provided with the EMDN code as reference standard.

RESULTS

The implemented ER method was able to correctly assign the correct manufacturer to the MD in each SN in 99% of the cases. Moreover, the correct EMDN code at level 1 was assigned in 2382 SN (97.62%), at level 2 in 2366 SN (96.97%) and at level 3 in 2329 SN (95.45%).

CONCLUSION

The proposed approach was able to cope with the incompleteness of the publicly available data in the SN. In this way, grouping of SN relevant to a specific MD category/group/type could be used as possible sentinel for increased rates in reported serious incidents in high-risk MD.

Challenges Associated with the Safety Signal Detection Process for Medical Devices

Background

Previous safety issues involving medical devices have stressed the need for better safety signal detection. Various European Union (EU) national competent authorities have started to focus on strengthening the analysis of vigilance data. Consequently, article 90 of the new EU regulation states that the European Commission shall put in place systems and processes to actively monitor medical device safety signals.

Methods

A systematic literature review was conducted to synthesize the current state of knowledge and investigate the present tools used for medical device safety signal detection. An electronic literature search was performed in Embase, Medline, Cochrane, Web of science, and Google scholar from inception until January 2017. Articles that included terms related to medical devices and terms associated with safety were selected. A further selection was based on the abstract review. A full review of the remaining articles was conducted to decide on which articles finally to consider relevant for this review. Completeness was assessed based on the content of the articles.

Results

Our search resulted in a total of 20,819 articles, of which 24 met the inclusion criteria and were subject to data extraction and completeness scoring. A wide range of data sources, especially spontaneous reporting systems and registries, used for the detection and assessment of product problems and patient harms associated with the use of medical devices, were studied. Coding is remarkably heterogeneous, no agreement on the preferred methods for signal detection exists, and no gold standard for signal detection has been established thus far.

Conclusion

Data source harmonization, the development of gold standard signal detection methodologies and the standardization of coding dictionaries are amongst the recommendations to support the implementation of a new proactive approach to signal detection. The new safety surveillance system will be able to use real-world evidence to support regulatory decision-making across all jurisdictions.

Blockchain technology applications to postmarket surveillance of medical devices

INTRODUCTION

The amount of mandatory data that needs to be analyzed as part of a medical device postmarket surveillance (PMS) system has grown exponentially in recent times. This is a consequence of increasingly demanding and complex regulatory requirements from Health Authorities, aimed at a better understanding of the medical device safety evaluation. Proactive approaches to PMS processes are becoming more necessary as regulators increase the scrutiny of device safety. New technologies have been explored to address some of the challenges associated with this changing regulatory environment.

AREAS COVERED

This paper focuses on the different technical aspects of blockchain and how this new technology has the potential to support the ongoing efforts to improve the PMS system for medical devices.

EXPERT OPINION

To address these challenges, we suggest to generate a private PMS data permissioned blockchain with a proof-of-authority consensus mechanism, to which only a restricted number of designated and audited participants have authorization to validate transactions and add them to the PMS data blockchain ledger. Blockchain has the potential to support a more efficient approach, which could offer many advantages to the different stakeholders involved in the PMS process, such as supporting with new regulatory initiatives.

Medical device active surveillance of spontaneous reports: A literature review of signal detection methods

PURPOSE

The collection and analysis of real-world data for the active monitoring of medical device performance and safety has become increasingly important. Spontaneous reports, such as those in the Food & Drug Administration's (FDA's) Manufacturer and User Facility Device Experience (MAUDE), provide early warning of potential issues with marketed devices. This review synthesizes the current literature on medical device surveillance signal detection and provides a framework for application of methods to active surveillance of spontaneous reports.

METHODS

Ovid MEDLINE, Ovid Embase, Scopus, and PubMed databases were systematically searched up to January 2019. Additionally, five methods articles from pharmacovigilance were added that had potential applications to medical devices.

RESULTS

Among 105 articles included, the most common source of data (84%) was registries; median time between data collection and publication was 8 years. Surgical procedure outcome signal detection articles comprised 83% while 14% were on device outcome signal detection. The most common family of methods cited (70%) was Sequential Probability Ratio.

CONCLUSION

Application of any signal detection algorithm requires careful consideration of influential factors, data limitations, and algorithmic assumptions. We recommend approaches using disproportionality, statistical process control, and sequential probability tests and provide R packages to further development efforts. The small number of published examples suggest that further development of statistical methods and technological solutions to analyze large amounts of data for device safety and performance is needed. Fundamental differences in products, data infrastructure, and the regulatory landscape suggest that medical device vigilance requires its own body of research distinct from pharmacovigilance.

Increasing Patient Engagement in Pharmacovigilance Through Online Community Outreach and Mobile Reporting Applications: An Analysis of Adverse Event Reporting for the Essure Device in the US

Background

Preparing and submitting a voluntary adverse event (AE) report to the US Food and Drug Administration (FDA) for a medical device typically takes 40 min. User-friendly Web and mobile reporting apps may increase efficiency. Further, coupled with strategies for direct patient involvement, patient engagement in AE reporting may be improved. In 2012, the FDA Center for Devices and Radiologic Health (CDRH) launched a free, public mobile AE reporting app, MedWatcher, for patients and clinicians. During the same year, a patient community on Facebook adopted the app to submit reports involving a hysteroscopic sterilization device, brand name Essure^®.

Methods

Patient community outreach was conducted to administrators of the group “Essure Problems” (approximately 18,000 members as of June 2015) to gather individual case safety reports (ICSRs). After agreeing on key reporting principles, group administrators encouraged members to report via the app. Semi-structured forms in the app mirrored fields of the MedWatch 3500 form. ICSRs were transmitted to CDRH via an electronic gateway, and anonymized versions were posted in the app. Data collected from May 11, 2013 to December 7, 2014 were analyzed. Narrative texts were coded by trained and certified MedDRA coders (version 17). Descriptive statistics and metrics, including VigiGrade completeness scores, were analyzed. Various incentives and motivations to report in the Facebook group were observed.

Results

The average Essure AE report took 11.4 min (±10) to complete. Submissions from 1349 women, average age 34 years, were analyzed. Serious events, including hospitalization, disability, and permanent damage after implantation, were reported by 1047 women (77.6 %). A total of 13,135 product–event pairs were reported, comprising 327 unique preferred terms, most frequently fatigue (n = 491), back pain (468), and pelvic pain (459). Important medical events (IMEs), most frequently mental impairment (142), device dislocation (108), and salpingectomy (62), were reported by 598 women (44.3 %). Other events of interest included loss of libido (n = 115); allergy to metals (109), primarily nickel; and alopecia (252). VigiGrade completeness scores were high, averaging 0.80 (±0.15). Reports received via the mobile app were considered “well documented” 55.9 % of the time, compared with an international average of 13 % for all medical products. On average, there were 15 times more reports submitted per month via the app with patient community support versus traditional pharmacovigilance portals.

Conclusions

Outreach via an online patient community, coupled with an easy-to-use app, allowed for rapid and detailed ICSRs to be submitted, with gains in efficiency. Two-way communication and public posting of narratives led to successful engagement within a Motivation-Incentive-Activation-Behavior framework, a conceptual model for successful crowdsourcing. Reports submitted by patients were considerably more complete than those submitted by physicians in routine spontaneous reports. Further research is needed to understand how biases operate differently from those of traditional pharmacovigilance.

Use of data mining at the Food and Drug Administration

Objectives This article summarizes past and current data mining activities at the United States Food and Drug Administration (FDA).

Target audience We address data miners in all sectors, anyone interested in the safety of products regulated by the FDA (predominantly medical products, food, veterinary products and nutrition, and tobacco products), and those interested in FDA activities.

Scope Topics include routine and developmental data mining activities, short descriptions of mined FDA data, advantages and challenges of data mining at the FDA, and future directions of data mining at the FDA.

Digital drug safety surveillance: monitoring pharmaceutical products in twitter

Background

Traditional adverse event (AE) reporting systems have been slow in adapting to online AE reporting from patients, relying instead on gatekeepers, such as clinicians and drug safety groups, to verify each potential event. In the meantime, increasing numbers of patients have turned to social media to share their experiences with drugs, medical devices, and vaccines.

Objective

The aim of the study was to evaluate the level of concordance between Twitter posts mentioning AE-like reactions and spontaneous reports received by a regulatory agency.

Methods

We collected public English-language Twitter posts mentioning 23 medical products from 1 November 2012 through 31 May 2013. Data were filtered using a semi-automated process to identify posts with resemblance to AEs (Proto-AEs). A dictionary was developed to translate Internet vernacular to a standardized regulatory ontology for analysis (MedDRA^®). Aggregated frequency of identified product-event pairs was then compared with data from the public FDA Adverse Event Reporting System (FAERS) by System Organ Class (SOC).

Results

Of the 6.9 million Twitter posts collected, 4,401 Proto-AEs were identified out of 60,000 examined. Automated, dictionary-based symptom classification had 72 % recall and 86 % precision. Similar overall distribution profiles were observed, with Spearman rank correlation rho of 0.75 (p < 0.0001) between Proto-AEs reported in Twitter and FAERS by SOC.

Conclusion

Patients reporting AEs on Twitter showed a range of sophistication when describing their experience. Despite the public availability of these data, their appropriate role in pharmacovigilance has not been established. Additional work is needed to improve data acquisition and automation.