Can machine learning complement traditional medical device surveillance? A case study of dual-chamber implantable cardioverter-defibrillators

A machine learning framework to adjust for learning effects in medical device safety evaluation

OBJECTIVES

Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects.

MATERIALS AND METHODS

A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection.

RESULTS

Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios' 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied.

DISCUSSION

A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present.

CONCLUSION

Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.

Role of artificial intelligence in defibrillators: a narrative review

Automated external defibrillators (AEDs) and implantable cardioverter defibrillators (ICDs) are used to treat life-threatening arrhythmias. AEDs and ICDs use shock advice algorithms to classify ECG tracings as shockable or non-shockable rhythms in clinical practice. Machine learning algorithms have recently been assessed for shock decision classification with increasing accuracy. Outside of rhythm classification alone, they have been evaluated in diagnosis of causes of cardiac arrest, prediction of success of defibrillation and rhythm classification without the need to interrupt cardiopulmonary resuscitation. This review explores the many applications of machine learning in AEDs and ICDs. While these technologies are exciting areas of research, there remain limitations to their widespread use including high processing power, cost and the ‘black-box’ phenomenon.

Indirectly-Supervised Anomaly Detection of Clinically-Meaningful Health Events from Smart Home Data

Anomaly detection techniques can extract a wealth of information about unusual events. Unfortunately, these methods yield an abundance of findings that are not of interest, obscuring relevant anomalies. In this work, we improve upon traditional anomaly detection methods by introducing Isudra, an Indirectly-Supervised Detector of Relevant Anomalies from time series data. Isudra employs Bayesian optimization to select time scales, features, base detector algorithms, and algorithm hyperparameters that increase true positive and decrease false positive detection. This optimization is driven by a small amount of example anomalies, driving an indirectly-supervised approach to anomaly detection. Additionally, we enhance the approach by introducing a warm start method that reduces optimization time between similar problems. We validate the feasibility of Isudra to detect clinically-relevant behavior anomalies from over 2 million sensor readings collected in 5 smart homes, reflecting 26 health events. Results indicate that indirectly-supervised anomaly detection outperforms both supervised and unsupervised algorithms at detecting instances of health-related anomalies such as falls, nocturia, depression, and weakness.

Association Between Industry Payments to Physicians and Device Selection in ICD Implantation

IMPORTANCE

Little is known about the association between industry payments and medical device selection.

OBJECTIVE

To examine the association between payments from device manufacturers to physicians and device selection for patients undergoing first-time implantation of a cardioverter-defibrillator (ICD) or cardiac resynchronization therapy-defibrillator (CRT-D).

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study, patients who received a first-time ICD or CRT-D device from any of the 4 major manufacturers (January 1, 2016-December 31, 2018) were identified. The data from the National Cardiovascular Data Registry ICD Registry was linked with the Open Payments Program's payment data. Patients were categorized into 4 groups (A, B, C, and D) corresponding to the manufacturer from which the physician who performed the implantation received the largest payment. For each patient group, the proportion of patients who received a device from the manufacturer that provided the largest payment to the physician who performed implantation was determined. Within each group, the absolute difference in proportional use of devices between the manufacturer that made the highest payment and the proportion of devices from the same manufacturer in the entire study cohort (expected prevalence) was calculated.

EXPOSURES

Manufacturers' payments to physicians who performed an ICD or CRT-D implantation.

MAIN OUTCOMES AND MEASURES

The primary outcome of the study was the manufacturer of the device used for the implantation.

RESULTS

Over a 3-year period, 145 900 patients (median age, 65 years; 29.6% women) received ICD or CRT-D devices from the 4 manufacturers implanted by 4435 physicians at 1763 facilities. Among these physicians, 4152 (94%) received payments from device manufacturers ranging from $2 to $323 559 with a median payment of $1211 (interquartile range, $390-$3702). Between 38.5% and 54.7% of patients received devices from the manufacturers that had provided physicians with the largest payments. Patients were substantially more likely to receive devices made by the manufacturer that provided the largest payment to the physician who performed implantation than they were from each other individual manufacturer. The absolute differences in proportional use from the expected prevalence were 22.4% (95% CI, 21.9%-22.9%) for manufacturer A; 14.5% (95% CI, 14.0%-15.0%) for manufacturer B; 18.8% (95% CI, 18.2%-19.4%) for manufacturer C; and 30.6% (95% CI, 30.0%-31.2%) for manufacturer D.

CONCLUSIONS AND RELEVANCE

In this cross-sectional study, a large proportion of ICD or CRT-D implantations were performed by physicians who received payments from device manufacturers. Patients were more likely to receive ICD or CRT-D devices from the manufacturer that provided the highest total payment to the physician who performed an ICD or CRT-D implantation than each other manufacturer individually.

Medical device active surveillance of spontaneous reports: A literature review of signal detection methods

PURPOSE

The collection and analysis of real-world data for the active monitoring of medical device performance and safety has become increasingly important. Spontaneous reports, such as those in the Food & Drug Administration's (FDA's) Manufacturer and User Facility Device Experience (MAUDE), provide early warning of potential issues with marketed devices. This review synthesizes the current literature on medical device surveillance signal detection and provides a framework for application of methods to active surveillance of spontaneous reports.

METHODS

Ovid MEDLINE, Ovid Embase, Scopus, and PubMed databases were systematically searched up to January 2019. Additionally, five methods articles from pharmacovigilance were added that had potential applications to medical devices.

RESULTS

Among 105 articles included, the most common source of data (84%) was registries; median time between data collection and publication was 8 years. Surgical procedure outcome signal detection articles comprised 83% while 14% were on device outcome signal detection. The most common family of methods cited (70%) was Sequential Probability Ratio.

CONCLUSION

Application of any signal detection algorithm requires careful consideration of influential factors, data limitations, and algorithmic assumptions. We recommend approaches using disproportionality, statistical process control, and sequential probability tests and provide R packages to further development efforts. The small number of published examples suggest that further development of statistical methods and technological solutions to analyze large amounts of data for device safety and performance is needed. Fundamental differences in products, data infrastructure, and the regulatory landscape suggest that medical device vigilance requires its own body of research distinct from pharmacovigilance.

Advanced implantable drug delivery technologies: transforming the clinical landscape of therapeutics for chronic diseases

Chronic diseases account for the majority of all deaths worldwide, and their prevalence is expected to escalate in the next 10 years. Because chronic disorders require long-term therapy, the healthcare system must address the needs of an increasing number of patients. The use of new drug administration routes, specifically implantable drug delivery devices, has the potential to reduce treatment-monitoring clinical visits and follow-ups with healthcare providers. Also, implantable drug delivery devices can be designed to maintain drug concentrations in the therapeutic window to achieve controlled, continuous release of therapeutics over extended periods, eliminating the risk of patient non-compliance to oral treatment. A higher local drug concentration can be achieved if the device is implanted in the affected tissue, reducing systemic adverse side effects and decreasing the challenges and discomfort of parenteral treatment. Although implantable drug delivery devices have existed for some time, interest in their therapeutic potential is growing, with a global market expected to reach over $12 billion USD by 2018. This review discusses implantable drug delivery technologies in an advanced stage of development or in clinical use and focuses on the state-of-the-art of reservoir-based implants including pumps, electromechanical systems, and polymers, sites of implantation and side effects, and deployment in developing countries.

Robotics in cardiac surgery

A summary of its uses in mitral valve surgery and coronary artery revascularisation.

Quantifying the utilization of medical devices necessary to detect postmarket safety differences: A case study of implantable cardioverter defibrillators

PURPOSE

To estimate medical device utilization needed to detect safety differences among implantable cardioverter defibrillators (ICDs) generator models and compare these estimates to utilization in practice.

METHODS

We conducted repeated sample size estimates to calculate the medical device utilization needed, systematically varying device-specific safety event rate ratios and significance levels while maintaining 80% power, testing 3 average adverse event rates (3.9, 6.1, and 12.6 events per 100 person-years) estimated from the American College of Cardiology's 2006 to 2010 National Cardiovascular Data Registry of ICDs. We then compared with actual medical device utilization.

RESULTS

At significance level 0.05 and 80% power, 34% or fewer ICD models accrued sufficient utilization in practice to detect safety differences for rate ratios <1.15 and an average event rate of 12.6 events per 100 person-years. For average event rates of 3.9 and 12.6 events per 100 person-years, 30% and 50% of ICD models, respectively, accrued sufficient utilization for a rate ratio of 1.25, whereas 52% and 67% for a rate ratio of 1.50. Because actual ICD utilization was not uniformly distributed across ICD models, the proportion of individuals receiving any ICD that accrued sufficient utilization in practice was 0% to 21%, 32% to 70%, and 67% to 84% for rate ratios of 1.05, 1.15, and 1.25, respectively, for the range of 3 average adverse event rates.

CONCLUSIONS

Small safety differences among ICD generator models are unlikely to be detected through routine surveillance given current ICD utilization in practice, but large safety differences can be detected for most patients at anticipated average adverse event rates.