1
|
Arshad F, Schuemie MJ, Bu F, Minty EP, Alshammari TM, Lai LYH, Duarte-Salles T, Fortin S, Nyberg F, Ryan PB, Hripcsak G, Prieto-Alhambra D, Suchard MA. Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance. Drug Saf 2023; 46:797-807. [PMID: 37328600 PMCID: PMC10345011 DOI: 10.1007/s40264-023-01324-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/29/2023] [Indexed: 06/18/2023]
Abstract
INTRODUCTION Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' The extent to which serial testing in real-world studies improves or hinders overall performance in terms of sensitivity and specificity remains unknown. METHODS We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared type I and II errors before and after empirical calibration for historical comparator, self-controlled case series (SCCS), and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes. RESULTS The historical comparator design mostly had fewer type II errors than SCCS. SCCS had fewer type I errors than the historical comparator. Before empirical calibration, the serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, type I errors returned to nominal; sensitivity was lowest when the methods were combined. CONCLUSION While serial combination produced fewer false-positive signals compared with the most specific method, it generated more false-negative signals compared with the most sensitive method. Using a historical comparator design followed by an SCCS analysis yielded decreased sensitivity in evaluating safety signals relative to a one-stage SCCS approach. While the current use of serial testing in vaccine surveillance may provide a practical paradigm for signal identification and triage, single epidemiological designs should be explored as valuable approaches to detecting signals.
Collapse
Affiliation(s)
- Faaizah Arshad
- Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA
- Observational Health Data Sciences and Informatics, New York, NY, USA
| | - Martijn J Schuemie
- Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA
- Observational Health Data Sciences and Informatics, New York, NY, USA
- Observational Health Data Analytics, Janssen R&D, Titusville, NJ, USA
| | - Fan Bu
- Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA
- Observational Health Data Sciences and Informatics, New York, NY, USA
| | - Evan P Minty
- O'Brien Institute for Public Health, Faculty of Medicine, University of Calgary, Calgary, AB, Canada
| | - Thamir M Alshammari
- Medication Safety Research Chair, King Saud University, Riyadh, Saudi Arabia
| | - Lana Y H Lai
- Division of Informatics, Imaging and Data Sciences, University of Manchester, Manchester, UK
| | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Stephen Fortin
- Observational Health Data Analytics, Janssen R&D, Titusville, NJ, USA
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Göteborg, Sweden
| | - Patrick B Ryan
- Observational Health Data Sciences and Informatics, New York, NY, USA
- Observational Health Data Analytics, Janssen R&D, Titusville, NJ, USA
| | - George Hripcsak
- Observational Health Data Sciences and Informatics, New York, NY, USA
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
- Medical Informatics Services, New York-Presbyterian Hospital, New York, NY, USA
| | - Daniel Prieto-Alhambra
- Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK
- Health Data Sciences, Medical Informatics, Erasmus Medical Center University, Rotterdam, The Netherlands
| | - Marc A Suchard
- Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA.
- Observational Health Data Sciences and Informatics, New York, NY, USA.
- Department of Human Genetics, University of California, Los Angeles, Los Angeles, CA, USA.
- VA Informatics and Computing Infrastructure, US Department of Veterans Affairs, Salt Lake City, UT, USA.
| |
Collapse
|
2
|
Kistler CE, Zimmerman S, Khairat S. Health Information Technology Challenges and Innovations in Long-Term Care. J Am Med Dir Assoc 2021; 22:981-983. [PMID: 33896713 DOI: 10.1016/j.jamda.2021.03.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2021] [Accepted: 03/23/2021] [Indexed: 10/21/2022]
Affiliation(s)
- Christine E Kistler
- Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Department of Family Medicine, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
| | - Sheryl Zimmerman
- Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Schools of Social Work and Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Saif Khairat
- School of Nursing, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Carolina Health Informatics Program, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
3
|
Walker AM. Complementary hypotheses in safety surveillance. Seq Anal 2021. [DOI: 10.1080/07474946.2020.1823195] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
4
|
Haynes K. Preparing for COVID-19 vaccine safety surveillance: A United States perspective. Pharmacoepidemiol Drug Saf 2020; 29:1529-1531. [PMID: 32978861 PMCID: PMC7537525 DOI: 10.1002/pds.5142] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2020] [Revised: 07/11/2020] [Accepted: 09/21/2020] [Indexed: 11/23/2022]
|
5
|
Nelson JC, Ulloa-Pérez E, Bobb JF, Maro JC. Leveraging the entire cohort in drug safety monitoring: part 1 methods for sequential surveillance that use regression adjustment or weighting to control confounding in a multisite, rare event, distributed data setting. J Clin Epidemiol 2019; 112:77-86. [PMID: 31108199 DOI: 10.1016/j.jclinepi.2019.04.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2017] [Revised: 03/01/2019] [Accepted: 04/04/2019] [Indexed: 10/26/2022]
Abstract
OBJECTIVE Study designs involving self-controlled or exposure-matched samples are commonly used to monitor postmarket vaccine and drug safety, and they use a subset of the available larger cohort. This article overviews group sequential methods designed for observational data safety monitoring that use the whole exposed and unexposed cohorts by implementing regression adjustment or weighting to control confounding. METHODS We summarize what is known about the performance of "whole cohort" methods in multisite health plan data networks such as the Sentinel System of the Food and Drug Administration, where outcomes are rare, individual-level patient data cannot be pooled across sites, site heterogeneity is large, and data are dynamically updated over time. RESULTS Group sequential estimation and testing methods that use regression or weighting can flexibly handle electronic health care data's unpredictability, including an uncertain rate of new product uptake, variable composition of the population over time, and data changes due to dynamic administrative updates. Regression and weighting methods generally have higher power, faster signal detection, and fewer practical challenges compared with some design-based confounder adjustment methods. CONCLUSION Group sequential regression adjustment and weighting approaches are feasible and underused in practice. They leverage more information than designs that involved sampling and increase power to detect rare adverse effects without increasing bias.
Collapse
Affiliation(s)
- Jennifer C Nelson
- Biostatistics Unit, Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA; Department of Biostatistics, University of Washington, Seattle, WA, USA.
| | - Ernesto Ulloa-Pérez
- Biostatistics Unit, Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA; Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Jennifer F Bobb
- Biostatistics Unit, Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA; Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Judith C Maro
- Department of Population Medicine, Harvard Medical School, Boston, MA, USA; Harvard Pilgrim Health Care Institute, Boston, MA, USA
| |
Collapse
|
6
|
Walker AM. Conditional power as an aid in making interim decisions in observational studies. Eur J Epidemiol 2018; 33:777-784. [PMID: 29808341 DOI: 10.1007/s10654-018-0413-9] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2017] [Accepted: 05/22/2018] [Indexed: 01/19/2023]
Abstract
Conditional power combines the findings of a partially completed study with assumptions about the future. The goal is to estimate the probability that the eventual study result will be incompatible with a criterion value, such as acceptable risk or the null hypothesis. Some history and motivation for conditional power calculations are provided, with examples illustrating the application to drug safety studies. This is an expository article suggesting that conditional power, which is well-established in clinical trials research, also has application to observational studies. The utility may be highest in regulatory settings where resources are limited and interim decisions have to be made accurately in the shortest possible time.
Collapse
Affiliation(s)
- Alexander Muir Walker
- World Health Information Science Consultants, 275 Grove Street, Suite 2-400, Newton, MA, 02466, USA.
| |
Collapse
|
7
|
Martin D, Gagne JJ, Gruber S, Izem R, Nelson JC, Nguyen MD, Ouellet-Hellstrom R, Schneeweiss S, Toh S, Walker AM. Sequential surveillance for drug safety in a regulatory environment. Pharmacoepidemiol Drug Saf 2018; 27:707-712. [PMID: 29504168 DOI: 10.1002/pds.4407] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2017] [Revised: 01/19/2018] [Accepted: 01/25/2018] [Indexed: 01/05/2023]
Affiliation(s)
- David Martin
- Office of the Center Director, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA
| | - Joshua J Gagne
- Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA
| | - Susan Gruber
- Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA
| | - Rima Izem
- Office of Biostatistics, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA
| | - Jennifer C Nelson
- Biostatistics Unit, Group Health Research Institute, Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Michael D Nguyen
- Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA
| | - Rita Ouellet-Hellstrom
- Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA
| | - Sebastian Schneeweiss
- Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA
| | - Sengwee Toh
- Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA
| | | |
Collapse
|
8
|
Zhou X, Bao W, Gaffney M, Shen R, Young S, Bate A. Assessing performance of sequential analysis methods for active drug safety surveillance using observational data. J Biopharm Stat 2017; 28:668-681. [PMID: 29157113 DOI: 10.1080/10543406.2017.1372776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
The routine use of sequential methods is well established in clinical studies. Recently, there has been increasing interest in applying these methods to prospectively monitor the safety of newly approved drugs through accrual of real-world data. However, the application to marketed drugs using real-world data has been limited and work is needed to determine which sequential approaches are most suited to such data. In this study, the conditional sequential sampling procedure (CSSP), a group sequential method, was compared with a log-linear model with Poisson distribution (LLMP) through a SAS procedure (PROC GENMOD) combined with an alpha-spending function on two large longitudinal US administrative health claims databases. Relative performance in identifying known drug-outcome associations was examined using a set of 50 well-studied drug-outcome pairs. The study finds that neither method correctly identified all pairs but that LLMP often provides better ability and shorter time for identifying the known drug-outcome associations with superior computational performance when compared with CSSP, albeit with more false positives. With the features of flexible confounding control and ease of implementation, LLMP may be a good alternative or complement to CSSP.
Collapse
Affiliation(s)
- Xiaofeng Zhou
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| | - Warren Bao
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| | - Mike Gaffney
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| | - Rongjun Shen
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| | - Sarah Young
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| | - Andrew Bate
- a Epidemiology , Worldwide Safety and Regulatory, Pfizer Inc , New York , NY , USA
| |
Collapse
|