Bias, Precision and Timeliness of Historical (Background) Rate Comparison Methods for Vaccine Safety Monitoring: An Empirical Multi-Database Analysis

Observational methods for COVID-19 vaccine effectiveness research: an empirical evaluation and target trial emulation

BACKGROUND

There are scarce data on best practices to control for confounding in observational studies assessing vaccine effectiveness to prevent COVID-19. We compared the performance of three well-established methods [overlap weighting, inverse probability treatment weighting and propensity score (PS) matching] to minimize confounding when comparing vaccinated and unvaccinated people. Subsequently, we conducted a target trial emulation to study the ability of these methods to replicate COVID-19 vaccine trials.

METHODS

We included all individuals aged ≥75 from primary care records from the UK [Clinical Practice Research Datalink (CPRD) AURUM], who were not infected with or vaccinated against SARS-CoV-2 as of 4 January 2021. Vaccination status was then defined based on first COVID-19 vaccine dose exposure between 4 January 2021 and 28 January 2021. Lasso regression was used to calculate PS. Location, age, prior observation time, regional vaccination rates, testing effort and COVID-19 incidence rates at index date were forced into the PS. Following PS weighting and matching, the three methods were compared for remaining covariate imbalance and residual confounding. Last, a target trial emulation comparing COVID-19 at 3 and 12 weeks after first vaccine dose vs unvaccinated was conducted.

RESULTS

Vaccinated and unvaccinated cohorts comprised 583 813 and 332 315 individuals for weighting, respectively, and 459 000 individuals in the matched cohorts. Overlap weighting performed best in terms of minimizing confounding and systematic error. Overlap weighting successfully replicated estimates from clinical trials for vaccine effectiveness for ChAdOx1 (57%) and BNT162b2 (75%) at 12 weeks.

CONCLUSION

Overlap weighting performed best in our setting. Our results based on overlap weighting replicate previous pivotal trials for the two first COVID-19 vaccines approved in Europe.

Sudden Sensorineural Hearing Loss Following Immunization With BNT162b2 or mRNA-1273: A Danish Population-Based Cohort Study

OBJECTIVE

To compare the occurrence of sudden sensorineural hearing loss following immunization with BNT162b2 (Comirnaty®; Pfizer BioNTech) or mRNA-1273 (Spikevax®; Moderna) to the occurrence among unvaccinated individuals.

STUDY DESIGN

Cohort study.

SETTING

Nationwide Danish health care registers comprised all Danish residents living in Denmark on October 1, 2020, who were 18 years or older or turned 18 in 2021.

METHODS

We compared the occurrence of sudden sensorineural hearing loss following immunization with BNT162b2 (Comirnaty®; Pfizer BioNTech) or mRNA-1273 (Spikevax®; Moderna) (first, second, or third dose) against unvaccinated person time. Secondary outcomes were a first-ever hospital diagnosis of vestibular neuritis and a hearing examination, by an ear-nose-throat (ENT) specialist, followed by a prescription of moderate to high-dose prednisolone.

RESULTS

BNT162b2 or mRNA-1273 vaccine was not associated with an increased risk of receiving a discharge diagnosis of sudden sensorineural hearing loss (adjusted hazard ratio [HR]: 0.99, confidence interval [CI]: 0.59-1.64) or vestibular neuritis (adjusted HR: 0.94, CI: 0.69-1.24). We found a slightly increased risk (adjusted HR: 1.40, CI, 1.08-1.81) of initiating moderate to high-dose oral prednisolone following a visit to an ENT specialist within 21 days from receiving a messenger RNA (mRNA)-based Covid-19 vaccine.

CONCLUSION

Our findings do not suggest an increased risk of sudden sensorineural hearing loss or vestibular neuritis following mRNA-based COVID-19 vaccination. mRNA-Covid-19 vaccination may be associated with a small excess risk of a visit to an ENT specialist visit followed by a prescription of moderate to high doses of prednisolone.

Adverse Events Following the BNT162b2 mRNA COVID-19 Vaccine (Pfizer-BioNTech) in Aotearoa New Zealand

INTRODUCTION

In February 2021, New Zealand began its largest ever immunisation programme with the BNT162b2 mRNA coronavirus disease 2019 (COVID-19) vaccine.

OBJECTIVE

We aimed to understand the association between 12 adverse events of special interest (AESIs) and a primary dose of BNT162b2 in the New Zealand population aged ≥5 years from 19 February 2021 through 10 February 2022.

METHODS

Using national electronic health records, the observed rates of AESIs within a risk period (1-21 days) following vaccination were compared with the expected rates based on background data (2014-2019). Standardised incidence ratios (SIRs) were estimated for each AESI with 95% confidence intervals (CIs) using age group-specific background rates. The risk difference was calculated to estimate the excess or reduced number of events per 100,000 persons vaccinated in the risk period.

RESULTS

As of 10 February 2022, 4,277,163 first doses and 4,114,364 second doses of BNT162b2 had been administered to the eligible New Zealand population aged ≥5 years. The SIRs for 11 of the 12 selected AESIs were not statistically significantly increased post vaccination. The SIR (95% CI) for myo/pericarditis following the first dose was 2.3 (1.8-2.7), with a risk difference (95% CI) of 1.3 (0.9-1.8), per 100,000 persons vaccinated, and 4.0 (3.4-4.6), with a risk difference of 3.1 (2.5-3.7), per 100,000 persons vaccinated following the second dose. The highest SIR was 25.6 (15.5-37.5) in the 5-19 years age group, following the second dose of the vaccine, with an estimated five additional myo/pericarditis cases per 100,000 persons vaccinated. A statistically significant increased SIR of single organ cutaneous vasculitis (SOCV) was also observed following the first dose of BNT162b2 in the 20-39 years age group only.

CONCLUSIONS

A statistically significant association between BNT162b2 vaccination and myo/pericarditis was observed. This association has been confirmed internationally. BNT162b2 was not found to be associated with the other AESIs investigated, except for SOCV following the first dose of BNT162b2 in the 20-39 years age group only, providing reassurances around the safety of the vaccine.

Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance

INTRODUCTION

Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' The extent to which serial testing in real-world studies improves or hinders overall performance in terms of sensitivity and specificity remains unknown.

METHODS

We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared type I and II errors before and after empirical calibration for historical comparator, self-controlled case series (SCCS), and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes.

RESULTS

The historical comparator design mostly had fewer type II errors than SCCS. SCCS had fewer type I errors than the historical comparator. Before empirical calibration, the serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, type I errors returned to nominal; sensitivity was lowest when the methods were combined.

CONCLUSION

While serial combination produced fewer false-positive signals compared with the most specific method, it generated more false-negative signals compared with the most sensitive method. Using a historical comparator design followed by an SCCS analysis yielded decreased sensitivity in evaluating safety signals relative to a one-stage SCCS approach. While the current use of serial testing in vaccine surveillance may provide a practical paradigm for signal identification and triage, single epidemiological designs should be explored as valuable approaches to detecting signals.

COVID-19 Vaccine Safety Monitoring Studies in Low- and Middle-Income Countries (LMICs)-A Systematic Review of Study Designs and Methods

BACKGROUND

Post-marketing vaccine safety surveillance aims to monitor and quantify adverse events following immunization in a population, but little is known about their implementation in low- and middle-income countries (LMICs). We aimed to synthesize methodological approaches used to assess adverse events following COVID-19 vaccination in LMICs.

METHODS

For this systematic review, we searched articles published from 1 December 2019 to 18 February 2022 in main databases, including MEDLINE and Embase. We included all peer-reviewed observational COVID-19 vaccine safety monitoring studies. We excluded randomized controlled trials and case reports. We extracted data using a standardized extraction form. Two authors assessed study quality using the modified Newcastle-Ottawa Quality Assessment Scale. All findings were summarized narratively using frequency tables and figures.

RESULTS

Our search found 4254 studies, of which 58 were eligible for analysis. Many of the studies included in this review were conducted in middle-income countries, with 26 studies (45%) in lower-middle-income and 28 (48%) in upper-middle-income countries. More specifically, 14 studies were conducted in the Middle East region, 16 in South Asia, 8 in Latin America, 8 in Europe and Central Asia, and 4 in Africa. Only 3% scored 7-8 points (good quality) on the Newcastle-Ottawa Scale methodological quality assessment, while 10% got 5-6 points (medium). About 15 studies (25.9%) used a cohort study design and the rest were cross-sectional. In half of them (50%), vaccination data were gathered from the participants' self-reporting methods. Seventeen studies (29.3%) used multivariable binary logistic regression and three (5.2%) used survival analyses. Only 12 studies (20.7%) performed model diagnostics and validity checks (e.g., the goodness of fit, identification of outliers, and co-linearity).

CONCLUSIONS

Published studies on COVID-19 vaccine safety surveillance in LMICs are limited in number and the methods used do not often address potential confounders. Active surveillance of vaccines in LMICs are needed to advocate vaccination programs. Implementing training programs in pharmacoepidemiology in LMICs is essential.

Comparative risk of thrombosis with thrombocytopenia syndrome or thromboembolic events associated with different covid-19 vaccines: international network cohort study from five European countries and the US

OBJECTIVE

To quantify the comparative risk of thrombosis with thrombocytopenia syndrome or thromboembolic events associated with use of adenovirus based covid-19 vaccines versus mRNA based covid-19 vaccines.

DESIGN

International network cohort study.

SETTING

Routinely collected health data from contributing datasets in France, Germany, the Netherlands, Spain, the UK, and the US.

PARTICIPANTS

Adults (age ≥18 years) registered at any contributing database and who received at least one dose of a covid-19 vaccine (ChAdOx1-S (Oxford-AstraZeneca), BNT162b2 (Pfizer-BioNTech), mRNA-1273 (Moderna), or Ad26.COV2.S (Janssen/Johnson & Johnson)), from December 2020 to mid-2021.

MAIN OUTCOME MEASURES

Thrombosis with thrombocytopenia syndrome or venous or arterial thromboembolic events within the 28 days after covid-19 vaccination. Incidence rate ratios were estimated after propensity scores matching and were calibrated using negative control outcomes. Estimates specific to the database were pooled by use of random effects meta-analyses.

RESULTS

Overall, 1 332 719 of 3 829 822 first dose ChAdOx1-S recipients were matched to 2 124 339 of 2 149 679 BNT162b2 recipients from Germany and the UK. Additionally, 762 517 of 772 678 people receiving Ad26.COV2.S were matched to 2 851 976 of 7 606 693 receiving BNT162b2 in Germany, Spain, and the US. All 628 164 Ad26.COV2.S recipients from the US were matched to 2 230 157 of 3 923 371 mRNA-1273 recipients. A total of 862 thrombocytopenia events were observed in the matched first dose ChAdOx1-S recipients from Germany and the UK, and 520 events after a first dose of BNT162b2. Comparing ChAdOx1-S with a first dose of BNT162b2 revealed an increased risk of thrombocytopenia (pooled calibrated incidence rate ratio 1.33 (95% confidence interval 1.18 to 1.50) and calibrated incidence rate difference of 1.18 (0.57 to 1.8) per 1000 person years). Additionally, a pooled calibrated incidence rate ratio of 2.26 (0.93 to 5.52) for venous thrombosis with thrombocytopenia syndrome was seen with Ad26.COV2.S compared with BNT162b2.

CONCLUSIONS

In this multinational study, a pooled 30% increased risk of thrombocytopenia after a first dose of the ChAdOx1-S vaccine was observed, as was a trend towards an increased risk of venous thrombosis with thrombocytopenia syndrome after Ad26.COV2.S compared with BNT162b2. Although rare, the observed risks after adenovirus based vaccines should be considered when planning further immunisation campaigns and future vaccine development.

Report of Adverse Effects Following Population-Wide COVID-19 Vaccination: A Comparative Study between Six Different Vaccines in Baja-California, Mexico

After emergency authorization, different COVID-19 vaccines were administered across Mexico in 2021, including mRNA, viral vector, and inactivated platform vaccines. In the state of Baja-California, 3,516,394 doses were administered, and 2285 adverse events (AE) were registered in the epidemiological surveillance system in 2021. Incidence rates per 100,000 doses were calculated for total, mild (local and systemic), and severe AE for each vaccine. Symptoms were compared between mRNA and viral vector/inactivated virus vaccines. The overall incidence rate for all AE was 64.98 per 100,000 administered doses; 79.05 AE per 100,000 doses for mRNA vaccines; and 56.9 AE per 100,000 doses for viral vector/inactivated virus vaccine platforms. AE were at least five times higher in recipients of the AstraZeneca vaccine from the Serum Institute of India (AZ from SII). Local injection site symptoms were more common in mRNA vaccines while systemic were more prevalent in viral vector/inactivated virus vaccines. Severe AE rates were similar across all administered vaccines (0.72–1.61 AE per 100,000 doses), except for AZ from SII, which documented 12.6 AE per 100,000 doses. Among 32 hospitalized severe cases, 28 (87.5%) were discharged. Guillain–Barré Syndrome was the most common serious AE reported (n = 7). Adverse events rates differed among vaccine manufacturers but were consistent with clinical trials and population-based reports in the literature.

A Case-Based Monitoring Approach to Evaluate Safety of COVID-19 Vaccines in a Partially Integrated Health Information System: A Study Protocol

In response to Coronavirus disease 2019 (COVID-19) global pandemic, various COVID-19 vaccines were rapidly administered under emergency use authorization. Rare outcomes associated with COVID-19 vaccines might be less likely to be captured in clinical trials, leading to a knowledge gap in real-world vaccine safety. In contrast with high-income countries, many low-to-middle income countries have limited capacity to conduct active surveillance, owing to the absence of large and fully-integrated health information databases. This paper describes the study protocol, which aims to investigate risk of prespecified adverse events of special interests following COVID-19 vaccination in a partially integrated health information system with non-shareable electronic health records. The SAFECOVAC study is a longitudinal, observational retrospective study of active safety surveillance using case-based monitoring approach. This involves linkage of several administrative databases and hospitalization data monitoring to identify adverse events of special interests following administration of COVID-19 vaccines in Malaysia. The source population comprises of all individuals who received at least one dose of COVID-19 vaccine. Self-controlled design and vaccinated case-coverage design will be employed to assess risk of adverse events of special interests and determine the association with vaccine exposure. Data on vaccination records will be obtained from the national COVID-19 vaccination register to identify the vaccination platforms, doses and the timing of vaccinations. The outcome of this study is hospitalization for the adverse events of special interests between March 2021 and June 2022. The outcomes will be obtained through linkage with hospital admission database and national pharmacovigilance database. Findings will provide analysis of real-world data which can inform deliberations by government and public health decision makers relative to the refinement of COVID-19 vaccination recommendations.

Vaccine Safety Surveillance Using Routinely Collected Healthcare Data-An Empirical Evaluation of Epidemiological Designs

Background: Routinely collected healthcare data such as administrative claims and electronic health records (EHR) can complement clinical trials and spontaneous reports to detect previously unknown risks of vaccines, but uncertainty remains about the behavior of alternative epidemiologic designs to detect and declare a true risk early. Methods: Using three claims and one EHR database, we evaluate several variants of the case-control, comparative cohort, historical comparator, and self-controlled designs against historical vaccinations using real negative control outcomes (outcomes with no evidence to suggest that they could be caused by the vaccines) and simulated positive control outcomes. Results: Most methods show large type 1 error, often identifying false positive signals. The cohort method appears either positively or negatively biased, depending on the choice of comparator index date. Empirical calibration using effect-size estimates for negative control outcomes can bring type 1 error closer to nominal, often at the cost of increasing type 2 error. After calibration, the self-controlled case series (SCCS) design most rapidly detects small true effect sizes, while the historical comparator performs well for strong effects. Conclusion: When applying any method for vaccine safety surveillance we recommend considering the potential for systematic error, especially due to confounding, which for many designs appears to be substantial. Adjusting for age and sex alone is likely not sufficient to address differences between vaccinated and unvaccinated, and for the cohort method the choice of index date is important for the comparability of the groups. Analysis of negative control outcomes allows both quantification of the systematic error and, if desired, subsequent empirical calibration to restore type 1 error to its nominal value. In order to detect weaker signals, one may have to accept a higher type 1 error.