A comparison of the test-negative and traditional case-control study designs with respect to the bias of estimates of rotavirus vaccine effectiveness

Effectiveness and impact of monovalent rotavirus vaccination in Afghanistan: a test-negative case-control analysis

BACKGROUND

Afghanistan introduced monovalent rotavirus vaccine (Rotarix) into its national immunisation schedule in January, 2018. While post-licensure studies have shown substantial declines in rotavirus gastroenteritis cases and deaths globally, there is little evidence of rotavirus vaccine effectiveness and impact from low-income countries in Asia. We aimed to evaluate the effectiveness of the Rotarix vaccine and the impact of Rotarix vaccine on rotavirus gastroenteritis hospitalisations (ie, hospital admissions) among children younger than 5 years in Afghanistan.

METHODS

We used a test-negative case-control design embedded in an active sentinel surveillance platform to evaluate vaccine effectiveness. Children born on or after Jan 1, 2018, who had documentation of their rotavirus vaccination status and who were admitted for acute gastroenteritis at one of four sentinel hospitals from May, 2018 to December, 2021 were eligible to be included. We used an unconditional logistic regression model to estimate vaccine effectiveness and 95% CIs for a complete series of doses compared with no rotavirus vaccine doses among patients admitted with acute gastroenteritis. Vaccine effectiveness against hospitalisation was calculated as (1 - [odds of being vaccinated in cases] / [odds of being vaccinated in controls]) × 100%. We compared pre-vaccine (2013-15) and post-vaccine (2019-21) surveillance data from two sites to calculate vaccine impact.

FINDINGS

The vaccine effectiveness analysis included 1172 cases and 2173 controls. Approximately 2108 (63·0%) of 3345 cases and controls were male, 1237 (37·0%) were female, and 2171 (65·0%) were aged 6-11 months. Two doses of Rotarix were 45% (95% CI 22-62) effective against rotavirus hospitalisation in children aged 6-59 months, adjusting for age, severity, admission year, and rotavirus season. Rotavirus positivity decreased from 51% pre-vaccine to 39% post-vaccine, resulting in a 39% adjusted reduction in rotavirus positivity among children younger than 5 years admitted with acute gastroenteritis.

INTERPRETATION

Rotarix showed moderate effectiveness in preventing rotavirus gastroenteritis hospitalisations, consistent with findings in other low-income countries. These findings support the continued administration of the rotavirus vaccine in Afghanistan.

FUNDING

Gavi, the Vaccine Alliance.

TRANSLATION

For the Dari translation of the abstract see Supplementary Materials section.

Vaccine effectiveness in reducing COVID-19-related hospitalization after a risk-age-based mass vaccination program in a Chilean municipality: A comparison of observational study designs

BACKGROUND

Case-control studies involving test-negative (TN) and syndrome-negative (SN) controls are reliable for evaluating influenza and rotavirus vaccine effectiveness (VE) during a random vaccination process. However, there is no empirical evidence regarding the impact in real-world mass vaccination campaigns against SARS-CoV-2 using TN and SN controls.

OBJECTIVE

To compare in the same population the effectiveness of SARS-CoV-2 vaccination on COVID-19-related hospitalization rates across a cohort design, TN and SN designs.

METHOD

We conducted an unmatched population-based cohort, TN and SN case-control designs linking data from four data sources (public primary healthcare system, hospitalization registers, epidemiological surveillance systems and the national immunization program) in a Chilean municipality (Rancagua) between March 1, 2021 and August 31, 2021. The outcome was COVID-19-related hospitalization. To ensure sufficient sample size in the unexposed group, completion of follow-up in the cohort design, and sufficient time between vaccination and hospitalization in the case-control design, VE was estimated comparing 8-week periods for each individual.

RESULTS

Among the 191,505 individuals registered in the primary healthcare system of Rancagua in Chile on March 1, 2021; 116,453 met the cohort study's inclusion criteria. Of the 9,471 hospitalizations registered during the study period in the same place, 526 were COVID-19 cases, 108 were TN controls, and 1,628 were SN controls. For any vaccine product, the age- and sex-adjusted vaccine effectiveness comparing fully and nonvaccinated individuals was 67.2 (55.7-76.3) in the cohort design, whereas it was 67.8 (44.1-81.4) and 77.9 (70.2-83.8) in the TN and SN control designs, respectively.

CONCLUSION

The VE of a COVID-19 vaccination program based on age and risk groups tended to differ across the three observational study designs. The SN case-control design may be an efficient option for evaluating COVID-19 VE in real-world settings.

Effectiveness of Messenger RNA Vaccines against SARS-CoV-2 Infection in Hemodialysis Patients: A Case-Control Study

Patients with end-stage kidney disease (ESKD) are at increased risk for SARS-CoV-2 infection and its complications compared with the general population. Several studies evaluated the effectiveness of COVID-19 vaccines in the dialysis population but showed mixed results. The aim of this study was to determine the effectiveness of COVID-19 mRNA vaccines against confirmed SARS-CoV-2 infection in hemodialysis (HD) patients in the State of Qatar. We included all adult ESKD patients on chronic HD who had at least one SARS-CoV-2 PCR test done after the introduction of the COVID-19 mRNA vaccines on 24 December 2020. Vaccinated patients who were only tested before receiving any dose of their COVID-19 vaccine or within 14 days after receiving the first vaccine dose were excluded from the study. We used a test-negative case−control design to determine the effectiveness of the COVID-19 vaccination. Sixty-eight patients had positive SARS-CoV-2 PCR tests (cases), while 714 patients had negative tests (controls). Ninety-one percent of patients received the COVID-19 mRNA vaccine. Compared with the controls, the cases were more likely to be older (62 ± 14 vs. 57 ± 15, p = 0.02), on dialysis for more than one year (84% vs. 72%, p = 0.03), unvaccinated (46% vs. 5%, p < 0.0001), and symptomatic (54% vs. 21%, p < 0.0001). The effectiveness of receiving two doses of COVID-19 mRNA vaccines against confirmed SARS-CoV-2 infection was 94.7% (95% CI: 89.9−97.2) in our HD population. The findings of this study support the importance of using the COVID-19 mRNA vaccine in chronic HD patients to prevent SARS-CoV-2 infection in such a high-risk population.

Comparison of test-negative and syndrome-negative controls in SARS-CoV-2 vaccine effectiveness evaluations for preventing COVID-19 hospitalizations in the United States

BACKGROUND

Test-negative design (TND) studies have produced validated estimates of vaccine effectiveness (VE) for influenza vaccine studies. However, syndrome-negative controls have been proposed for differentiating bias and true estimates in VE evaluations for COVID-19. To understand the use of alternative control groups, we compared characteristics and VE estimates of syndrome-negative and test-negative VE controls.

METHODS

Adults hospitalized at 21 medical centers in 18 states March 11-August 31, 2021 were eligible for analysis. Case patients had symptomatic acute respiratory infection (ARI) and tested positive for SARS-CoV-2. Control groups were test-negative patients with ARI but negative SARS-CoV-2 testing, and syndrome-negative controls were without ARI and negative SARS-CoV-2 testing. Chi square and Wilcoxon rank sum tests were used to detect differences in baseline characteristics. VE against COVID-19 hospitalization was calculated using logistic regression comparing adjusted odds of prior mRNA vaccination between cases hospitalized with COVID-19 and each control group.

RESULTS

5811 adults (2726 cases, 1696 test-negative controls, and 1389 syndrome-negative controls) were included. Control groups differed across characteristics including age, race/ethnicity, employment, previous hospitalizations, medical conditions, and immunosuppression. However, control-group-specific VE estimates were very similar. Among immunocompetent patients aged 18-64 years, VE was 93 % (95 % CI: 90-94) using syndrome-negative controls and 91 % (95 % CI: 88-93) using test-negative controls.

CONCLUSIONS

Despite demographic and clinical differences between control groups, the use of either control group produced similar VE estimates across age groups and immunosuppression status. These findings support the use of test-negative controls and increase confidence in COVID-19 VE estimates produced by test-negative design studies.

Understanding Variation in Rotavirus Vaccine Effectiveness Estimates in the United States: The Role of Rotavirus Activity and Diagnostic Misclassification

BACKGROUND

Estimates of rotavirus vaccine effectiveness (VE) in the United States appear higher in years with more rotavirus activity. We hypothesized rotavirus VE is constant over time but appears to vary as a function of temporal variation in local rotavirus cases and/or misclassified diagnoses.

METHODS

We analyzed 6 years of data from eight US surveillance sites on 8- to 59-month olds with acute gastroenteritis symptoms. Children's stool samples were tested via enzyme immunoassay (EIA); rotavirus-positive results were confirmed with molecular testing at the US Centers for Disease Control and Prevention. We defined rotavirus gastroenteritis cases by either positive on-site EIA results alone or positive EIA with Centers for Disease Control and Prevention confirmation. For each case definition, we estimated VE against any rotavirus gastroenteritis, moderate-to-severe disease, and hospitalization using two mixed-effect regression models: the first including year plus a year-vaccination interaction, and the second including the annual percent of rotavirus-positive tests plus a percent positive-vaccination interaction. We used multiple overimputation to bias-adjust for misclassification of cases defined by positive EIA alone.

RESULTS

Estimates of annual rotavirus VE against all outcomes fluctuated temporally, particularly when we defined cases by on-site EIA alone and used a year-vaccination interaction. Use of confirmatory testing to define cases reduced, but did not eliminate, fluctuations. Temporal fluctuations in VE estimates further attenuated when we used a percent positive-vaccination interaction. Fluctuations persisted until bias-adjustment for diagnostic misclassification.

CONCLUSIONS

Both controlling for time-varying rotavirus activity and bias-adjusting for diagnostic misclassification are critical for estimating the most valid annual rotavirus VE.

Use of Recently Vaccinated Individuals to Detect Bias in Test-Negative Case-Control Studies of COVID-19 Vaccine Effectiveness

Postauthorization observational studies play a key role in understanding COVID-19 vaccine effectiveness following the demonstration of efficacy in clinical trials. Although bias due to confounding, selection bias, and misclassification can be mitigated through careful study design, unmeasured confounding is likely to remain in these observational studies. Phase III trials of COVID-19 vaccines have shown that protection from vaccination does not occur immediately, meaning that COVID-19 risk should be similar in recently vaccinated and unvaccinated individuals, in the absence of confounding or other bias. Several studies have used the estimated effectiveness among recently vaccinated individuals as a negative control exposure to detect bias in vaccine effectiveness estimates. In this paper, we introduce a theoretical framework to describe the interpretation of such a bias indicator in test-negative studies, and outline strong assumptions that would allow vaccine effectiveness among recently vaccinated individuals to serve as a negative control exposure.

A comparison of the test-negative and the matched case-control study designs for estimation of EV71 vaccine immunological surrogate endpoints from a randomized controlled trial

Since TND could be an appropriate method to assess vaccine effectiveness, we want to know whether it may be used for the estimation of vaccine immunological surrogate endpoints, like case-control study. We conducted two study designs (test-negative design (TND) VS matched case-control design (MCC)) to evaluate immunological surrogate endpoint against EV71-associated diseases. We calculated sensitivity (proportion of participants with EV71-associated disease who have a titer less than the cutoff at day 56), specificity (proportion of matched controls who have a titer equal or greater than the cutoff at day 56), and corresponding Youden index ([sensitivity + specificity] − 1). Then, we compared them between TND and MCC. In test-negative design, we totally enrolled 7029 subjects, 49 tested positive as cases and 6980 tested negative as controls in per-protocol population. In matched case-control design, we totally enrolled 305 subjects, 51 as cases, and 254 as controls in whole cohort. In sensitivity and specificity comparison, TND and MCC’s results were similar to each other, except for a titer of 1:4. Nonetheless, in Youden index comparison, MCC’s results were slightly higher than the TND’s, except for a titer of 1:4. EV71 vaccine immunological surrogate endpoints derived from TND was similar to MCC’s. Our results supported that TND could become an alternative research design with the progress of surveillance.

Potential test-negative design study bias in outbreak settings: application to Ebola vaccination in Democratic Republic of Congo

BACKGROUND

Infectious disease outbreaks present unique challenges to study designs for vaccine evaluation. Test-negative design (TND) studies have previously been used to estimate vaccine effectiveness and have been proposed for Ebola virus disease (EVD) vaccines. However, there are key differences in how cases and controls are recruited during outbreaks and pandemics of novel pathogens, whcih have implications for the reliability of effectiveness estimates using this design.

METHODS

We use a modelling approach to quantify TND bias for a prophylactic vaccine under varying study and epidemiological scenarios. Our model accounts for heterogeneity in vaccine distribution and for two potential routes to testing and recruitment into the study: self-reporting and contact-tracing. We derive conventional and hybrid TND estimators for this model and suggest ways to translate public health response data into the parameters of the model.

RESULTS

Using a conventional TND study, our model finds biases in vaccine effectiveness estimates. Bias arises due to differential recruitment from self-reporting and contact-tracing, and due to clustering of vaccination. We estimate the degree of bias when recruitment route is not available, and propose a study design to eliminate the bias if recruitment route is recorded.

CONCLUSIONS

Hybrid TND studies can resolve the design bias with conventional TND studies applied to outbreak and pandemic response testing data, if those efforts collect individuals' routes to testing. Without route to testing, other epidemiological data will be required to estimate the magnitude of potential bias in a conventional TND study. Since these studies may need to be conducted retrospectively, public health responses should obtain these data, and generic protocols for outbreak and pandemic response studies should emphasize the need to record routes to testing.

Bias correction methods for test-negative designs in the presence of misclassification

The test-negative design (TND) has become a standard approach for vaccine effectiveness (VE) studies. However, previous studies suggested that it may be more vulnerable than other designs to misclassification of disease outcome caused by imperfect diagnostic tests. This could be a particular limitation in VE studies where simple tests (e.g. rapid influenza diagnostic tests) are used for logistical convenience. To address this issue, we derived a mathematical representation of the TND with imperfect tests, then developed a bias correction framework for possible misclassification. TND studies usually include multiple covariates other than vaccine history to adjust for potential confounders; our methods can also address multivariate analyses and be easily coupled with existing estimation tools. We validated the performance of these methods using simulations of common scenarios for vaccine efficacy and were able to obtain unbiased estimates in a variety of parameter settings.

Uses of pathogen detection data to estimate vaccine direct effects in case-control studies

The fact that many pathogens can be carried or shed without causing symptoms complicates the interpretation of microbiological data when diagnosing certain infectious disease syndromes. Diagnostic criteria that attribute symptoms to a pathogen which is detectable, whether it is or is not the aetiological agent of disease, may lead to outcome misclassification in epidemiological studies. Case-control studies are commonly undertaken to estimate vaccine effectiveness (VE) and present an opportunity to compare pathogen detection among individuals with and without clinically relevant symptoms. Considering this study context, we present a mathematical framework yielding simple estimators for the direct effects of vaccination on various aspects of host susceptibility. These include protection against acquisition of the pathogen of interest and protection against progression of this pathogen to disease following acquisition. We assess the impact of test sensitivity on these estimators and extend our framework to identify a 'vaccine probe' estimator for pathogen-specific aetiological fractions. We also derive biases affecting VE estimates under the test-negative design, a special case enrolling only symptomatic persons. Our results provide strategies for estimating pathogen-specific VE in the absence of a diagnostic gold standard. These approaches can inform the design and analysis of studies addressing numerous pathogens and vaccines.

Challenges in estimating influenza vaccine effectiveness

Introduction: Influenza vaccination is regarded as the most effective way to prevent influenza infection. Due to the rapid genetic changes that influenza viruses undergo, seasonal influenza vaccines must be reformulated and re-administered annually necessitating the evaluation of influenza vaccine effectiveness (VE) each year. The estimation of influenza VE presents numerous challenges. Areas Covered: This review aims to identify, discuss, and, where possible, offer suggestions for dealing with the following challenges in estimating influenza VE: different outcomes of interest against which VE is estimated, study designs used to assess VE, sources of bias and confounding, repeat vaccination, waning immunity, population level effects of vaccination, and VE in at-risk populations. Expert Opinion: The estimation of influenza VE has improved with surveillance networks, better understanding of sources of bias and confounding, and the implementation of advanced statistical methods. Future research should focus on better estimates of the indirect effects of vaccination, the biological effects of vaccination, and how vaccines interact with the immune system. Specifically, little is known about how influenza vaccination impacts an individual's infectiousness, how vaccines wane over time, and the impact of repeated vaccination.

Effect of propensity of seeking medical care on the bias of the estimated effectiveness of rotavirus vaccines from studies using a test-negative case-control design

BACKGROUND

Rotavirus is the leading cause of severe diarrhea among children worldwide, and vaccines can reduce morbidity and mortality by 50-98%. The test-negative control (TNC) study design is increasingly used for evaluating the effectiveness of vaccines against rotavirus and other vaccine-preventable diseases. In this study design, symptomatic patients who seek medical care are tested for the pathogen of interest. Those who test positive (negative) are classified as cases (controls).

METHODS

We use a probability model to evaluate the bias of estimates of rotavirus vaccine effectiveness (VE) against rotavirus diarrhea resulting in hospitalization in the presence of possible confounding and selection biases due to differences in the propensity of seeking medical care (PSMC) between vaccinated and unvaccinated children.

RESULTS

The TNC-based VE estimate corrects for confounding bias when the confounder's effects on the probabilities of rotavirus and non-rotavirus related hospitalizations are equal. If this condition is not met, then the estimated VE may be substantially biased. The bias is more severe in low-income countries, where VE is known to be lower. Under our model, differences in PSMC between vaccinated and unvaccinated children do not result in selection bias when the TNC study design is used.

CONCLUSIONS

In practice, one can expect the association of PSMC (or other potential confounders) with the probabilities of rotavirus and non-rotavirus related hospitalization to be similar, in which case the confounding effects will only result in small bias in the VE estimate from TNC studies. The results of this work, along with those of our previous paper, confirm the TNC design can be expected to provide reliable estimates of rotavirus VE in both high- and low-income countries.

Rotavirus vaccine impact assessment surveillance in India: protocol and methods

INTRODUCTION

Rotavirus infection accounts for 39% of under-five diarrhoeal deaths globally and 22% of these deaths occur in India. Introduction of rotavirus vaccine in a national immunisation programme is considered to be the most effective intervention in preventing severe rotavirus disease. In 2016, India introduced an indigenous rotavirus vaccine (Rotavac) into the Universal Immunisation Programme in a phased manner. This paper describes the protocol for surveillance to monitor the performance of rotavirus vaccine following its introduction into the routine childhood immunisation programme.

METHODS

An active surveillance system was established to identify acute gastroenteritis cases among children less than 5 years of age. For all children enrolled at sentinel sites, case reporting forms are completed and a copy of vaccination record and a stool specimen obtained. The forms and specimens are sent to the referral laboratory for data entry, analysis, testing and storage. Data from sentinel sites in states that have introduced rotavirus vaccine into their routine immunisation schedule will be used to determine rotavirus vaccine impact and effectiveness.

ETHICS AND DISSEMINATION

The Institutional Review Board of Christian Medical College, Vellore, and all the site institutional ethics committees approved the project. Results will be disseminated in peer-reviewed journals and with stakeholders of the universal immunisation programme in India.