A systematic review of quantitative bias analysis applied to epidemiological research

The economic rationale for cell-based influenza vaccines in children and adults: A review of cost-effectiveness analyses

Seasonal influenza significantly affects both health and economic costs in children and adults. This narrative review summarizes published cost-effectiveness analyses (CEAs) of cell-based influenza vaccines in children and adults <65 years of age, critically assesses the assumptions and approaches used in these analyses, and considers the role of cell-based influenza vaccines for children and adults. CEAs from multiple countries demonstrated the cost-effectiveness of cell-based quadrivalent influenza vaccines (QIVc) compared with egg-based trivalent/quadrivalent influenza vaccines (TIVe/QIVe). CEA findings were consistent across models relying on different relative vaccine effectiveness (rVE) estimate inputs, with the rVE of QIVc versus QIVe ranging from 8.1% to 36.2% in favor of QIVc. Across multiple scenarios and types of analyses, QIVc was consistently cost-effective compared with QIVe, including in children and adults across different regions of the world.

The Methodological Quality of Observational Studies Examining the Risk of Pregnancy Drug Use on Congenital Malformations Needs Substantial Improvement: A Cross-Sectional Survey

BACKGROUND AND OBJECTIVE

An increasing number of observational studies have investigated the risk of using drugs during pregnancy on congenital malformations. However, the credibility of the causal relationships drawn from these studies remains uncertain. This study aims to evaluate the potential methodological issues in existing observational studies.

METHODS

We used a stepwise approach to investigate this issue. First, we identified observational studies published in 2020 that examined the risk of congenital malformations associated with medication use during pregnancy. We assessed the methodological characteristics for establishing causality, including study design, confounding control, and sensitivity analysis, and compared them between "core clinical journals" and "general journals." For studies reporting an increased risk of congenital malformations in core clinical journals, we searched for subsequent studies addressing the same research question published between January 2021 and May 2023 to assess the consistency of the literature.

RESULTS

A total of 40 eligible studies were published in 2020, primarily focused on the safety of vitamin B12 and folic acid (n = 4), antidepressants (n = 4), and others (n = 32). Our findings suggest that only two (5.00%) studies used causal models to guide the identification of confounding, and only eight (20.00%) studies assessed the potential dose-response relationship. In all, 15 (37.50%) studies used propensity score analysis strategy to achieve "mimic-randomization." In addition, 22 studies (55.00%) performed sensitivity analyses, while 10 (45.45%) showed inconsistency with the primary outcome. Furthermore, 5 studies reported positive outcomes, whereas only 1 out of 11 studies demonstrated a positive correlation between drug usage during pregnancy and major malformations in subsequent studies.

CONCLUSION

A significant portion of the studies has failed to sufficiently consider the essential methodological characteristics required to improve the credibility of causal inferences. The increased risk of congenital malformations documented in core clinical journal was not adequately replicated in subsequent studies.

Probabilistic bias analysis for exposure misclassification of household income by neighbourhood in a cohort of individuals with colorectal cancer

INTRODUCTION

Despite poor agreement, neighbourhood income is used as a proxy for household income, due to a lack of data availability. We quantified misclassification between household and neighbourhood income and demonstrate quantitative bias analysis (QBA) in scenarios where only neighbourhood income is available in assessing income inequalities on colorectal cancer mortality.

METHODS

This was a retrospective study of adults with colorectal cancer diagnosed 2006-14 from Statistics Canada's Canadian Census Health and Environment Cohort. Neighbourhood income quintiles from Statistics Canada were used. Census household income quintiles were used to determine bias parameters and confirm results of the QBA. We calculated positive and negative predictive values using multinomial models, adjusting for age, sex and rural residence. Probabilistic QBA was conducted to explore the implication of exposure misclassification when estimating the effect of income on 5-year mortality.

RESULTS

We found poor agreement between neighbourhood and household income: positive predictive values ranged from 21% to 37%. The bias-adjusted risk of neighbourhood income on 5-year mortality was similar to the risk of mortality by household income. The bias-adjusted relative risk of the lowest income quintile compared with the highest was 1.42 [95% simulation interval (SI) 1.32-1.53] compared with 1.46 [95% confidence interval (CI) 1.39-1.54] for household income and 1.18 (95% CI 1.12-1.24) for neighbourhood income.

CONCLUSION

QBA can be used to estimate adjusted effects of neighbourhood income on mortality which represent household income. The predictive values from our study can be applied to similar cohorts with only neighbourhood income to estimate the effects of household income on cancer mortality.

Evaluation of quantitative bias analysis in epidemiological research: A systematic review from 2010 to mid-2023

OBJECTIVE

We aimed to demonstrate the use of quantitative bias analysis (QBA), which reveals the effects of systematic error, including confounding, misclassification and selection bias, on study results in epidemiological studies published in the period from 2010 to mid-23.

METHOD

The articles identified through a keyword search using Pubmed and Scopus were included in the study. The articles obtained from this search were eliminated according to the exclusion criteria, and the articles in which QBA analysis was applied were included in the detailed evaluation.

RESULTS

It can be said that the application of QBA analysis has gradually increased over the 13-year period. Accordingly, the number of articles in which simple is used as a method in QBA analysis is 9 (9.89%), the number of articles in which the multidimensional approach is used is 10 (10.99%), the number of articles in which the probabilistic approach is used is 60 (65.93%) and the number of articles in which the method is not specified is 12 (13.19%). The number of articles with misclassification bias model is 44 (48.35%), the number of articles with uncontrolled confounder(s) bias model is 32 (35.16%), the number of articles with selection bias model is 7 (7.69%) and the number of articles using more than one bias model is 8 (8.79%). Of the 49 (53.85%) articles in which the bias parameter source was specified, 19 (38.78%) used internal validation, 26 (53.06%) used external validation and 4 (8.16%) used educated guess, data constraints and hypothetical data. Probabilistic approach was used as a bias method in 60 (65.93%) of the articles, and mostly beta (8 [13.33%)], normal (9 [15.00%]) and uniform (8 [13.33%]) distributions were selected.

CONCLUSION

The application of QBA is rare in the literature but is increasing over time. Future researchers should include detailed analyzes such as QBA analysis to obtain inferences with higher evidence value, taking into account systematic errors.

Quantitative bias analysis methods for summary-level epidemiologic data in the peer-reviewed literature: a systematic review

OBJECTIVES

Quantitative bias analysis (QBA) methods evaluate the impact of biases arising from systematic errors on observational study results. This systematic review aimed to summarize the range and characteristics of QBA methods for summary-level data published in the peer-reviewed literature.

STUDY DESIGN AND SETTING

We searched MEDLINE, Embase, Scopus, and Web of Science for English-language articles describing QBA methods. For each QBA method, we recorded key characteristics, including applicable study designs, bias(es) addressed; bias parameters, and publicly available software. The study protocol was preregistered on the Open Science Framework (https://osf.io/ue6vm/).

RESULTS

Our search identified 10,249 records, of which 53 were articles describing 57 QBA methods for summary-level data. Of the 57 QBA methods, 53 (93%) were explicitly designed for observational studies, and 4 (7%) for meta-analyses. There were 29 (51%) QBA methods that addressed unmeasured confounding, 19 (33%) misclassification bias, 6 (11%) selection bias, and 3 (5%) multiple biases. Thirty-eight (67%) QBA methods were designed to generate bias-adjusted effect estimates and 18 (32%) were designed to describe how bias could explain away observed findings. Twenty-two (39%) articles provided code or online tools to implement the QBA methods.

CONCLUSION

In this systematic review, we identified a total of 57 QBA methods for summary-level epidemiologic data published in the peer-reviewed literature. Future investigators can use this systematic review to identify different QBA methods for summary-level epidemiologic data.

PLAIN LANGUAGE SUMMARY

Quantitative bias analysis (QBA) methods can be used to evaluate the impact of biases on observational study results. However, little is known about the full range and characteristics of available methods in the peer-reviewed literature that can be used to conduct QBA using information reported in manuscripts and other publicly available sources without requiring the raw data from a study. In this systematic review, we identified 57 QBA methods for summary-level data from observational studies. Overall, there were 29 methods that addressed unmeasured confounding, 19 that addressed misclassification bias, six that addressed selection bias, and three that addressed multiple biases. This systematic review may help future investigators identify different QBA methods for summary-level data.

Bias analyses to investigate the impact of differential participation: Application to a birth defects case-control study

BACKGROUND

Certain associations observed in the National Birth Defects Prevention Study (NBDPS) contrasted with other research or were from areas with mixed findings, including no decrease in odds of spina bifida with periconceptional folic acid supplementation, moderately increased cleft palate odds with ondansetron use and reduced hypospadias odds with maternal smoking.

OBJECTIVES

To investigate the plausibility and extent of differential participation to produce effect estimates observed in NBDPS.

METHODS

We searched the literature for factors related to these exposures and participation and conducted deterministic quantitative bias analyses. We estimated case-control participation and expected exposure prevalence based on internal and external reports, respectively. For the folic acid-spina bifida and ondansetron-cleft palate analyses, we hypothesized the true odds ratio (OR) based on prior studies and quantified the degree of exposure over- (or under-) representation to produce the crude OR (cOR) in NBDPS. For the smoking-hypospadias analysis, we estimated the extent of selection bias needed to nullify the association as well as the maximum potential harmful OR.

RESULTS

Under our assumptions (participation, exposure prevalence, true OR), there was overrepresentation of folic acid use and underrepresentation of ondansetron use and smoking among participants. Folic acid-exposed spina bifida cases would need to have been ≥1.2× more likely to participate than exposed controls to yield the observed null cOR. Ondansetron-exposed cleft palate cases would need to have been 1.6× more likely to participate than exposed controls if the true OR is null. Smoking-exposed hypospadias cases would need to have been ≥1.2 times less likely to participate than exposed controls for the association to falsely appear protective (upper bound of selection bias adjusted smoking-hypospadias OR = 2.02).

CONCLUSIONS

Differential participation could partly explain certain associations observed in NBDPS, but questions remain about why. Potential impacts of other systematic errors (e.g. exposure misclassification) could be informed by additional research.

Estimating the impact of bias in causal epidemiological studies: the case of health outcomes following assisted reproduction

Researchers interested in causal questions must deal with two sources of error: random error (random deviation from the true mean value of a distribution), and bias (systematic deviance from the true mean value due to extraneous factors). For some causal questions, randomization is not feasible, and observational studies are necessary. Bias poses a substantial threat to the validity of observational research and can have important consequences for health policy developed from the findings. The current piece describes bias and its sources, outlines proposed methods to estimate its impacts in an observational study, and demonstrates how these methods may be used to inform debate on the causal relationship between medically assisted reproduction (MAR) and health outcomes, using cancer as an example. In doing so, we aim to enlighten researchers who work with observational data, especially regarding the health effects of MAR and infertility, on the pitfalls of bias, and how to address them. We hope that, in combination with the provided example, we can convince readers that estimating the impact of bias in causal epidemiologic research is not only important but necessary to inform the development of robust health policy and clinical practice recommendations.

Investigation of the structure and magnitude of time-varying uncontrolled confounding in simulated cohort data analyzed using g-computation

BACKGROUND

When estimating the effect of time-varying exposures on longer-term outcomes, the assumption of conditional exchangeability or no uncontrolled confounding extends beyond baseline confounding to include time-varying confounding. We illustrate the structures and magnitude of uncontrolled time-varying confounding in exposure effect estimates obtained from g-computation when sequential conditional exchangeability is violated.

METHODS

We used directed acyclic graphs (DAGs) to depict time-varying uncontrolled confounding. We performed simulations and used g-computation to quantify the effects of each time-varying exposure for each DAG type. Models adjusting all time-varying confounders were considered the true (bias-adjusted) estimate. The exclusion of time-varying uncontrolled confounders represented the biased effect estimate and an unmet 'no uncontrolled confounding' assumption. True and biased estimates were compared across DAGs, with different magnitudes of uncontrolled confounding.

RESULTS

Time-varying uncontrolled confounding can present in several scenarios, including relationships into subsequently measured exposure(s), outcome, unmeasured confounder(s) and other measured confounder(s). In simulations, effect estimates obtained from g-computation were more biased in DAGs when the uncontrolled confounders were directly related to the outcome. Complex DAGs that included relationships between uncontrolled confounders and other variables and relationships where exposures caused uncontrolled confounders at the next time point resulted in the most biased effect estimates. In these complex DAGs, excluding uncontrolled confounders affected the multiple effect estimates.

CONCLUSIONS

Time-varying uncontrolled confounding has the potential to substantially impact observed effect estimates. Given the importance of longitudinal studies in advising public health, the impact of time-varying uncontrolled confounding warrants more recognition and evaluation using quantitative bias analysis.

Quantitative Assessment of Systematic Bias: A Guide for Researchers

Observational research provides valuable opportunities to advance oral health science but is limited by vulnerabilities to systematic bias, including unmeasured confounding, errors in variable measurement, or bias in the creation of study populations and/or analytic samples. The potential influence of systematic biases on observed results is often only briefly mentioned among the discussion of limitations of a given study, despite existing methods that support detailed assessments of their potential effects. Quantitative bias analysis is a set of methodological techniques that, when applied to observational data, can provide important context to aid in the interpretation and integration of observational research findings into the broader body of oral health research. Specifically, these methods were developed to provide quantitative estimates of the potential magnitude and direction of the influence of systematic biases on observed results. We aim to encourage and facilitate the broad adoption of quantitative bias analyses into observational oral health research. To this end, we provide an overview of quantitative bias analysis techniques, including a step-by-step implementation guide. We also provide a detailed appendix that guides readers through an applied example using real data obtained from a prospective observational cohort study of preconception periodontitis in relation to time to pregnancy. Quantitative bias analysis methods are available to all investigators. When appropriately applied to observational studies, findings from such studies can have a greater impact in the broader research context.

Performance of French medico-administrative databases in epidemiology of infectious diseases: a scoping review

The development of medico-administrative databases over the last few decades has led to an evolution and to a significant production of epidemiological studies on infectious diseases based on retrospective medical data and consumption of care. This new form of epidemiological research faces numerous methodological challenges, among which the assessment of the validity of targeting algorithm. We conducted a scoping review of studies that undertook an estimation of the completeness and validity of French medico-administrative databases for infectious disease epidemiological research. Nineteen validation studies and nine capture-recapture studies were identified. These studies covered 20 infectious diseases and were mostly based on the evaluation of hospital claimed data. The evaluation of their methodological qualities highlighted the difficulties associated with these types of research, particularly those linked to the assessment of their underlying hypotheses. We recall several recommendations relating to the problems addressed, which should contribute to the quality of future evaluation studies based on medico-administrative data and consequently to the quality of the epidemiological indicators produced from these information systems.

Quantitative bias analysis of prevalence under misclassification: evaluation indicators, calculation method and case analysis

Prevalence estimates are fundamental to epidemiological studies. Although they are highly vulnerable to misclassification bias, the risk of bias assessment of prevalence estimates is often neglected. Quantitative bias analysis (QBA) can effectively estimate misclassification bias in epidemiological studies; however, relatively few applications are identified. One reason for its low usage is the lack of knowledge and tools for these methods among researchers. To expand existing evaluation methods, based on the QBA principles, three indicators are proposed. One is the relative bias that quantifies the bias direction through its signs and the bias magnitude through its quantity. The second is the critical point of positive test proportion in case of a misclassification bias that is equal to zero. The third is the bound of positive test proportion equal to adjusted prevalence at misclassification bias level α. These indicators express the magnitude, direction and uncertainty of the misclassification bias of prevalence estimates, respectively. Using these indicators, it was found that slight oscillations of the positive test proportion within a certain range can lead to substantial increases in the misclassification bias. Hence, researchers should account for misclassification error analytically when interpreting the significance of adjusted prevalence for epidemiological decision making. This highlights the importance of applying QBA to these analyses. In this article, we have used three real-world cases to illustrate the characteristics and calculation methods of presented indicators. To facilitate application, an Excel-based calculation tool is provided.

Accounting for Misclassification and Selection Bias in Estimating Effectiveness of Self-managed Medication Abortion

BACKGROUND

Studies on the effectiveness of self-managed medication abortion may suffer from misclassification and selection bias due to self-reported outcomes and loss of follow-up. Monte Carlo sensitivity analysis can estimate self-managed abortion effectiveness accounting for these potential biases.

METHODS

We conducted a Monte Carlo sensitivity analysis based on data from the Studying Accompaniment model Feasibility and Effectiveness Study (the SAFE Study), to generate bias-adjusted estimates of the effectiveness of self-managed abortion with accompaniment group support. Between July 2019 and April 2020, we enrolled a total of 1051 callers who contacted accompaniment groups in Argentina and Nigeria for self-managed abortion information; 961 took abortion medications and completed at least one follow-up. Using these data, we calculated measures of effectiveness adjusted for ineligibility, misclassification, and selection bias across 50,000 simulations with bias parameters drawn from pre-specified Beta distributions in R.

RESULTS

After accounting for the potential influence of various sources of bias, bias-adjusted estimates of effectiveness were similar to observed estimates, conditional on chosen bias parameters: 92.68% (95% simulation interval: 87.80%, 95.74%) for mifepristone in combination with misoprostol (versus 93.7% in the observed data) and 98.47% (95% simulation interval: 96.79%, 99.39%) for misoprostol alone (versus 99.3% in the observed data).

CONCLUSIONS

After adjustment for multiple potential sources of bias, estimates of self-managed medication abortion effectiveness remain high. Monte Carlo sensitivity analysis may be useful in studies measuring an epidemiologic proportion (i.e., effectiveness, prevalence, cumulative incidence) while accounting for possible selection or misclassification bias.

Causal Modeling to Mitigate Selection Bias and Unmeasured Confounding in Internet-Based Epidemiology of COVID-19: Model Development and Validation

BACKGROUND

Selection bias and unmeasured confounding are fundamental problems in epidemiology that threaten study internal and external validity. These phenomena are particularly dangerous in internet-based public health surveillance, where traditional mitigation and adjustment methods are inapplicable, unavailable, or out of date. Recent theoretical advances in causal modeling can mitigate these threats, but these innovations have not been widely deployed in the epidemiological community.

OBJECTIVE

The purpose of our paper is to demonstrate the practical utility of causal modeling to both detect unmeasured confounding and selection bias and guide model selection to minimize bias. We implemented this approach in an applied epidemiological study of the COVID-19 cumulative infection rate in the New York City (NYC) spring 2020 epidemic.

METHODS

We collected primary data from Qualtrics surveys of Amazon Mechanical Turk (MTurk) crowd workers residing in New Jersey and New York State across 2 sampling periods: April 11-14 and May 8-11, 2020. The surveys queried the subjects on household health status and demographic characteristics. We constructed a set of possible causal models of household infection and survey selection mechanisms and ranked them by compatibility with the collected survey data. The most compatible causal model was then used to estimate the cumulative infection rate in each survey period.

RESULTS

There were 527 and 513 responses collected for the 2 periods, respectively. Response demographics were highly skewed toward a younger age in both survey periods. Despite the extremely strong relationship between age and COVID-19 symptoms, we recovered minimally biased estimates of the cumulative infection rate using only primary data and the most compatible causal model, with a relative bias of +3.8% and -1.9% from the reported cumulative infection rate for the first and second survey periods, respectively.

CONCLUSIONS

We successfully recovered accurate estimates of the cumulative infection rate from an internet-based crowdsourced sample despite considerable selection bias and unmeasured confounding in the primary data. This implementation demonstrates how simple applications of structural causal modeling can be effectively used to determine falsifiable model conditions, detect selection bias and confounding factors, and minimize estimate bias through model selection in a novel epidemiological context. As the disease and social dynamics of COVID-19 continue to evolve, public health surveillance protocols must continue to adapt; the emergence of Omicron variants and shift to at-home testing as recent challenges. Rigorous and transparent methods to develop, deploy, and diagnosis adapted surveillance protocols will be critical to their success.

Quantifying the influence of bias in reproductive and perinatal epidemiology through simulation

PURPOSE

The application of simulated data in epidemiological studies enables the illustration and quantification of the magnitude of various types of bias commonly found in observational studies. This was a review of the application of simulation methods to the quantification of bias in reproductive and perinatal epidemiology and an assessment of value gained.

METHODS

A search of published studies available in English was conducted in August 2020 using PubMed, Medline, Embase, CINAHL, and Scopus. A gray literature search of Google and Google Scholar, and a hand search using the reference lists of included studies was undertaken.

RESULTS

Thirty-nine papers were included in this study, covering information (n = 14), selection (n = 14), confounding (n = 9), protection (n = 1), and attenuation bias (n = 1). The methods of simulating data and reporting of results varied, with more recent studies including causal diagrams. Few studies included code for replication.

CONCLUSIONS

Although there has been an increasing application of simulation in reproductive and perinatal epidemiology since 2015, overall this remains an underexplored area. Further efforts are required to increase knowledge of how the application of simulation can quantify the influence of bias, including improved design, analysis and reporting. This will improve causal interpretation in reproductive and perinatal studies.