Regression Discontinuity Design: Simulation and Application in Two Cardiovascular Trials with Continuous Outcomes

The challenge of dementia prevention trials and the role of quasi-experimental studies

Observational studies have shown consistently that modifiable risk factors during life are associated with increased dementia risk in old age but randomized controlled trials (RCTs) on dementia prevention evaluating the treatment of these risk factors did not find consistent effects on cognitive outcomes. The discrepancy in findings is potentially attributable to inherent differences between the two study designs. Although RCTs are the gold standard for establishing causality, designing and conducting an RCT for dementia prevention is complex. Quasi-experimental studies (QESs) may contribute to investigating causality without randomization. QESs use variation in exposure to a risk factor or intervention in an observational setting to deduct causal effects. Design-specific approaches are used to control for confounding, the main caveat of QESs. In this article we address the challenges, opportunities, and limitations of QESs for research into dementia prevention. HIGHLIGHTS: Despite consistent associations between modifiable risk factors and dementia, the mostly neutral effects of randomized controlled trials (RCTs) challenge the causality of these associations. RCTs in the field of dementia prevention are often problematic due to ethical, practical, or financial constraints, and their results may have limited generalizability. Four quasi-experimental study (QES) designs may be suitable to study causality between risk factors and dementia; we critically appraise these study designs for dementia-prevention studies. We describe how specific QES designs can be used to study the effects of risk-factor modification for 12 known risk factors for dementia.

Dealing with confounding in observational studies: A scoping review of methods evaluated in simulation studies with single-point exposure

The aim of this article was to perform a scoping review of methods available for dealing with confounding when analyzing the effect of health care treatments with single-point exposure in observational data. We aim to provide an overview of methods and their performance assessed by simulation studies indexed in PubMed. We searched PubMed for simulation studies published until January 2021. Our search was restricted to studies evaluating binary treatments and binary and/or continuous outcomes. Information was extracted on the methods' assumptions, performance, and technical properties. Of 28,548 identified references, 127 studies were eligible for inclusion. Of them, 84 assessed 14 different methods (ie, groups of estimators that share assumptions and implementation) for dealing with measured confounding, and 43 assessed 10 different methods for dealing with unmeasured confounding. Results suggest that there are large differences in performance between methods and that the performance of a specific method is highly dependent on the estimator. Furthermore, the methods' assumptions regarding the specific data features also substantially influence the methods' performance. Finally, the methods result in different estimands (ie, target of inference), which can even vary within methods. In conclusion, when choosing a method to adjust for measured or unmeasured confounding it is important to choose the most appropriate estimand, while considering the population of interest, data structure, and whether the plausibility of the methods' required assumptions hold.

Bias? Clarifying the language barrier between epidemiologists and economists

In health intervention research, epidemiologists and economists are more and more interested in estimating causal effects based on observational data. However, collaboration and interaction between both disciplines are regularly clouded by differences in the terminology used. Amongst others, this is reflected in differences in labeling, handling, and interpreting the sources of bias in parameter estimates. For example, both epidemiologists and economists use the term selection bias. However, what economists label as selection bias is labeled as confounding by epidemiologists. This paper aims to shed light on this and other subtle differences between both fields and illustrate them with hypothetical examples. We expect that clarification of these differences will improve the multidisciplinary collaboration between epidemiologists and economists. Furthermore, we hope to empower researchers to select the most suitable analytical technique from either field for the research problem at hand.

Personalized Decision Making on Genomic Testing in Early Breast Cancer: Expanding the MINDACT Trial with Decision-Analytic Modeling

BACKGROUND

Genomic tests may improve upon clinical risk estimation with traditional prognostic factors. We aimed to explore how evidence on the prognostic strength of a genomic signature (clinical validity) can contribute to individualized decision making on starting chemotherapy for women with breast cancer (clinical utility).

METHODS

The MINDACT trial was a randomized trial that enrolled 6693 women with early-stage breast cancer. A 70-gene signature (Mammaprint) was used to estimate genomic risk, and clinical risk was estimated by a dichotomized version of the Adjuvant!Online risk calculator. Women with discordant risk results were randomized to the use of chemotherapy. We simulated the full risk distribution of these women and estimated individual benefit, assuming a constant relative effect of chemotherapy.

RESULTS

The trial showed a prognostic effect of the genomic signature (adjusted hazard ratio 2.4). A decision-analytic modeling approach identified far fewer women as candidates for genetic testing (4% rather than 50%) and fewer benefiting from chemotherapy (3% rather than 27%) as compared with the MINDACT trial report. The selection of women benefitting from genetic testing and chemotherapy depended strongly on the required benefit from treatment and the assumed therapeutic effect of chemotherapy.

CONCLUSIONS

A high-quality pragmatic trial was insufficient to directly inform clinical practice on the utility of a genomic test for individual women. The indication for genomic testing may be far more limited than suggested by the MINDACT trial.

Regression Discontinuity Designs in Health: A Systematic Review

BACKGROUND

Regression discontinuity designs are non-randomized study designs that permit strong causal inference with relatively weak assumptions. Interest in these designs is growing but there is limited knowledge of the extent of their application in health. We aimed to conduct a comprehensive systematic review of the use of regression discontinuity designs in health research.

METHODS

We included studies that used regression discontinuity designs to investigate the physical or mental health outcomes of any interventions or exposures in any populations. We searched 32 health, social science, and gray literature databases (1 January 1960 to 1 January 2019). We critically appraised studies using eight criteria adapted from the What Works Clearinghouse Standards for regression discontinuity designs. We conducted a narrative synthesis, analyzing the forcing variables and threshold rules used in each study.

RESULTS

The literature search retrieved 7658 records, producing 325 studies that met the inclusion criteria. A broad range of health topics was represented. The forcing variables used to implement the design were age, socioeconomic measures, date or time of exposure or implementation, environmental measures such as air quality, geographic location, and clinical measures that act as a threshold for treatment. Twelve percent of the studies fully met the eight quality appraisal criteria. Fifteen percent of studies reported a prespecified primary outcome or study protocol.

CONCLUSIONS

This systematic review demonstrates that regression discontinuity designs have been widely applied in health research and could be used more widely still. Shortcomings in study quality and reporting suggest that the potential benefits of this method have not yet been fully realized.

Influence of the Fukushima Daiichi Nuclear Power Plant Accident on the Use of Computed Tomography in Children With Mild Head Injuries

Background

Computed tomography (CT) is commonly used in children with mild head injuries. People in Japan are concerned about radiation exposure and radiation-induced cancer because of the Fukushima Daiichi Nuclear Power Plant accident on March 11, 2011. This study investigated whether the accident influenced the use of CT in children with mild head injuries.

Methods

Using the Japan Medical Data Center database, we identified patients aged ≤15 years visiting hospitals because of mild head injuries from January 1, 2008, to December 31, 2013. We excluded patients who were admitted to the hospital or received other medical examinations. Regression discontinuity analysis was used to compare proportions of patients undergoing head CT and having clinically important traumatic brain injury (ciTBI) overlooked before versus after the accident, adjusting for patient characteristics, secular trends, and hospital effect.

Results

Eligible patients (n = 40,440) were classified as visiting the hospital before (n = 11,659) or after (n = 28,781) the accident. The regression discontinuity analysis showed that the accident was associated with a reduction in the proportion of patients undergoing head CT (odds ratio [OR] 0.73; 95% confidence interval [CI], 0.63–0.86), whereas the accident was not associated with an increase in cases where ciTBI was overlooked (OR 0.72; 95% CI, 0.13–4.00).

Conclusions

The use of CT in children with mild head injuries declined after the Fukushima Daiichi Nuclear Power Plant accident. Improving awareness of radiation exposure risks among patients and physicians could reduce unnecessary CT.

Probing the Effective Treatment Thresholds for Alteplase in Acute Ischemic Stroke With Regression Discontinuity Designs

Randomized Controlled Trials (RCTs) are considered the gold standard for measuring the efficacy of medical interventions. However, RCTs are expensive, and use a limited population. Techniques to estimate the effects of stroke interventions from observational data that minimize confounding would be useful. We used regression discontinuity design (RDD), a technique well-established in economics, on the Get With The Guidelines-Stroke (GWTG-Stroke) data set. RDD, based on regression, measures the occurrence of a discontinuity in an outcome (e.g., odds of home discharge) as a function of an intervention (e.g., alteplase) that becomes significantly more likely when crossing the threshold of a continuous variable that determines that intervention (e.g., time from symptom onset, since alteplase is only given if symptom onset is less than e.g., 3 h). The technique assumes that patients near either side of a threshold (e.g., 2.99 and 3.01 h from symptom onset) are indistinguishable other than the use of the treatment. We compared outcomes of patients whose estimated onset to treatment time fell on either side of the treatment threshold for three cohorts of patients in the GWTG-Stroke data set. This data set spanned three different treatment thresholds for alteplase (3 h, 2003-2007, N = 1,869; 3 h, 2009-2016, N = 13,086, and 4.5 h, 2009-2016, N = 6,550). Patient demographic characteristics were overall similar across the treatment thresholds. We did not find evidence of a discontinuity in clinical outcome at any treatment threshold attributable to alteplase. Potential reasons for failing to find an effect include violation of some RDD assumptions in clinical care, large sample sizes required, or already-well-chosen treatment threshold.

Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research

Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.

The regression discontinuity design showed to be a valid alternative to a randomized controlled trial for estimating treatment effects

OBJECTIVES

To compare treatment effect estimates obtained from a regression discontinuity (RD) design with results from an actual randomized controlled trial (RCT).

STUDY DESIGN AND SETTING

Data from an RCT (EVIDENT), which studied the effect of an Internet intervention on depressive symptoms measured with the Patient Health Questionnaire (PHQ-9), were used to perform an RD analysis, in which treatment allocation was determined by a cutoff value at baseline (PHQ-9 = 10). A linear regression model was fitted to the data, selecting participants above the cutoff who had received the intervention (n = 317) and control participants below the cutoff (n = 187). Outcome was PHQ-9 sum score 12 weeks after baseline. Robustness of the effect estimate was studied; the estimate was compared with the RCT treatment effect.

RESULTS

The final regression model showed a regression coefficient of -2.29 [95% confidence interval (CI): -3.72 to -.85] compared with a treatment effect found in the RCT of -1.57 (95% CI: -2.07 to -1.07).

CONCLUSION

Although the estimates obtained from two designs are not equal, their confidence intervals overlap, suggesting that an RD design can be a valid alternative for RCTs. This finding is particularly important for situations where an RCT may not be feasible or ethical as is often the case in clinical research settings.