The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions

Impact of randomised wmel Wolbachia deployments on notified dengue cases and insecticide fogging for dengue control in Yogyakarta City

BACKGROUND

Releases of Wolbachia (wMel)-infected Aedes aegypti mosquitoes significantly reduced the incidence of virologically confirmed dengue in a previous cluster randomised trial in Yogyakarta City, Indonesia. Following the trial, wMel releases were extended to the untreated control areas, to achieve city-wide coverage of Wolbachia.

OBJECTIVE

In this predefined analysis, we evaluated the impact of the wMel deployments in Yogyakarta on dengue hemorrhagic fever (DHF) case notifications and on the frequency of perifocal insecticide spraying by public health teams.

METHODS

Monthly counts of DHF cases notified to the Yogyakarta District Health Office between January 2006 and May 2022 were modelled as a function of time-varying local wMel treatment status (fully- and partially-treated vs untreated, and by quintile of wMel prevalence). The frequency of insecticide fogging in wMel-treated and untreated areas was analysed using negative binomial regression.

RESULTS

Notified DHF incidence was 83% lower in fully treated vs untreated periods (IRR 0.17 [95% CI 0.14, 0.20]), and 78% lower in areas with 80-100% wMel prevalence compared to areas with 0-20% wMel (IRR 0.23 [0.17, 0.30]). A similar intervention effect was observed at 60-80% wMel prevalence as at 80-100% prevalence (76% vs 78% efficacy, respectively). Pre-intervention, insecticide fogging occurred at similar frequencies in areas later randomised to wMel-treated and untreated arms of the trial. After wMel deployment, fogging occurred significantly less frequently in treated areas (IRR 0.17 [0.10, 0.30]).

CONCLUSIONS

Deployments of wMel-infected Aedes aegypti mosquitoes resulted in an 83% reduction in the application of perifocal insecticide spraying, consistent with lower dengue case notifications in wMel-treated areas. These results show that the Wolbachia intervention effect demonstrated previously in a cluster randomised trial was also measurable from routine surveillance data.

The validity of causal claims with repeated measures designs: A within-study comparison evaluation of differences-in-differences and the comparative interrupted time series

Modern policies are commonly evaluated not with randomized experiments but with repeated measures designs like difference-in-differences (DID) and the comparative interrupted time series (CITS). The key benefit of these designs is that they control for unobserved confounders that are fixed over time. However, DID and CITS designs only result in unbiased impact estimates when the model assumptions are consistent with the data at hand. In this paper, we empirically test whether the assumptions of repeated measures designs are met in field settings. Using a within-study comparison design, we compare experimental estimates of the impact of patient-directed care on medical expenditures to non-experimental DID and CITS estimates for the same target population and outcome. Our data come from a multi-site experiment that includes participants receiving Medicaid in Arkansas, Florida, and New Jersey. We present summary measures of repeated measures bias across three states, four comparison groups, two model specifications, and two outcomes. We find that, on average, bias resulting from repeated measures designs are very close to zero (less than 0.01 standard deviations; SDs). Further, we find that comparison groups which have pre-treatment trends that are visibly parallel to the treatment group result in less bias than those with visibly divergent trends. However, CITS models that control for baseline trends produced slightly more bias and were less precise than DID models that only control for baseline means. Overall, we offer optimistic evidence in favor of repeated measures designs when randomization is not feasible.

Using Simulation to Analyze Interrupted Time Series Designs

We are sometimes forced to use the Interrupted Time Series (ITS) design as an identification strategy for potential policy change, such as when we only have a single treated unit and cannot obtain comparable controls. For example, with recent county- and state-wide criminal justice reform efforts, where judicial bodies have changed bail setting practices for everyone in their jurisdiction in order to reduce rates of pre-trial detention while maintaining court order and public safety, we have no natural and available comparison group other than the past. In these contexts, it is imperative to model pre-policy trends with a light touch, allowing for structures such as autoregressive departures from any pre-existing trend, in order to accurately and realistically assess the uncertainty of our projections. We aim to provide a methodological approach rooted in commonly understood and used modeling tools to achieve this. We quantify uncertainty with simulation, generating a distribution of plausible counterfactual trajectories to compare to the observed; this approach naturally allows for incorporating seasonality and other time-varying covariates, and provides confidence intervals along with point estimates for the potential impacts of policy change. We find simulation provides a natural framework to capture and show uncertainty in the ITS designs. It also allows for easy extensions such as nonparametric smoothing in order to handle multiple post-policy time points.

The comparative interrupted time series design for assessment of diagnostic impact: methodological considerations and an example using point-of-care C-reactive protein testing

BACKGROUND

In diagnostic evaluation, it is necessary to assess the clinical impact of a new diagnostic as well as its diagnostic accuracy. The comparative interrupted time series design has been proposed as a quasi-experimental approach to evaluating interventions. We show how it can be used in the design of a study to evaluate a point-of-care diagnostic test for C-reactive protein in out-of-hours primary care services, to guide antibiotic prescribing among patients presenting with possible respiratory tract infection. This study consisted of a retrospective phase that used routinely collected monthly antibiotic prescribing data from different study sites, and a prospective phase in which antibiotic prescribing rates were monitored after the C-reactive protein diagnostic was introduced at some of the sites.

METHODS

Of 8 study sites, 3 were assigned to receive the diagnostic and 5 were assigned as controls. We obtained retrospective monthly time series of respiratory tract targeted antibiotic prescriptions at each site. Separate ARIMA models at each site were used these to forecast monthly prescription counts that would be expected in the prospective phase, using simulation to obtain a set of 1-year predictions alongside their standard errors. We show how these forecasts can be combined to test for a change in prescription rates after introduction of the diagnostic and estimate power to detect this change.

RESULTS

Fitted time series models at each site were stationary and showed second-order annual seasonality, with a clear December peak in prescriptions, although the timing and extent of the peak varied between sites and between years. Mean one-year predictions of antibiotic prescribing rates based on the retrospective time series analysis differed between sites assigned to receive the diagnostic and those assigned to control. Adjusting for the trend in the retrospective time series at each site removed these differences.

CONCLUSIONS

Quasi-experimental designs such as comparative interrupted time series can be used in diagnostic evaluation to estimate effect sizes before conducting a full randomised controlled trial or if a randomised trial is infeasible. In multi-site studies, existing retrospective data should be used to adjust for underlying differences between sites to make outcome data from different sites comparable, when possible.

Impact of the national home safety equipment scheme 'Safe At Home' on hospital admissions for unintentional injury in children under 5: a controlled interrupted time series analysis

BACKGROUND

Unintentional home injuries are a leading cause of preventable death in young children. Safety education and equipment provision improve home safety practices, but their impact on injuries is less clear. Between 2009 and 2011, a national home safety equipment scheme was implemented in England (Safe At Home), targeting high-injury-rate areas and socioeconomically disadvantaged families with children under 5. This provided a 'natural experiment' for evaluating the scheme's impact on hospital admissions for unintentional injuries.

METHODS

Controlled interrupted time series analysis of unintentional injury hospital admission rates in small areas (Lower Layer Super Output Areas (LSOAs)) in England where the scheme was implemented (intervention areas, n=9466) and matched with LSOAs in England and Wales where it was not implemented (control areas, n=9466), with subgroup analyses by density of equipment provision.

RESULTS

57 656 homes receiving safety equipment were included in the analysis. In the 2 years after the scheme ended, monthly admission rates declined in intervention areas (-0.33% (-0.47% to -0.18%)) but did not decline in control areas (0.04% (-0.11%-0.19%), p value for difference in trend=0.001). Greater reductions in admission rates were seen as equipment provision density increased. Effects were not maintained beyond 2 years after the scheme ended.

CONCLUSIONS

A national home safety equipment scheme was associated with a reduction in injury-related hospital admissions in children under 5 in the 2 years after the scheme ended. Providing a higher number of items of safety equipment appears to be more effective in reducing injury rates than providing fewer items.

The detection and prevention of adverse drug events in nursing home and home care patients: Study protocol of a quasi-experimental study

AIM

To estimate the cost-effectiveness of an intervention facilitating the early detection of adverse drug events through the means of health professional training and the application of a digital screening tool.

DESIGN

Multi-centred non-randomized controlled trial from August 2018 to March 2020 including 65 nursing homes or home care providers.

METHODS

We aim to estimate the effect of the intervention on the rate of adverse drug events as primary outcome through a quasi-experimental empirical study design. As secondary outcomes, we use hospital admissions and falls. All outcomes will be measured on patient-month level. Once the causal effect of the intervention is estimated, cost-effectiveness will be calculated. For cost-effectiveness, we include all patient costs observed by the German statutory health insurance.

RESULTS

The results of this study will inform about the cost-effectiveness of the optimized drug supply intervention and provide evidence for potential reimbursement within the German statutory health insurance system.

What happens when performance-based financing meets free healthcare? Evidence from an interrupted time-series analysis

In spite of the wide attention performance-based financing (PBF) has received over the past decade, no evidence is available on its impacts on quantity and mix of service provision nor on its interaction with parallel health financing interventions. Our study aimed to examine the PBF impact on quantity and mix of service provision in Burkina Faso, while accounting for the parallel introduction of a free healthcare policy. We used Health Management Information System data from 838 primary-level health facilities across 24 districts and relied on an interrupted time-series analysis with independent controls. We placed two interruptions, one to account for PBF and one to account for the free healthcare policy. In the period before the free healthcare policy, PBF produced significant but modest increases across a wide range of maternal and child services, but a significant decrease in child immunization coverage. In the period after the introduction of the free healthcare policy, PBF did not affect service provision in intervention compared with control facilities, possibly indicating a saturation effect. Our findings indicate that PBF can produce modest increases in service provision, without altering the overall service mix. Our findings, however, also indicate that the introduction of other health financing reforms can quickly crowd out the effects produced by PBF. Further qualitative research is required to understand what factors allow healthcare providers to increase the provision of some, but not all services and how they react to the joint implementation of PBF and free health care.

Economic Analysis of Mandated Protocolized Sepsis Care in New York Hospitals

OBJECTIVES

Initial evidence suggests that state-level regulatory mandates for sepsis quality improvement are associated with decreased sepsis mortality. However, sepsis mandates require financial investments on the part of hospitals and may lead to increased spending. We evaluated the effects of the 2013 New York State sepsis regulations on the costs of care for patients hospitalized with sepsis.

DESIGN

Retrospective cohort study using state discharge data from the U.S. Healthcare Costs and Utilization Project and a comparative interrupted time series analytic approach. Costs were calculated from admission-level charge data using hospital-specific cost-to-charge ratios.

SETTING

General, short stay, acute care hospitals in New York, and four control states: Florida, Massachusetts, Maryland, and New Jersey.

PATIENTS

All patients hospitalized with sepsis between January 1, 2011, and September 30, 2015.

INTERVENTIONS

The 2013 New York mandate that all hospitals develop and implement protocols for sepsis identification and treatment, educate staff, and report performance data to the state.

MEASUREMENTS AND MAIN RESULTS

The analysis included 1,026,664 admissions in 520 hospitals. Mean unadjusted costs per hospitalization in New York State were $42,036 ± $60,940 in the pre-regulation period and $39,719 ± $59,063 in the post-regulation period, compared with $34,642 ± $52,403 pre-regulation and $31,414 ± $48,155 post-regulation in control states. In the comparative interrupted time series analysis, the regulations were not associated with a significant difference in risk-adjusted mean cost per hospitalization (p = 0.12) or risk-adjusted mean cost per hospital day (p = 0.44). For example, in the 10th quarter after implementation of the regulations, risk-adjusted mean cost per hospitalization was $3,627 (95% CI, -$681 to $7,934) more than expected in New York State relative to control states.

CONCLUSIONS

Mandated protocolized sepsis care was not associated with significant changes in hospital costs in patients hospitalized with sepsis in New York State.

EVALUATING CONTRADICTORY EXPERIMENTAL AND NON-EXPERIMENTAL ESTIMATES OF NEIGHBORHOOD EFFECTS ON ECONOMIC OUTCOMES FOR ADULTS

Although non-experimental studies find robust neighborhood effects on adults, such findings have been challenged by results from the Moving to Opportunity (MTO) residential mobility experiment. Using a within-study comparison design, this paper compares experimental and non-experimental estimates from MTO and a parallel analysis of the Panel Study of Income Dynamics (PSID). Striking similarities were found between non-experimental estimates based on MTO and PSID. No clear evidence was found that different estimates are related to duration of adult exposure to disadvantaged neighborhoods, non-linear effects of neighborhood conditions, magnitude of the change in neighborhood context, frequency of moves, treatment effect heterogeneity, or measurement, although uncertainty bands around our estimates were sometimes large. One other possibility is that MTO-induced moves might have been unusually disruptive, but results are inconsistent for that hypothesis. Taken together, the findings suggest that selection bias might account for evidence of neighborhood effects on adult economic outcomes in non-experimental studies.

Pediatric Outcomes After Regulatory Mandates for Sepsis Care

BACKGROUND

In 2013, New York introduced regulations mandating that hospitals develop pediatric-specific protocols for sepsis recognition and treatment.

METHODS

We used hospital discharge data from 2011 to 2015 to compare changes in pediatric sepsis outcomes in New York and 4 control states: Florida, Massachusetts, Maryland, and New Jersey. We examined the effect of the New York regulations on 30-day in-hospital mortality using a comparative interrupted time-series approach, controlling for patient and hospital characteristics and preregulation temporal trends.

RESULTS

We studied 9436 children admitted to 237 hospitals. Unadjusted pediatric sepsis mortality decreased in both New York (14.0% to 11.5%) and control states (14.4% to 11.2%). In the primary analysis, there was no significant effect of the regulations on mortality trends (differential quarterly change in mortality in New York compared with control states: -0.96%; 95% confidence interval [CI]: -1.95% to 0.02%; P = .06). However, in a prespecified sensitivity analysis excluding metropolitan New York hospitals that participated in earlier sepsis quality improvement, the regulations were associated with improved mortality trends (differential change: -2.08%; 95% CI: -3.79% to -0.37%; P = .02). The regulations were also associated with improved mortality trends in several prespecified subgroups, including previously healthy children (differential change: -1.36%; 95% CI: -2.62% to -0.09%; P = .04) and children not admitted through the emergency department (differential change: -2.42%; 95% CI: -4.24% to -0.61%; P = .01).

CONCLUSIONS

Implementation of statewide sepsis regulations was generally associated with improved mortality trends in New York State, particularly in prespecified subpopulations of patients, suggesting that the regulations were successful in affecting sepsis outcomes.

Association Between State-Mandated Protocolized Sepsis Care and In-hospital Mortality Among Adults With Sepsis

IMPORTANCE

Beginning in 2013, New York State implemented regulations mandating that hospitals implement evidence-based protocols for sepsis management, as well as report data on protocol adherence and clinical outcomes to the state government. The association between these mandates and sepsis outcomes is unknown.

OBJECTIVE

To evaluate the association between New York State sepsis regulations and the outcomes of patients hospitalized with sepsis.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective cohort study of adult patients hospitalized with sepsis in New York State and in 4 control states (Florida, Maryland, Massachusetts, and New Jersey) using all-payer hospital discharge data (January 1, 2011-September 30, 2015) and a comparative interrupted time series analytic approach.

EXPOSURES

Hospitalization for sepsis before (January 1, 2011-March 31, 2013) vs after (April 1, 2013-September 30, 2015) implementation of the 2013 New York State sepsis regulations.

MAIN OUTCOMES AND MEASURES

The primary outcome was 30-day in-hospital mortality. Secondary outcomes were intensive care unit admission rates, central venous catheter use, Clostridium difficile infection rates, and hospital length of stay.

RESULTS

The final analysis included 1 012 410 sepsis admissions to 509 hospitals. The mean age was 69.5 years (SD, 16.4 years) and 47.9% were female. In New York State and in the control states, 139 019 and 289 225 patients, respectively, were admitted before implementation of the sepsis regulations and 186 767 and 397 399 patients, respectively, were admitted after implementation of the sepsis regulations. Unadjusted 30-day in-hospital mortality was 26.3% in New York State and 22.0% in the control states before the regulations, and was 22.0% in New York State and 19.1% in the control states after the regulations. Adjusting for patient and hospital characteristics as well as preregulation temporal trends and season, mortality after implementation of the regulations decreased significantly in New York State relative to the control states (P = .02 for the joint test of the comparative interrupted time series estimates). For example, by the 10th quarter after implementation of the regulations, adjusted absolute mortality was 3.2% (95% CI, 1.0% to 5.4%) lower than expected in New York State relative to the control states (P = .004). The regulations were associated with no significant differences in intensive care unit admission rates (P = .09) (10th quarter adjusted difference, 2.8% [95% CI, -1.7% to 7.2%], P = .22), a significant relative decrease in hospital length of stay (P = .04) (10th quarter adjusted difference, 0.50 days [95% CI, -0.47 to 1.47 days], P = .31), a significant relative decrease in the C difficile infection rate (P < .001) (10th quarter adjusted difference, -1.8% [95% CI, -2.6% to -1.0%], P < .001), and a significant relative increase in central venous catheter use (P = .02) (10th quarter adjusted difference, 4.8% [95% CI, 2.3% to 7.4%], P < .001).

CONCLUSIONS AND RELEVANCE

In New York State, mandated protocolized sepsis care was associated with a greater decrease in sepsis mortality compared with sepsis mortality in control states that did not implement sepsis regulations. Because baseline mortality rates differed between New York and comparison states, it is uncertain whether these findings are generalizable to other states.

Effectiveness of antimalarial interventions in Nigeria: Evidence from facility-level longitudinal data

OBJECTIVE

To evaluate the effectiveness of a program of antimalarial interventions implemented in 2010-2013 in Niger State, Nigeria.

DATA SOURCES

Utilization reports from 99 intervention and 51 non-intervention health facilities from the Niger State Malaria Elimination Program, supplemented by data on facility-level characteristics from the Niger State Primary Health Care Development Agency and Local Government Malaria Control units.

STUDY DESIGN

Estimated with mixed-effects negative binomial modeling, a difference-in-differences method was used to quantify the impact of the program on the number of febrile illness cases and confirmed malaria cases. Potential confounding factors, non-stationarity, seasonality, and autocorrelation were explicitly accounted for.

DATA EXTRACTION METHODS

Data were retrieved from hard copies of utilization reports and manually inputted to create a panel of 5550 facility-month observations.

PRINCIPAL FINDINGS

The program was implemented in two phases. The first phase (August 2010-June 2012) involved the provision of free artemisinin-based combination therapies, long-lasting insecticidal nets, and intermittent preventive treatments. In the second phase (July 2012-March 2013), the program introduced an additional intervention: free parasite-based rapid diagnostic tests. Compared to the pre-intervention period, the average number of monthly febrile illness and malaria cases increased by 20.876 (P < 0.01) and 22.835 (P < 0.01) in the first phase, and by 19.007 (P < 0.05) and 19.681 (P < 0.05) in the second phase, respectively. The results are consistent across different evaluation methods.

CONCLUSIONS

This study suggests that user-fee removal leads to increased utilization of antimalarial services. It motivates future studies to cautiously select their investigative methods.

What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?

Given the widespread use of nonexperimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the same target population. The goal is to assess whether the stringent assumptions required for NE methods are likely to be met in practice. This essay provides an overview of recent efforts to empirically evaluate NE method performance in field settings. We discuss a brief history of the design, highlighting methodological innovations along the way. We also describe papers that are included in this two-volume special issue on WSC approaches and suggest future areas for consideration in the design, implementation, and analysis of WSCs.