The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Impact of transforming mental health services for young people in England on patient access, resource use and health: a quasi-experimental study

OBJECTIVE

To evaluate the impact of child and adolescent mental health services (CAMHS) transformation in South East England on patient access, resource utilisation and health outcomes.

DESIGN

In an observational study, we use difference-in-differences analysis with propensity score matching to analyse routinely collected patient level data.

SETTING

Three CAMHS services in South East England.

PARTICIPANTS

All patients attending CAMHS between April 2012 and December 2018, with more than 57 000 spells of care included.

MAIN OUTCOME MEASURES

The rate and volume of people accessing CAMHS; waiting times to the first contact and waiting times between the first and second contact; and health outcomes, including the Strengths and Difficulties Questionnaire (SDQ) and the Revised Child Anxiety and Depression Scale (RCADS).

RESULTS

The intervention led to 20% (incidence rate ratio: 1.20; 95% CI: 1.15 to 1.24) more new patients starting per month. There was mixed evidence on waiting times for the first contact. The intervention led to 10% (incidence rate ratio: 1.10; 95% CI: 1.02 to 1.18) higher waiting time for the second contact. The number of contacts per spell (OR: 1.08; 95% CI: 0.94 to 1.25) and the rereferral rate (OR: 1.06; 95% CI: 0.96 to 1.17) were not significantly different. During the post intervention period, patients in the intervention group scored on average 3.3 (95% CI: -5.0 to -1.6) points lower on the RCADS and 1.0 (95% CI: -1.8 to -0.3) points lower on the SDQ compared with the control group after adjusting for the baseline score.

CONCLUSIONS

Overall, there are signs that transformation can help CAMHS achieve the objectives of greater access and improved health outcomes, but trade-offs exist among different performance metrics, particularly between access and waiting times. Commissioners and providers should be conscious of any trade-offs when undertaking service redesign and transformation.

Generalizability of randomized trial results to target populations: Design and analysis possibilities

Randomized trials play an important role in estimating the effect of a policy or social work program in a given population. While most trial designs benefit from strong internal validity, they often lack external validity, or generalizability, to the target population of interest. In other words, one can obtain an unbiased estimate of the study sample average treatment effect (SATE) from a randomized trial; however, this estimate may not equal the target population average treatment effect (TATE) if the study sample is not fully representative of the target population. This paper provides an overview of existing strategies to assess and improve upon the generalizability of randomized trials, both through statistical methods and study design, as well as recommendations on how to implement these ideas in social work research.

Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data

BACKGROUND

Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population.

OBJECTIVES

This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample.

RESEARCH DESIGN

We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental program for Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study.

RESULTS

For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging.

CONCLUSIONS

To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered.

Characteristics of School Districts That Participate in Rigorous National Educational Evaluations

Given increasing interest in evidence-based policy, there is growing attention to how well the results from rigorous program evaluations may inform policy decisions. However, little attention has been paid to documenting the characteristics of schools or districts that participate in rigorous educational evaluations, and how they compare to potential target populations for the interventions that were evaluated. Utilizing a list of the actual districts that participated in 11 large-scale rigorous educational evaluations, we compare those districts to several different target populations of districts that could potentially be affected by policy decisions regarding the interventions under study. We find that school districts that participated in the 11 rigorous educational evaluations differ from the interventions' target populations in several ways, including size, student performance on state assessments, and location (urban/rural). These findings raise questions about whether, as currently implemented, the results from rigorous impact studies in education are likely to generalize to the larger set of school districts-and thus schools and students-of potential interest to policymakers, and how we can improve our study designs to retain strong internal validity while also enhancing external validity.