The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Functional Family Therapy for families of youth (age 11-18) with behaviour problems: A systematic review and meta-analysis

Background

Functional Family Therapy (FFT) is a short-term family-based intervention for youth with behaviour problems. FFT has been widely implemented in the USA and other high-income countries. It is often described as an evidence-based program with consistent, positive effects.

Objectives

We aimed to synthesise the best available data to assess the effectiveness of FFT for families of youth with behaviour problems.

Search Methods

Searches were performed in 2013-2014 and August 2020. We searched 22 bibliographic databases (including PsycINFO, ERIC, MEDLINE, Science Direct, Sociological Abstracts, Social Services Abstracts, World CAT dissertations and theses, and the Web of Science Core Collection), as well as government policy databanks and professional websites. Reference lists of articles were examined, and experts were contacted to search for missing information.

Selection Criteria

We included randomised controlled trials (RCTs) and quasi-experimental designs (QEDs) with parallel cohorts and statistical controls for between-group differences at baseline. Participants were families of young people aged 11-18 with behaviour problems. FFT programmes were compared with usual services, alternative treatment, and no treatment. There were no publication, geographic, or language restrictions.

Data Collection and Analysis

Two reviewers independently screened 1039 titles and abstracts, read all available study reports, assessed study eligibility, and extracted data onto structured electronic forms. We assessed risks of bias (ROB) using modified versions of the Cochrane ROB tool and the What Works Clearinghouse standards. Where possible, we used random effects models with inverse variance weights to pool results across studies. We used odds ratios for dichotomous outcomes and standardised mean differences for continuous outcomes. We used Hedges g to adjust for small sample sizes. We assessed the heterogeneity of effects with χ 2 and I 2. We produced separate forest plots for conceptually distinct outcomes and for different endpoints (<9, 9-14, 15-23, and 24-42 months after referral). We grouped studies by study design (RCT or QED), and then assessed differences between these two subgroups of studies with χ 2 tests. We generated robust variance estimates, using correlated effects (CE) models with small sample corrections to synthesise all available outcome data. Exploratory CE analyses assessed potential moderators of effects within these domains. We used GRADE guidelines to assess the certainty of evidence on six primary outcomes at 1 year after referral.

Main Results

Twenty studies (14 RCTs and 6 QEDs) met our inclusion criteria. Fifteen of these studies provided some valid data for meta-analysis; these studies included 10,980 families in relevant FFT and comparison groups. All included studies had high risks of bias on at least one indicator. Half of the studies had high risks of bias on baseline equivalence, support for intent-to-treat analysis, selective reporting, and conflicts of interest. Fifteen studies had incomplete reporting of outcomes and endpoints. Using the GRADE rubric, we found that the certainty of evidence for FFT was very low for all of our primary outcomes. Using pairwise meta-analysis, we found no evidence of effects of FFT compared with other active treatments on any primary or secondary outcomes. Primary outcomes were: recidivism, out-of-home placement, internalising behaviour problems, external behaviour problems, self-reported delinquency, and drug or alcohol use. Secondary outcomes were: peer relations and prosocial behaviour, youth self esteem, parent symptoms and behaviour, family functioning, school attendance, and school performance. There were few studies in the pairwise meta-analysis (k < 7) and little heterogeneity of effects across studies in most of these analyses. There were few differences between effect estimates obtained in RCTs versus QEDs. More comprehensive CE models showed positive results of FFT in some domains and negative results in others, but these effects were small (standardised mean difference [SMD] <|0.20|) and not significantly different from no effect with one exception: Two studies found positive effects of FFT on youth substance abuse and two studies found null results in this domain, and the overall effect estimate for this outcome was statistically different from zero. Over all outcomes (15 studies and 293 effect sizes), small positive effects were detected (SMD = 0.19, SE = 0.09), but these were not significantly different from zero effect. Prediction intervals showed that future FFT evaluations are likely to produce a wide range of results, including moderate negative effects and strong positive results (-0.37 to 0.75).

Authors’ Conclusions

Results of 10 RCTs and five QEDs show that FFT does not produce consistent benefits or harms for youth with behavioural problems and their families. The positive or negative direction of results is inconsistent within and across studies. Most outcomes are not fully reported, the quality of available evidence is suboptimal, and the certainty of this evidence is very low. Overall estimates of effects of FFT may be inflated, due to selective reporting and publication biases.

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

BACKGROUND

Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards.

ISSUES

In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review.

CONCLUSIONS

Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Do haphazard reviews provide sound directions for dissemination efforts?

Comments on The dissemination and implementation of evidence-based psychological treatments: A review of current efforts (see record 2010-02208-010) by Kathryn R. McHugh and David H. Barlow. The lead article in the February–March issue by McHugh and Barlow (2010) emphasized the need for “dissemination and implementation of evidence-based psychological treatments.” The authors identified a number of intervention programs as evidence based and in need of dissemination. One is multisystemic therapy (MST). They claimed that this program is among “the most successful dissemination efforts...pursued by treatment developers” (p. 79). McHugh and Barlow’s (2010) discussion of the implementation of MST in Hawaii is troubling, because it neglected to mention concerns about the perceived lack of cultural sensitivity of the MST program in that state.

Correlates of problem recognition and intentions to change among caregivers of abused and neglected children

OBJECTIVE

To identify individual, family, and caseworker characteristics associated with problem recognition (PR) and intentions to change (ITC) in a sample of caregivers who received in-home child welfare services following substantiated reports of child abuse or neglect.

METHODS

Caregivers were interviewed at 4 weeks, 16 weeks, and 1 year after referral for in-home services. In these interviews, the University of Rhode Island Change Assessment scale assessed PR and ITC in relation to caregiving practices. Additional data were obtained from administrative records and surveys of in-home services caseworkers. We used growth models to identify caregiver, family, and caseworker characteristics associated with initial levels of PR and ITC, and with changes in PR and ITC over time.

RESULTS

Contrary to expectations, there were no overall increases in PR and ITC during the first 4 months of in-home services. PR and ITC scores fluctuated over time, in relation to some external events, case characteristics, and caseworker characteristics; however, we were able to account for small portions of the variance in PR and ITC. Controlling for social desirability bias and other variables in the analysis, negative life events, lack of network support, and the severity of caregiver depression were associated with greater PR; housing problems were associated with lower PR. Caregiver depression and age were associated with ITC. Caregivers whose children had been removed from their homes showed significant increases in ITC in the first few months of in-home services. The duration of caseworkers' child welfare experience predicted increases in their clients' PR in the first 4 months and more frequent contacts with an experienced caseworker predicted small, but significant increases in ITC over time.

CONCLUSIONS

PR and ITC are associated with somewhat different case characteristics and may be affected by caseworkers' experience.

Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10-17

BACKGROUND

Multisystemic Therapy (MST) is an intensive, home-based intervention for families of youth with social, emotional, and behavioral problems. Masters-level therapists engage family members in identifying and changing individual, family, and environmental factors thought to contribute to problem behavior. Intervention may include efforts to improve communication, parenting skills, peer relations, school performance, and social networks. Most MST trials were conducted by program developers in the USA; results of one independent trial are available and others are in progress.

OBJECTIVES

To provide unbiased estimates of the impacts of MST on restrictive out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

SEARCH STRATEGY

Electronic searches were made of bibliographic databases (including the Cochrane Library, C2-SPECTR, PsycINFO, Science Direct and Sociological Abstracts) as well as government and professional websites, from 1985 to January 2003. Reference lists of articles were examined, and experts were contacted.

SELECTION CRITERIA

Studies where youth (age 10-17) with social, emotional, and/or behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments).

DATA COLLECTION AND ANALYSIS

Two reviewers independently reviewed 266 titles and abstracts; 95 full-text reports were retrieved, and 35 unique studies were identified. Two reviewers independently read all study reports for inclusion. Eight studies were eligible for inclusion. Two reviewers independently assessed study quality and extracted data from these studies. Significant heterogeneity among studies was identified (assessed using Chi-square and I(2)), hence random effects models were used to pool data across studies. Odds ratios were used in analyses of dichotomous outcomes; standardised mean differences were used with continuous outcomes. Adjustments were made for small sample sizes (using Hedges g). Pooled estimates were weighted with inverse variance methods, and 95% confidence intervals were used.

MAIN RESULTS

The most rigorous (intent-to-treat) analysis found no significant differences between MST and usual services in restrictive out-of-home placements and arrests or convictions. Pooled results that include studies with data of varying quality tend to favor MST, but these relative effects are not significantly different from zero. The study sample size is small and effects are not consistent across studies; hence, it is not clear whether MST has clinically significant advantages over other services.

AUTHORS' CONCLUSIONS

There is inconclusive evidence of the effectiveness of MST compared with other interventions with youth. There is no evidence that MST has harmful effects.

Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10-17

BACKGROUND

Multisystemic Therapy (MST) is an intensive, home-based intervention for families of youth with social, emotional, and behavioral problems. Masters-level therapists engage family members in identifying and changing individual, family, and environmental factors thought to contribute to problem behavior. Intervention may include efforts to improve communication, parenting skills, peer relations, school performance, and social networks. Most MST trials were conducted by program developers in the USA; results of one independent trial are available and others are in progress.

OBJECTIVES

To provide unbiased estimates of the impacts of MST on restrictive out-of-home living placements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

SEARCH STRATEGY

Electronic searches were made of bibliographic databases including the Cochrane Library, C2-SPECTR, PsycINFO, Science Direct and Sociological Abstracts) as well as government and professional websites, from 1985 to January 2003. Reference lists of articles were examined, and experts were contacted.

SELECTION CRITERIA

Studies where youth (age 10-17) with social, emotional, and/or behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments).

DATA COLLECTION AND ANALYSIS

Two reviewers independently reviewed 266 titles and abstracts; 95 full-text reports were retrieved, and 35 unique studies were identified. Two reviewers independently read all study reports for inclusion. Eight studies were eligible for inclusion. Two reviewers independently assessed study quality and extracted data from these studies.Significant heterogeneity among studies was identified (assessed using Chi-square and I(2)), hence random effects models were used to pool data across studies. Odds ratios were used in analyses of dichotomous outcomes; standardised mean differences were used with continuous outcomes. Adjustments were made for small sample sizes (using Hedges g). Pooled estimates were weighted with inverse variance methods, and 95% confidence intervals were used.

MAIN RESULTS

Pooled results show no significant effects of MST on the likelihood or duration of restrictive out-of-home placements, proportion of youth who were arrested or convicted, or numbers of arrests/convictions within one-year post-intervention. There were no significant differences on drug tests or self-reported drug use at a 6-month follow-up. In analyses of post-treatment data for program completers, there were no significant between-group differences on self-reported delinquency, peer relations, youth behavior problems, youth psychiatric symptoms, parent psychiatric symptoms, or family functioning.

AUTHORS' CONCLUSIONS

There is little evidence of the superiority of MST over other interventions with youth. There is also no evidence that MST has harmful effects.

Caregivers' readiness for change: predictive validity in a child welfare sample

OBJECTIVE

To assess the predictive validity of continuous measures of problem recognition (PR), intentions to change (ITC), and overall readiness for change (RFC) among primary caregivers who received in-home services following substantiated reports of child abuse or neglect.

METHOD

A modified version of the University of Rhode Island Change Assessment scale was included in interviews with a sample of 353 primary caregivers at 4 weeks, 16 weeks, and 1 year after referral for in-home services. Additional data were obtained from administrative records and caseworker surveys. Hierarchical linear and nonlinear models were used to assess relationships between PR, ITC, RFC and changes over time in measures of individual and family functioning (e.g., parenting behaviors, children's behaviors, housing and economic problems, social support, and life events). Bivariate probit regression analysis was used to examine relationships between PR, ITC, RFC and the likelihood of subsequent reports of child maltreatment and out-of-home placements within 1 year after referral.

RESULTS

Initial problem recognition and intentions to change predict a few improvements in individual and family functioning, along with significant reductions in the likelihood of additional reports of child maltreatment within 1 year. Initial intentions to change also predict reductions in the substantiation of subsequent reports of maltreatment. An overall measure of readiness for change predicts reductions in the likelihood of out-of-home placement.

CONCLUSIONS

Problem recognition and intentions to change predict somewhat different outcomes; hence, there are few advantages of a combined measure of readiness for change. Further inquiry is needed to determine whether and how these associations are mediated by intervention processes or other factors in child welfare services populations.

Ready or not: uses of the stages of change model in child welfare

This article reviews the popular stages of change model, its potential applications in child welfare, and relevant research. Empirical evidence indicates that behavioral change does not occur in a series of stages. The article considers the validity of the stage model, its underlying assumptions, and other conceptualizations of readiness for change. Although the stage model may have some heuristic value, the empirical evidence suggests that its practical applications are severely limited.

Stages of change. A critique

The stages of change proposed by Prochaska and DiClemente have been applied to change efforts within and outside of formal treatment and in relation to virtually any problem behavior. This model has gained widespread popularity in health psychology and addictions and is being used to guide interventions and allocate treatment resources in several fields. In this article, the authors review 87 studies on the stages of change across problem behaviors. Research findings suggest that the proposed stages are not mutually exclusive and that there is scant evidence of sequential movement through discrete stages in studies of specific problem behaviors, such as smoking and substance abuse. Although the stage model may have considerable heuristic value, its practical utility is limited by concerns about the validity of stage assessments. The model's underlying concepts and alternative views of readiness for change are considered, along with directions for future research.