Using dissertations to examine potential bias in child and adolescent clinical trials

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Key actors in behavioral health services availability and accessibility research: a scoping review bibliometric analysis

This bibliometric review aims to identify key actors in the behavioral health services availability/accessibility literature. Coalescing information about these actors could support subsequent research efforts to improve the availability and accessibility of behavioral health services. The authors used a scoping review method and a bibliometric approach. The articles came from Medline, Embase, Web of Science, CINAHL, and PsycINFO. Articles were included if they assessed behavioral health service availability or accessibility quantitatively and were written in English. The final sample included 265 articles. Bibliometric data were extracted, coded, and verified. The authors analyzed the data using univariate and social network analyses. Publishing in this area has become more consistent and has grown since 2002. Psychiatric Services and Graduate Theses were the most frequently used publication venues. The National Institute on Drug Abuse, National Institute of Mental Health, and the Veterans Administration funded the most research. The most frequently used keyword was "health services accessibility." The findings suggest that this literature is growing. There are a few clusters of researchers in this area. Government organizations primarily fund this research. The paper and supplementary materials list the top researchers, publication venues, funding sources, and key terms to promote further behavioral health availability/accessibility research.

Publication bias in psychology: A closer look at the correlation between sample size and effect size

Previously observed negative correlations between sample size and effect size (n-ES correlation) in psychological research have been interpreted as evidence for publication bias and related undesirable biases. Here, we present two studies aimed at better understanding to what extent negative n-ES correlations reflect such biases or might be explained by unproblematic adjustments of sample size to expected effect sizes. In Study 1, we analysed n-ES correlations in 150 meta-analyses from cognitive, organizational, and social psychology and in 57 multiple replications, which are free from relevant biases. In Study 2, we used a random sample of 160 psychology papers to compare the n-ES correlation for effects that are central to these papers and effects selected at random from these papers. n-ES correlations proved inconspicuous in meta-analyses. In line with previous research, they do not suggest that publication bias and related biases have a strong impact on meta-analyses in psychology. A much higher n-ES correlation emerged for publications' focal effects. To what extent this should be attributed to publication bias and related biases remains unclear.

Family-Based Interventions for Youth Depression: Meta-Analysis of Randomized Clinical Trials

Youth depression is an impairing pediatric condition for which psychotherapy effects are modest. Can outcomes be improved by treatments that address the family context, as proposed in practice parameters of the American Academy of Child and Adolescent Psychiatry? To find out, we searched five decades of research for randomized controlled trials testing family-based interventions; 11 trials were found for ages 4-18 years. Using robust variance estimation, we obtained estimates of effect size (ES) and tested candidate moderators that might explain variation in ES. Overall pooled ES of the studies was 0.33 at posttreatment, similar to that reported for all youth depression treatments in three prior meta-analyses (i.e., 0.36, 0.30, and 0.29), but higher for adolescents (ages ≥ 13, 0.50) than children (0.04). Taken together, our findings do not show superior outcomes for family-based interventions, and raise questions about how much confidence can be placed in the evidence base to date. The small number of relevant studies in more than five decades, together with the publication bias and risk of bias concerns, highlights the need for more trials testing family-based treatment, and for pre-registered reports, publicly-available preprints, and other mechanisms for improving the dissemination of completed research.

Systematic Review and Meta-analysis: Psychosocial Treatments for Disruptive Behavior Symptoms and Disorders in Adolescence

OBJECTIVE

Disruptive behavior disorders (DBDs) represent a common motive for referral among youths. This meta-analysis aimed at estimating the efficacy of psychosocial interventions for adolescents with DBDs.

METHOD

A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-compliant systematic review of MEDLINE/PubMED/PsycINFO/Cochrane Central Register of Controlled Trials was conducted. Eligible studies were randomized controlled trials (RCTs) administering psychosocial interventions to adolescents with DBD, published before April 5, 2020. From the initial set of 6,006 records, 17 RCTs involving 18 cohorts (16 publications) were subjected to a random-effect meta-analysis (involving sensitivity, subgroup, and meta-regression analyses). Primary and secondary outcomes were externalizing symptoms at RCT endpoint (ie, standardized mean difference [SMD]) and acceptability (drop-out odds ratio [OR]), respectively. Risk of bias was assessed using the Risk of Bias 2 tool.

RESULTS

A total of 17 RCTs, involving 1,954 adolescents, were included. The mean age was 14.09 (SD 1.33) years; 61% were male. The mean RCT duration was 12 weeks, with a mean follow-up of 8 (SD 3.98) months. Concern over risk of bias emerged in 12 studies, with high concern in 6. Psychosocial interventions had a large effect size at RCT endpoint (SMD = 0.98, 95% CI = -0.55 to -1.38, k = 18) and were acceptable (drop-out OR = 1.29, 95% CI = 0.62-2.70, k = 13). However, this beneficial effect did not persist at follow-up (SMD = -0.36, 95% CI = 0.06 to -0.78, k = 10). Family format was the most effective variable. No other clinically significant moderator was found.

CONCLUSION

Psychosocial interventions involving the families of DBD adolescents are effective and acceptable in the short term. Future studies should focus on strategies to achieve their long-term efficacy.

"Nothing to see here": No structural brain differences as a function of the Big Five personality traits from a systematic review and meta-analysis

Personality reflects social, affective, and cognitive predispositions that emerge from genetic and environmental influences. Contemporary personality theories conceptualize a Big Five Model of personality based on the traits of neuroticism, extraversion, agreeableness, conscientiousness, and openness to experience. Starting around the turn of the millennium, neuroimaging studies began to investigate functional and structural brain features associated with these traits. Here, we present the first study to systematically evaluate the entire published literature of the association between the Big Five traits and three different measures of brain structure. Qualitative results were highly heterogeneous, and a quantitative meta-analysis did not produce any replicable results. The present study provides a comprehensive evaluation of the literature and its limitations, including sample heterogeneity, Big Five personality instruments, structural image data acquisition, processing, and analytic strategies, and the heterogeneous nature of personality and brain structures. We propose to rethink the biological basis of personality traits and identify ways in which the field of personality neuroscience can be strengthened in its methodological rigor and replicability.

Do insomnia and/or sleep disturbances predict the onset, relapse or worsening of depression in community and clinical samples of children and youth? Protocol for a systematic review and meta-analysis

INTRODUCTION

Disturbed sleep represents a potentially important modifiable risk factor for the development of depression in children and youth. This protocol for a systematic review proposes to investigate whether insomnia and/or sleep disturbances predict child and youth depression in community and clinical-based samples.

METHODS AND ANALYSIS

The protocol adheres to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines. English-written, longitudinal studies that quantitatively estimated the prediction of depression by insomnia and/or sleep disturbances in individuals 5-24 years of age will be included. EMBASE, MEDLINE, PsychINFO, Scopus and Web of Science and grey literature will be searched from 1980 to the present. For the selection of studies, two reviewers will be involved. Data extraction will be conducted by one author and checked independently by a second author. Risk of bias will be appraised using the Research Triangle Institute Item Bank tool. Heterogeneity will be measured using the I2 statistics. Meta-analysis will be carried out if ≥3 results are available and if outcome measures can be pooled. The choice between a random-effect or fixed-effect model will be based both on the I2 statistics and the participant and study characteristics of the combined studies. Results of the meta-analyses will be summarised by a forest plot. Analyses will be performed in subgroups stratified by key variables defined depending on the amount and type of information retrieved.A narrative synthesis will be conducted in place of the meta-analysis should the pooling of data not be possible. Quality of evidence will be rated using the Grading of Recommendations Assessment, Development and Evaluation guidelines.As this is a protocol for systematic review and meta-analysis of published data, ethics review and approval are not required. The findings will be published in a peer-reviewed journal and disseminated at scientific conferences and in patient advocacy organisations.

PROSPERO REGISTRATION NUMBER

CRD42019136729.

The Effects of Physical Exercise on Functional Outcomes in the Treatment of ADHD: A Meta-Analysis

Objective: An increasing number of studies suggest possible beneficial effects of exercise in alleviating ADHD functional outcomes. The current study provides a quantitative meta-analysis of the available studies investigating this relationship. Method: Studies reporting on the effects of physical exercise on motor skills and executive functions in children with ADHD were identified through Cochrane, PsycInfo, PubMed, Web of Science databases. Ten publications were selected. Random-effects model was used to calculate effect sizes. Results: There was a significant effect of exercise on ADHD functional outcomes (g = 0.627). Longer exercise intervention duration was consistently associated with larger effect sizes. Effect sizes were not related to exercise intensity, mean age of participants, or gender distribution. Conclusion: Results suggest that exercise has a modest positive impact on ADHD functional outcomes, such as executive functions and motor skills, with longer interventions yielding better results.

Meta-Analysis: 13-Year Follow-up of Psychotherapy Effects on Youth Depression

OBJECTIVE

Youth depression is a debilitating condition that constitutes a major public health concern. A 2006 meta-analysis found modest benefits for psychotherapy versus control. Has 13 more years of research improved that picture? We sought to find out.

METHOD

We searched PubMed, PsychINFO, and Dissertation Abstracts International for 1960 to 2017, identifying 655 randomized, English-language psychotherapy trials for individuals aged 4 to 18 years. Of these, 55 assessed psychotherapy versus control for youth depression with outcome measures administered to both treatment and control conditions at post (κ = 53) and/or follow-up (κ = 32). Twelve study and outcome characteristics were extracted, and effect sizes were calculated for all psychotherapy versus control comparisons. Using a three-level random-effects model, we obtained an overall estimate of the psychotherapy versus control difference while accounting for the dependency among effect sizes. We then fitted a three-level mixed-effects model to identify moderators that might explain variation in effect size within and between studies.

RESULTS

The overall effect size (g) was 0.36 at posttreatment and 0.21 at follow-up (averaging 42 weeks after posttreatment). Three moderator effects were identified: effects were significantly larger for interpersonal therapy than for cognitive behavioral therapy, for youth self-reported outcomes than parent-reports, and for comparisons with inactive control conditions (eg, waitlist) than active controls (eg, usual care). Effects showed specificity, with significantly smaller effects for anxiety and externalizing behavior outcomes than for depression measures.

CONCLUSION

Youth depression psychotherapy effects are modest, with no significant change over the past 13 years. The findings highlight the need for treatment development and research to improve both immediate and longer-term benefits.

The perceived role of null results in school psychology research and publication

Researchers appear to assume that published research is limited to significant findings. If that is the case, it may be related to perceived or actual publication bias (i.e., journals publishing only significant findings) and/or the file-drawer problem (i.e., researchers not pursuing publication of null results). The lack of published null results can result in faulty decision-making based upon incomplete evidence. Thus, it is important to know the prevalence of, and the contributing factors to, researchers' failure to submit null results. Few studies have addressed this issue in psychology and none have targeted school psychology. Consequently, this study examined the file drawer problem and perception of publication bias among school psychologists. Survey data from 95 school psychology faculty indicated that participants published about half of the studies that they had conducted, suggesting that the file drawer problem is experienced by this population. While lack of time appeared to impact publication pursuit, participants' responses also suggested they believed in publication bias. Obtaining null results substantially impacted the decision to write up studies in pursuit of publication. Therefore, it seems that a sizeable percentage of school psychology research is not available for review by researchers or practitioners.

Treatment Features Associated with Youth Cognitive Behavioral Therapy Follow-Up Effects for Internalizing Disorders: A Meta-Analysis

Our aim was to investigate whether four treatment features (i.e., the inclusion of parental involvement, goal-setting strategies, maintenance/relapse prevention sessions, the addition of booster sessions) were associated with posttreatment and follow-up effect size of youth cognitive behavioral therapies (yCBTs) for anxiety, depression, posttraumatic stress disorder, and obsessive-compulsive disorder in age groups spanning young children to adolescents. We conducted a random-effects meta-analysis of 106 yCBTs tested in 76 randomized clinical trials from the PracticeWise Database to examine average effects of yCBTs posttreatment and at a later follow-up assessment. We coded the use of parental involvement, goal setting, booster sessions, and maintenance/relapse prevention in each yCBT and conducted random-effects meta-regression analyses to investigate whether these treatment features were associated with yCBT effects at posttreatment as well as at follow-up. Overall, yCBTs produced large pre- to posttreatment effects (d = 1.05), 95% confidence interval [0.94, 1.15], and larger pre- to follow-up effects (d = 1.29), 95% confidence interval [1.18, 1.40]. Metaregression results indicated that parental involvement was significantly associated with larger pre- to posttreatment effect sizes as well as pre- to follow-up effect sizes. Booster sessions, goal setting, and maintenance/relapse prevention were not significantly related to effect sizes at posttreatment or follow-up. Parental involvement may be helpful for maximizing long-term effectiveness of yCBT. Future studies should investigate for whom and under what conditions inclusion of yCBT treatment features is related to the durability of treatment gains.

A meta-analysis of longitudinal associations between substance use and interpersonal attachment security

Substance use has long been associated with close relationship distress. Although the direction of influence for this association has not been established, it has often been assumed that substance use is the causal agent and that close relationship distress is the effect. But research seeking to establish temporal precedence in this link has produced mixed findings. Further, theoretical models of substance use and close relationship processes present the plausibility of the inverse pathway-that insecure close relationships may serve as a vulnerability factor for the development of later substance problems. The current review applies an attachment-theoretical framework to the association between close social bonds and substance use and substance-related problems. Targeting longitudinal studies of attachment and substance use, we examined 665 effect sizes drawn from 34 samples (total N = 56,721) spanning time frames ranging from 1 month to 20 years (M = 3.8 years). Results revealed a significant prospective correlation between earlier attachment and later substance use (r = -.11, 95% CI [-.14, -0.08]). Further, cross-lagged coefficients were calculated which parsed auto-regressive effects, indicating that lower attachment security temporally preceded increases in substance use (r = -.05, 95% CI [-.06, -.04]). Analyses further indicated that the pathway from earlier attachment to later substance use was significantly stronger than that from earlier substance use to later attachment. Results also revealed several moderators of the attachment-substance use link. These findings suggest that insecure attachment may be a vulnerability factor for substance use, and indicate close relationship quality as a promising line of inquiry in research on substance use disorder risk. (PsycINFO Database Record

"Are you gonna publish that?" Peer-reviewed publication outcomes of doctoral dissertations in psychology

If a doctoral dissertation represents an original investigation that makes a contribution to one's field, then dissertation research could, and arguably should, be disseminated into the scientific literature. However, the extent and nature of dissertation publication remains largely unknown within psychology. The present study investigated the peer-reviewed publication outcomes of psychology dissertation research in the United States. Additionally, we examined publication lag, scientific impact, and variations across subfields. To investigate these questions, we first drew a stratified random cohort sample of 910 psychology Ph.D. dissertations from ProQuest Dissertations & Theses. Next, we conducted comprehensive literature searches for peer-reviewed journal articles derived from these dissertations published 0-7 years thereafter. Published dissertation articles were coded for their bibliographic details, citation rates, and journal impact metrics. Results showed that only one-quarter (25.6% [95% CI: 23.0, 28.4]) of dissertations were ultimately published in peer-reviewed journals, with significant variations across subfields (range: 10.1 to 59.4%). Rates of dissertation publication were lower in professional/applied subfields (e.g., clinical, counseling) compared to research/academic subfields (e.g., experimental, cognitive). When dissertations were published, however, they often appeared in influential journals (e.g., Thomson Reuters Impact Factor M = 2.84 [2.45, 3.23], 5-year Impact Factor M = 3.49 [3.07, 3.90]) and were cited numerous times (Web of Science citations per year M = 3.65 [2.88, 4.42]). Publication typically occurred within 2-3 years after the dissertation year. Overall, these results indicate that the large majority of Ph.D. dissertation research in psychology does not get disseminated into the peer-reviewed literature. The non-publication of dissertation research appears to be a systemic problem affecting both research and training in psychology. Efforts to improve the quality and "publishability" of doctoral dissertation research could benefit psychological science on multiple fronts.

Does psychotherapy work with school-aged youth? A meta-analytic examination of moderator variables that influence therapeutic outcomes

The present study is a quantitative synthesis of the available literature to investigate the efficacy of psychotherapy for children's mental health outcomes. In particular, this study focuses on potential moderating variables-study design, treatment, client, and therapist characteristics-that may influence therapeutic outcomes for youth but have not been thoroughly accounted for in prior meta-analytic studies. An electronic search of relevant databases resulted in 190 unpublished and published studies that met criteria for inclusion in the analysis. Effect sizes differed by study design. Pre-post-test designs resulted in absolute magnitudes of treatment effects ranging from |-0.02| to |-0.76| while treatment versus control group comparison designs resulted in absolute magnitudes of treatment effects ranging from |-0.14| to |-2.39|. Changes in youth outcomes larger than 20% were found, irrespective of study design, for outcomes focused on psychosomatization (29% reduction), school attendance (25% increase), and stress (48% reduction). The magnitude of changes after psychotherapy ranged from 6% (externalizing problems) to 48% (stress). Several moderator variables significantly influenced psychotherapy treatment effect sizes, including frequency and length of treatment as well as treatment format. However, results did not support the superiority of a single type of intervention for most outcomes. Implications for therapy with school-aged youth and future research are discussed.

Historical Synthesis-Analysis of Changes in Grain Nitrogen Dynamics in Sorghum

Unraveling the complexity underpinning nitrogen (N) use efficiency (NUE) can be physiologically approached via examining grain N sources and N internal efficiency (NIE) (yield to plant N content ratio). The main objective of this original research paper is to document and understand sorghum NUE and physiological mechanisms related to grain N dynamics. The study of different grain N sources, herein defined as the reproductive-stage shoot N remobilization (Remobilized N), reproductive-stage whole-plant N content (Reproductive N), and vegetative-stage whole-plant N content (Vegetative N), was pursued with the goal of synthesizing scientific literature for sorghum [Sorghum bicolor (L.) Moench] crop. A detailed literature review was performed and summarized on sorghum NUE (13 studies; >250 means) with three Eras, defined by the year of the study, named as Old Era (1965-1980); Transient Era (1981-2000); and New Era (2001-2014). The most remarkable outcomes from this synthesis were: (1) overall historical (1965-2014) cumulative yield gain was >0.5 Mg ha(-1) (yields >7 Mg ha(-1)); (2) NIE did not change across the same time period; (3) grain N concentration (grain %N) accounted for a large proportion (63%) of the variation in NIE; (4) NIE increased as grain %N diminished, regardless of the Eras; (5) Remobilized N was strongly (>R (2) 0.6) and positively associated with Vegetative N, presenting a unique slope across Eras; and (6) a trade-off was documented for the Remobilized N and Reproductive N (with large variation, <R (2)) relationship, suggesting complex regulation processes governing N forces. Improvements in NUE are subjected to the interplay between N supply (N from non-reproductive organs) and grain N demand, sink- (driven by grain number) and source-modulated (via restriction of grain N demand).

Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: a systematic review and meta-analysis

The popularity, demand, and increased federal and private funding for after-school programs have resulted in a marked increase in after-school programs over the past two decades. After-school programs are used to prevent adverse outcomes, decrease risks, or improve functioning with at-risk youth in several areas, including academic achievement, crime and behavioral problems, socio-emotional functioning, and school engagement and attendance; however, the evidence of effects of after-school programs remains equivocal. This systematic review and meta-analysis, following Campbell Collaboration guidelines, examined the effects of after-school programs on externalizing behaviors and school attendance with at-risk students. A systematic search for published and unpublished literature resulted in the inclusion of 24 studies. A total of 64 effect sizes (16 for attendance outcomes; 49 for externalizing behavior outcomes) extracted from 31 reports were included in the meta-analysis using robust variance estimation to handle dependencies among effect sizes. Mean effects were small and non-significant for attendance and externalizing behaviors. A moderate to large amount of heterogeneity was present; however, no moderator variable tested explained the variance between studies. Significant methodological shortcomings were identified across the corpus of studies included in this review. Implications for practice, policy and research are discussed.

A retrospective analysis of dissemination biases in the brief alcohol intervention literature

This study examined dissemination and reporting biases in the brief alcohol intervention literature. We used retrospective data from 179 controlled trials included in a meta-analysis on brief alcohol interventions for adolescents and young adults. We examined whether the magnitude and direction of effect sizes were associated with publication type, identification source, language, funding, time lag between intervention and publication, number of reports, journal impact factor, and subsequent citations. Results indicated that effect sizes were larger for studies that had been funded (b = 0.14, 95% confidence interval [CI] [0.04, 0.23]), had a shorter time lag between intervention and publication (b = -0.03, 95% CI [-0.05, -.001]), and were cited more frequently (b = 0.01, 95% CI [+0.00, 0.01]). Studies that were cited more frequently by other authors also had greater odds of reporting positive effects (odds ratio = 1.10, 95% CI [1.02, 1.18]). Results indicated that time lag bias has increased recently: Larger and positive effect sizes were published more quickly in recent years. We found no evidence, however, that the magnitude or direction of effects was associated with location source, language, or journal impact factor. We conclude that dissemination biases may indeed occur in the social and behavioral science literature, as has been consistently documented in the medical literature. As such, primary researchers, journal reviewers, editors, systematic reviewers, and meta-analysts must be cognizant of the causes and consequences of these biases, and commit to engage in ethical research practices that attempt to minimize them. (PsycINFO Database Record

Do evidence-based interventions work when tested in the "real world?" A systematic review and meta-analysis of parent management training for the treatment of child disruptive behavior

Evidence-based interventions are often unavailable in everyday clinical settings. This may partly reflect practitioners' assumptions that research evidence does not reflect "real-world" conditions. To examine this further, we systematically assessed the clinical effectiveness of parent management training (PMT) for the treatment of child disruptive behavior across different real-world practice contexts. We identified 28 relevant randomized controlled trials from a systematic search of electronic bibliographic databases and conducted a meta-analysis of child outcomes across trials. Planned subgroup analyses involved comparisons between studies grouped according to individual real-world practice criteria and total real-world practice criteria scores, reflecting the extent to which PMT was delivered by non-specialist therapists, to a clinic-referred population, in a routine setting, and as part of a routine service. Meta-analysis revealed a significant overall advantage for PMT compared with waitlist control conditions. Subgroup analyses did not demonstrate significant differences in effect size estimates according to the total number of real-world practice criteria met by studies. Moreover, no consistent relationships were found between specific practice criteria and effect size estimates. In conclusion, PMT appears to be an effective treatment for children with disruptive behavior problems. There was no clear evidence that conducting PMT in real-world practice contexts is a deterrent to achieving effective child behavior outcomes, although relative advantage to "usual care" was not directly examined and the power of the analysis was limited as a result of significant heterogeneity. More research is needed to investigate whether this finding is generalizable to other psychological interventions. Suggestions are also made for developing more differentiated criteria to assist with evaluating the specific applicability of research evidence to different care providers.

Making a Case for Treatment Integrity as a Psychosocial Treatment Quality Indicator for Youth Mental Health Care

Measures of treatment integrity are needed to advance clinical research in general and are viewed as particularly relevant for dissemination and implementation research. Although some efforts to develop such measures are underway, a conceptual and methodological framework will help guide these efforts. The purpose of this article is to demonstrate how frameworks adapted from the psychosocial treatment, therapy process, healthcare, and business literatures can be used to address this gap. We propose that components of treatment integrity (i.e., adherence, differentiation, competence, alliance, client involvement) pulled from the treatment technology and process literatures can be used as quality indicators of treatment implementation and thereby guide quality improvement efforts in practice settings. Further, we discuss how treatment integrity indices can be used in feedback systems that utilize benchmarking to expedite the process of translating evidence-based practices to service settings.

Advances in Applying Treatment Integrity Research for Dissemination and Implementation Science: Introduction to Special Issue

This special series focuses upon the ways in which research on treatment integrity, a multidimensional construct including assessment of the content and quality of a psychosocial treatment delivered to a client as well as relational elements, can inform dissemination and implementation science. The five articles for this special series illustrate how treatment integrity concepts and methods can be applied across different levels of the mental health service system to advance dissemination and implementation science. In this introductory article, we provide an overview of treatment integrity research and describe three broad conceptual models that are relevant to the articles in the series. We conclude with a brief description of each of the five articles in the series.

Relationship between camp attendance and self-perceptions in children with chronic health conditions: a meta-analysis

OBJECTIVE

A meta-analysis examined the association between camp attendance and changes in self-perceptions in children with chronic health conditions.

METHODS

Studies using quantitative methods to assess changes in self-perceptions while attending camps designed for children with chronic health conditions were included in analyses. A random-effects model was used, and Cohen's d was used to calculate effect sizes at both post-camp and follow-up. Some potential moderators of effects were examined (i.e., type of measure of self-perceptions, children's chronic health condition, camp components).

RESULTS

31 studies were included in the analyses. Children experienced small, but statistically significant, improvements in self-perceptions at both post-camp (d = .25, 95% CI [.16-.34]) and extended follow-up (d = .15, 95% CI [.05-.26]). This relationship was moderated by type of measure of self-perceptions and child health condition.

CONCLUSIONS

Camp attendance is associated with small improvements in self-perceptions for children with some chronic health conditions.

Improvement in symptoms versus functioning: how do our best treatments measure up?

We examined the effects of redefining standards of evidence for treatments targeting childhood mental health problems by expanding outcomes beyond symptom reduction to include functioning. Over 750 treatment protocols from 435 randomized controlled trials were rated based on empirical evidence. Nearly two-thirds (63.9%) demonstrated at least a minimum level of evidence for reducing symptoms; however, only 18.8% of treatments demonstrated evidence for reducing functional impairment. Of those treatments with empirical support for symptom reduction, the majority did not demonstrate empirical support for improvement in functioning because measures of functioning were not included in the studies in which these treatments were tested. However, even when measures of functioning were included, it was much more difficult for treatments to achieve improvement. Among treatments that achieved improvement in functioning, the most notable were Collaborative Problem Solving for disruptive behavior and Cognitive Behavioral Therapy plus Medication for traumatic stress because they demonstrated no support for symptom reduction but good support for improvement in functioning. Results are discussed within the context of evaluating the standards of evidence for treatments and the opportunity to move towards a multidimensional framework whose utility has the potential to exceed the sum of its parts.

Treatment Effects for Common Outcomes of Child Sexual Abuse: A Current Meta-Analysis

The present meta-analysis examined the effects of psychosocial treatments at reducing deleterious outcomes of sexual abuse. The meta-analysis included a total of 35 published and unpublished studies written in English, focusing on youth under the age of 18, and evaluating the effectiveness of treatments for the most common negative outcomes of sexual abuse: PTSD symptoms, externalizing problems, and internalizing problems. Results revealed medium effect sizes for PTSD symptoms, externalizing problems, and internalizing problems following treatment for sexual abuse. This study also examined the potential moderating effects of treatment (e.g., modality, duration, inclusion of caregiver) and participant (e.g., age, gender, ethnicity) characteristics. Results indicated that longer interventions were associated with greater treatment gains while group and individual treatments were equally effective. These findings shed new light on treatment effectiveness and provide useful information regarding the conditions under which treatment may be most effective. Future directions for research in this area are discussed.

How to do a meta-analysis

Meta-analysis is a statistical tool for estimating the mean and variance of underlying population effects from a collection of empirical studies addressing ostensibly the same research question. Meta-analysis has become an increasing popular and valuable tool in psychological research, and major review articles typically employ these methods. This article describes the process of conducting meta-analysis: selecting articles, developing inclusion criteria, calculating effect sizes, conducting the actual analysis (including information on how to do the analysis on popular computer packages such as IBM SPSS and R) and estimating the effects of publication bias. Guidance is also given on how to write up a meta-analysis.

Meditation therapies for attention-deficit/hyperactivity disorder (ADHD)

BACKGROUND

Attention-deficit/hyperactivity disorder (ADHD) is one of the most common developmental disorders experienced in childhood and can persist into adulthood. The disorder has early onset and is characterized by a combination of overactive, poorly modulated behavior with marked inattention. In the long term it can impair academic performance, vocational success and social-emotional development. Meditation is increasingly used for psychological conditions and could be used as a tool for attentional training in the ADHD population.

OBJECTIVES

To assess the effectiveness of meditation therapies as a treatment for ADHD.

SEARCH STRATEGY

Our extensive search included: CENTRAL, MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, C2-SPECTR, dissertation abstracts, LILACS, Virtual Health Library (VHL) in BIREME, Complementary and Alternative Medicine specific databases, HSTAT, Informit, JST, Thai Psychiatric databases and ISI Proceedings, plus grey literature and trial registries from inception to January 2010.

SELECTION CRITERIA

Randomized controlled trials that investigated the efficacy of meditation therapy in children or adults diagnosed with ADHD.

DATA COLLECTION AND ANALYSIS

Two authors extracted data independently using a pre-designed data extraction form. We contacted study authors for additional information required. We analyzed data using mean difference (MD) to calculate the treatment effect. The results are presented in tables, figures and narrative form.

MAIN RESULTS

Four studies, including 83 participants, are included in this review. Two studies used mantra meditation while the other two used yoga compared with drugs, relaxation training, non-specific exercises and standard treatment control. Design limitations caused high risk of bias across the studies. Only one out of four studies provided data appropriate for analysis. For this study there was no statistically significant difference between the meditation therapy group and the drug therapy group on the teacher rating ADHD scale (MD -2.72, 95% CI -8.49 to 3.05, 15 patients). Likewise, there was no statistically significant difference between the meditation therapy group and the standard therapy group on the teacher rating ADHD scale (MD -0.52, 95% CI -5.88 to 4.84, 17 patients). There was also no statistically significant difference between the meditation therapy group and the standard therapy group in the distraction test (MD -8.34, 95% CI -107.05 to 90.37, 17 patients).

AUTHORS' CONCLUSIONS

As a result of the limited number of included studies, the small sample sizes and the high risk of bias, we are unable to draw any conclusions regarding the effectiveness of meditation therapy for ADHD. The adverse effects of meditation have not been reported. More trials are needed.

Published versus unpublished dissertations in psycho-oncology intervention research

OBJECTIVE

There are conflicting views regarding whether gray literature, including unpublished doctoral dissertations, should be included in systematic reviews and meta-analyses. Although publication status frequently is used as a proxy for study quality, some research suggests that dissertations are often of superior quality to published studies.

METHODS

We examined 107 projects involving doctoral dissertations (42 published, 65 unpublished) that studied psychosocial interventions for cancer patients.

RESULTS

Published dissertations were more likely to be supported by research funding but were not more likely than unpublished dissertations to examine specific types of interventions. Across several indices of methodological quality there were minimal differences. Dissertations with significant findings tended to be more likely to be published than those without significant findings.

CONCLUSIONS

Unpublished dissertations focusing on psychosocial interventions for cancer patients are not necessarily of vastly inferior quality to those that eventually are published. Because doctoral dissertations are easy to access relative to other forms of gray literature, are free from some types of bias, and are reported thoroughly, they merit inclusion in comprehensive literature reviews.

Systematic reviews of psychosocial interventions for autism: an umbrella review

AIM

A wide range of psychosocial interventions for the treatment of individuals with autism-spectrum disorders (ASDs) have been evaluated in systematic reviews. We conducted an umbrella review of systematic reviews of the effectiveness of psychosocial interventions for ASD.

METHOD

Comprehensive searches were conducted in 25 bibliographic databases, relevant journals and reference lists up to May 2007. Studies included were systematic reviews on any psychosocial intervention for individuals with ASDs. Two reviewers independently assessed study relevance and quality.

RESULTS

Thirty systematic reviews were included. The majority of reviews evaluated interventions based on behavioural theory (n=9) or communication-focused (n=7) therapies. Positive intervention outcomes were reported in the majority of the reviews. Methodological quality of the reviews was generally poor.

INTERPRETATION

The reviews reported positive outcomes for many of the interventions, suggesting that some form of treatment is favourable over no treatment. However, there is little evidence for the relative effectiveness of these treatment options. Many of the systematic reviews had methodological weaknesses that make them vulnerable to bias. There is a need for further systematic reviews that adhere to strict scientific methods and for primary studies that make direct comparisons between different treatment options.

Psychosocial interventions for disruptive and aggressive behaviour in children and adolescents: a meta-analysis

BACKGROUND

The effects of psychotherapy in reducing aggressive behaviours in children and adolescents using meta-analysis were estimated.

METHOD

Sixty-five studies were included, covering 4,971 cases. Teacher reported change in aggression, change in social functioning, and changes in parental distress were calculated.

RESULTS

The mean effect size (ES) of change in aggression in studies with untreated controls was 0.62 and in studies without untreated controls the ES was 0.95. In studies with or without untreated controls, the ESs in teacher reported aggression was 0.41 and 0.63, the ESs in changes in social functioning was 0.42 and 0.49, and the ESs in changes in parental distress was 0.39 and 0.47, respectively.

CONCLUSION

Psychosocial treatments aimed at reducing aggressive behaviour have positive effects and additional treatment effects are moderate. In the moderator analysis, studies with untreated controls obtained significantly larger ESs if the sample size was small. Similarly, in studies without untreated controls, studies presenting diagnostic information, and studies with younger children resulted in significantly larger ESs, and studies applying behavioural interventions obtained significantly larger ESs as compared to studies applying family therapeutic interventions. There is still a need to further develop effective outpatient interventions for children being disruptive, and especially for adolescents.

Evidence-Based Psychotherapy for Children and Adolescents: Data from the Present and a Model for the Future

What is the best way to help children cope with behavioural and emotional problems? This question has been a focus of concern - to children and the adults who care for them - across many centuries and in cultures around the world. In this article we examine the current state of efforts to help children by means of an array of non-medical interventions designed to alleviate psychological distress, reduce maladaptive behaviour, and/or increase adaptive behaviour. We refer to these interventions, collectively, as 'psychotherapy'. In the article, we will note some accomplishments of the effort to develop effective interventions through clinical research. Then we will note some concerns about these efforts, and suggest ways to address the concerns through an alternate model of intervention development and testing.

Sensation seeking and alcohol use: a meta-analytic review

A meta-analysis was conducted to investigate the association between sensation seeking and alcohol use. Analysis of 61 studies revealed a small to moderate size, heterogeneous effect (mean weighted r=.263) between alcohol use and sensation seeking total scale scores. Analysis of the four sensation seeking components indicated that disinhibition was most strongly correlated with alcohol use (mean weighted r=.368). Categorical moderator analyses revealed that studies that did, versus did not, analyze covariates yielded smaller effect sizes. Continuous moderator analyses indicated that earlier published studies and higher percentages of male and white Caucasian participants were associated with stronger effect sizes between sensation seeking and alcohol use. Novel statistical methodologies for analyzing the association between sensation seeking and alcohol use were proposed and recommendations for future research were suggested.

Differential effectiveness of behavioral parent-training and cognitive-behavioral therapy for antisocial youth: a meta-analysis

Extended the findings from previous meta-analytic work by comparing the effectiveness of behavioral parent-training (BPT) and cognitive-behavioral therapy (CBT) for youth with antisocial behavior problems. Youth demographic variables were also examined as potential moderators of the effectiveness of these 2 types of interventions. Thirty BPT studies and 41 CBT studies met inclusion criteria for this meta-analysis. The weighted mean effect size (ES) for all interventions was 0.40. Youth age was found to moderate the outcome of the 2 interventions, with BPT having a stronger effect for preschool and school-aged youth and CBT having a stronger effect for adolescents. The results also indicate that there may be systematic differences in the outcomes associated with BPT and CBT when the setting of the intervention is considered, suggesting the need to carefully consider the effect of setting in future research. This study also highlights the need for outcome research dealing with more diverse populations and the better classification of research participants on different developmental trajectories of antisocial behavior.

Challenges in evaluating psychosocial interventions for Autistic Spectrum Disorders

In 2002, the National Institutes of Health sponsored a meeting concerning methodological challenges of research in psychosocial interventions in Autism Spectrum Disorders. This paper provides a summary of the presentations and the discussions that occurred during this meeting. Recommendations to federal and private agencies included the need for randomized clinical trials of comprehensive interventions for autism as the highest, but not the sole priority. Ongoing working groups were proposed to address psychosocial interventions with a focus on relevant statistics, standardized documentation and methods of diagnosis, development of outcome measures, establishment of standards in research; and the need for innovative treatment designs, including application of designs from other research areas to the study of interventions in ASD.

Wirksamkeit von Präventionsmaßnahmen bei Kindern und Jugendlichen:

Zusammenfassung. Theoretischer Hintergrund: In den letzten Jahren finden wir international eine sehr umfangreiche Präventionsforschung. Fragestellung und Methode: Die vorliegende Arbeit fasst die Ergebnisse von 23 quantitativen Überblicksarbeiten und Meta-Analysen zu dieser Thematik zusammen. Ergebnisse: Die meisten Arbeiten lagen zur Prävention externalisierender Störungen, zur Suchtprävention und zur Wirksamkeit von Programmen zur allgemeinen Entwicklungsförderung vor. Als generelle Wirksamkeitsschätzung kann von durchschnittlichen Effekten von einer drittel bis einer halben Standardabweichung ausgegangen werden. Gezielte Präventionsmaßnahmen wiesen in der Regel höhere Effektstärken auf als universelle Strategien. Ebenso ergaben Pilotprojekte und Eigen-Evaluationen eine höhere Wirksamkeit der Programme. Schlussfolgerungen: Präventive Maßnahmen haben signifikante und zum Teil beträchtliche Wirkungen auf die Entwicklung von Kindern und Jugendlichen. Weitere Untersuchungen zu den längerfristigen Effekten, zu Fragen der Programm-Implementation, zu Kosten-Nutzen-Aspekten und vergleichenden Analysen von Präventionsalternativen sowie zur Integration erfolgreicher Präventionsprogramme in die Versorgungspraxis sind jedoch dringend erforderlich.

A meta-analysis of the effects of psychological interventions in pediatric oncology on outcomes of psychological distress and adjustment

OBJECTIVE

To estimate the effectiveness of psychological interventions in pediatric oncology on decreasing psychological distress and increasing psychological adjustment using meta-analytic methods.

METHODS

A meta-analysis was conducted on 12 psychological intervention studies in pediatric oncology using a weighted least squares (WLS) approach and random effect models.

RESULTS

Effect sizes significantly different from zero were found for parent distress (mean = 0.35, 95% CI = 0.20-0.49, n = 7) and parent adjustment (mean = 0.23, 95% CI = 0.07-0.40, n = 5). Effect sizes for child distress, child adjustment, parent-reported child distress, and parent-reported child adjustment were not significantly different from zero.

CONCLUSIONS

Psychological interventions in pediatric oncology show promise in decreasing distress and improving the adjustment of parents of children with cancer but may have minimal effects for child outcomes. Methodological issues of intervention research are discussed.

Promoting and protecting youth mental health through evidence-based prevention and treatment

For decades, empirically tested youth interventions have prevented dysfunction by addressing risk and ameliorated dysfunction through treatment. The authors propose linking prevention and treatment within an integrated model. The model suggests a research agenda: Identify effective programs for a broadened array of problems and disorders, examine ethnicity and culture in relation to intervention adoption and impact, clarify conditions under which programs do and do not work, identify change mechanisms that account for effects, test interventions in real-world contexts, and make tested interventions accessible and effective in community and practice settings. Connecting the science and practice of prevention and treatment will be good for science, for practice, and for children, adolescents, and their families.

Effects of psychotherapy for depression in children and adolescents: a meta-analysis

Serious sequelae of youth depression, plus recent concerns over medication safety, prompt growing interest in the effects of youth psychotherapy. In previous meta-analyses, effect sizes (ESs) have averaged .99, well above conventional standards for a large effect and well above mean ES for other conditions. The authors applied rigorous analytic methods to the largest study sample to date and found a mean ES of .34, not superior but significantly inferior to mean ES for other conditions. Cognitive treatments (e.g., cognitive-behavioral therapy) fared no better than noncognitive approaches. Effects showed both generality (anxiety was reduced) and specificity (externalizing problems were not), plus short- but not long-term holding power. Youth depression treatments appear to produce effects that are significant but modest in their strength, breadth, and durability.