The case for preregistering all region of interest (ROI) analyses in neuroimaging research

In neuroimaging studies, small sample sizes and the resultant reduced statistical power to detect effects that are not large, combined with inadequate analytic choices, concur to produce inflated or false-positive findings. To mitigate these issues, researchers often restrict analyses to specific brain areas, using the region of interest (ROI) approach. Crucially, ROI analysis assumes the a priori justified definition of the target region. Nonetheless, reports often lack details about where in the timeline, ranging from study conception to the data analysis and interpretation of findings, were ROIs selected. Frequently, the rationale for ROI selection is vague or inadequately founded on the existing literature. These shortcomings have important implications for ROI-based studies, augmenting the risk that observed effects are inflated or even false positives. Tools like preregistration and registered reports could address this problem, ensuring the validity of ROI-based studies. The benefits could be enhanced by additional practices such as selection of ROIs using quantitative methods (i.e., meta-analysis) and the sharing of whole-brain unthresholded maps of effect size, as well as of binary ROIs, in publicly accessible repositories.

Psychological interventions using virtual reality for pain associated with medical procedures: a systematic review and meta-analysis

BACKGROUND

Virtual reality (VR) may enhance the effectiveness of psychological interventions for acute pain. We conducted a meta-analysis to assess the efficacy and safety of VR-based interventions for pain associated with medical procedures.

METHODS

We searched PubMed, EMBASE, the Cochrane Library, and PsycINFO until June 17th 2018. We identified randomized controlled trials (RCTs), comparing VR-based psychological interventions to usual care, for pain intensity (primary outcome) or affective and cognitive components of pain (secondary outcomes), assessed real-time or retrospectively. Two independent reviewers performed study selection and data extraction. Risk of bias was independently evaluated by three raters using the revised Cochrane Collaboration tool. A random-effects model using the Paule and Mandel estimator was used for pooling effect sizes.

RESULTS

27 RCTs (1452 patients) provided enough data for meta-analysis. Compared to usual care, VR-based interventions reduced pain intensity both real-time (9 RCTs, Hedges' g = 0.95, 95% CI 0.32-1.57) and retrospectively (22 RCTs, g = 0.87, 95% CI 0.54-1.21). Results were similar for cognitive (8 RCTs, g = 0.82, 95% CI 0.39-1.26) and affective pain components (14 RCTs, g = 0.55, 95% CI 0.34-0.77). There was marked heterogeneity, which remained similarly high in sensitivity analyses. Across domains, few trials were rated as low risk of bias and there was evidence of publication bias. Adverse events were rare.

CONCLUSIONS

Though VR-based interventions reduced pain for patients undergoing medical procedures, inferring clinical effectiveness is precluded by the predominance of small trials, with substantial risk of bias, and by incomplete reporting.

Efficacy of cognitive bias modification interventions in anxiety and depressive disorders: a systematic review and network meta-analysis

BACKGROUND

Cognitive bias modification (CBM) therapies, including attention bias modification, interpretation bias modification, or approach and avoidance training, are prototypical examples of mechanistically derived treatments, but their effectiveness is contentious. We aimed to assess the relative effectiveness of various CBM interventions for anxious and depressive symptomatology.

METHODS

For this systematic review and network meta-analysis, we searched PubMed, PsycINFO, Embase, and Cochrane Central Register from database inception up until Feb 7, 2020. We included randomised controlled trials of CBM versus control conditions or other forms of CBM for adults aged 18 years and older with clinical or subclinical anxiety or depression measured with a diagnostic interview or a validated clinical scale. We excluded studies comparing CBM with a non-CBM active intervention. Two researchers independently selected studies and evaluated risk of bias with the Cochrane Collaboration tool. Primary outcomes encompassed anxiety and depressive symptoms measured with validated clinical scales. We computed standardised mean differences (SMDs) with a restricted maximum likelihood random effects model. This study is registered with PROSPERO, CRD42018086113.

FINDINGS

From 2125 records we selected 85 trials, 65 (n=3897) on anxiety and 20 (n=1116) on depression. In a well connected network of anxiety trials, interpretation bias modification outperformed waitlist (SMD -0·55, 95% CI -0·91 to -0·19) and sham training (SMD -0·30, -0·50 to -0·10) for the primary outcome. Attention bias modification showed benefits only in post-hoc sensitivity analyses excluding post-traumatic stress disorder trials. Prediction intervals for all findings were large, including an SMD of 0. Networks of depression trials displayed evidence of inconsistency. Only four randomised controlled trials had low risk of bias on all six domains assessed.

INTERPRETATION

CBM interventions showed consistent but small benefits; however heterogeneity and risk of bias undermine the reliability of these findings. Larger, definitive trials for interpretation bias modification for anxiety might be warranted, but insufficient evidence precludes conclusions for depression.

FUNDING

Romanian Ministry of Research and Innovation, The National Council for Scientific Research-The Executive Agency for Higher Education, Research, Development and Innovation Funding.

Eye movement desensitization and reprocessing for mental health problems: a systematic review and meta-analysis

There is no comprehensive meta-analysis of randomised trials examining the effects of Eye Movement Desensitization and Reprocessing (EMDR) on post-traumatic stress disorder (PTSD) and no systematic review at all of the effects of EMDR on other mental health problems. We conducted a systematic review and meta-analysis of 76 trials. Most trials examined the effects on PTSD (62%). The effect size of EMDR compared to control conditions was g = 0.93 (95% CI: 0.67-.18), with high heterogeneity (I²= 72%). Only four of 27 studies had low risk of bias, and there were indications for publication bias. EMDR was more effective than other therapies (g = 0.36; 95% CI: 0.14-0.57), but not in studies with low risk of bias. Significant results were also found for EMDR in phobias and test anxiety, but the number of studies was small and risk of bias was high. EMDR was examined in several other mental health problems, but for none of these problems, sufficient studies were available to pool outcomes. EMDR may be effective in the treatment of PTSD in the short term, but the quality of studies is too low to draw definite conclusions. There is not enough evidence to advise it for the use in other mental health problems.

Open science practices in clinical psychology journals: An audit study

We conducted an audit of 60 clinical psychology journals, covering the first 2 quartiles by impact factor on Web of Science. We evaluated editorial policies in 5 domains crucial to reproducibility and transparency (prospective registration, data sharing, preprints, endorsement of reporting guidelines and conflict of interest [COI] disclosure). We examined implementation in a randomly selected cross-sectional sample of 201 articles published in 2017 in the "best practice" journals, defined as having explicit supportive policies in 4 out of 5 domains. Our findings showed that 15 journals cited prospective registration, 40 data sharing, 15 explicitly permitted preprints, 28 endorsed reporting guidelines, and 52 had mandatory policies for COI disclosure. Except for COI disclosure, few policies were mandatory: registration in 15 journals, data sharing in 1, and reporting guidelines for randomized trials in 18 and for meta-analyses in 15. Seventeen journals were identified as "best practice." An analysis of recent articles showed extremely low compliance for prospective registration (3% articles) and data sharing (2%). One preprint could be identified. Reporting guidelines were endorsed in 19% of the articles, though for most articles this domain was rated as nonapplicable. Only half of the articles included a COI disclosure. Desired open science policies should become clear and mandatory, and their enforcement streamlined by reducing the multiplicity of guidelines and templates. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Assessing scientists for hiring, promotion, and tenure

Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.

Component studies of psychological treatments of adult depression: A systematic review and meta-analysis

OBJECTIVES

A recent report from the US Institute of Medicine indicated that identifying core elements of psychosocial interventions is a key step in successfully bringing evidence-based psychosocial interventions into clinical practice. Component studies have the best design to examine these core elements. Earlier reviews resulted in heterogeneous sets of studies and probably missed many studies.

METHODS

We conducted a comprehensive search of component studies on psychotherapies for adult depression and included 16 studies with 22 comparisons.

RESULTS

Fifteen components were examined of which four were examined in more than one comparison. The pooled difference between the full treatments and treatments with one component removed was g = 0.21 (95% CI: 0.03∼0.39). One study had sufficient statistical power to detect a small effect size and found that adding emotion regulation skills increased the effects of CBT. None of the other studies had enough power to detect an effect size smaller than g = 0.55. Only one study had low risk of bias.

CONCLUSIONS

The currently available component studies do not have the statistical power nor the quality to draw any meaningful conclusion about key ingredients of psychotherapies for adult depression.

Invited commentary on … Confusing procedures with process in cognitive bias modification research†

The notion that cognitive bias modification should be appraised exclusively on the basis of trials where its postulated mechanisms were successfully changed starkly contradicts the standards of evidence-based psychotherapy. In the laboratory or as a treatment, cognitive bias modification cannot continue to eschew the rigorous scrutiny applied to other interventions.

The effects of cognitive behavioral therapy are not systematically falling: A revision of Johnsen and Friborg (2015)

In a meta-analysis, Johnsen and Friborg (2015) reported a significant negative relationship between publication year and the effect sizes (ESs) of cognitive-behavioral therapy (CBT) for depressive disorders, suggesting its effectiveness was falling. We identified a series of methodological and conceptual caveats and consequently redid the meta-analysis. We used the same inclusion criteria, but only included randomized controlled trials and searched for additional eligible trials. We computed both within-group and between-group ESs for the CBT arm for the Beck Depression Inventory (BDI) and the Hamilton Rating Scale for Depression (HRSD). We assessed risk of bias, sample size, type of control group, and the study's country of origin and conducted subgroup, single, and multiple meta-regression analyses including publication year and other moderators. We identified 30 additional eligible trials. Within-group ESs presented huge heterogeneity estimates (I2 around 90%). Year of publication was significant in some single meta-regression analyses on the BDI, but not significant in others, in most analyses on the HRSD, and in any of the analyses on between-group ESs. Multiple regression models indicated that either year was not significantly related or that both year and country were significantly related to outcomes, with a temporal trend present solely in US studies. Year of publication does not appear to be a reliable and independent moderator of the effectiveness of CBT for depression. The linear "fall" reported by Johnsen and Friborg (2015) is most likely a spurious finding. (PsycINFO Database Record

Efficacy of Psychotherapies for Borderline Personality Disorder: A Systematic Review and Meta-analysis

IMPORTANCE

Borderline personality disorder (BPD) is a debilitating condition, but several psychotherapies are considered effective.

OBJECTIVE

To conduct an updated systematic review and meta-analysis of randomized clinical trials to assess the efficacy of psychotherapies for BPD populations.

DATA SOURCES

Search terms were combined for borderline personality and randomized trials in PubMed, PsycINFO, EMBASE, and the Cochrane Central Register of Controlled Trials (from database inception to November 2015), as well as the reference lists of earlier meta-analyses.

STUDY SELECTION

Included were randomized clinical trials of adults with diagnosed BPD randomized to psychotherapy exclusively or to a control intervention. Study selection differentiated stand-alone designs (in which an independent psychotherapy was compared with control interventions) from add-on designs (in which an experimental intervention added to usual treatment was compared with usual treatment alone).

DATA EXTRACTION AND SYNTHESIS

Data extraction coded characteristics of trials, participants, and interventions and assessed risk of bias using 4 domains of the Cochrane Collaboration Risk of Bias tool (independent extraction by 2 assessors). Outcomes were pooled using a random-effects model. Subgroup and meta-regression analyses were conducted.

MAIN OUTCOMES AND MEASURES

Standardized mean differences (Hedges g) were calculated using all outcomes reported in the trials for borderline symptoms, self-harm, suicide, health service use, and general psychopathology at posttest and follow-up. Differential treatment retention at posttest was analyzed, reporting odds ratios.

RESULTS

Thirty-three trials (2256 participants) were included. For borderline-relevant outcomes combined (symptoms, self-harm, and suicide) at posttest, the investigated psychotherapies were moderately more effective than control interventions in stand-alone designs (g = 0.32; 95% CI, 0.14-0.51) and add-on designs (g = 0.40; 95% CI, 0.15-0.65). Results were similar for other outcomes, including stand-alone designs: self-harm (g = 0.32; 95% CI, 0.09-0.54), suicide (g = 0.44; 95% CI, 0.15-0.74), health service use (g = 0.40; 95% CI, 0.22-0.58), and general psychopathology (g = 0.32; 95% CI, 0.09-0.55), with no differences between design types. There were no significant differences in the odds ratios for treatment retention (1.32; 95% CI, 0.87-2.00 for stand-alone designs and 1.01; 95% CI, 0.55-1.87 for add-on designs). Thirteen trials reported borderline-relevant outcomes at follow-up (g = 0.45; 95% CI, 0.15-0.75). Dialectical behavior therapy (g = 0.34; 95% CI, 0.15-0.53) and psychodynamic approaches (g = 0.41; 95% CI, 0.12-0.69) were the only types of psychotherapies more effective than control interventions. Risk of bias was a significant moderator in subgroup and meta-regression analyses (slope β = -0.16; 95% CI, -0.29 to -0.03; P = .02). Publication bias was persistent, particularly for follow-up.

CONCLUSIONS AND RELEVANCE

Psychotherapies, most notably dialectical behavior therapy and psychodynamic approaches, are effective for borderline symptoms and related problems. Nonetheless, effects are small, inflated by risk of bias and publication bias, and particularly unstable at follow-up.

Sponsorship bias in the comparative efficacy of psychotherapy and pharmacotherapy for adult depression: meta-analysis

BACKGROUND

Sponsorship bias has never been investigated for non-pharmacological treatments like psychotherapy.

AIMS

We examined industry funding and author financial conflict of interest (COI) in randomised controlled trials directly comparing psychotherapy and pharmacotherapy in depression.

METHOD

We conducted a meta-analysis with subgroup comparisons for industry v. non-industry-funded trials, and respectively for trial reports with author financial COI v. those without.

RESULTS

In total, 45 studies were included. In most analyses, pharmacotherapy consistently showed significant effectiveness over psychotherapy, g = -0.11 (95% CI -0.21 to -0.02) in industry-funded trials. Differences between industry and non-industry-funded trials were significant, a result only partly confirmed in sensitivity analyses. We identified five instances where authors of the original article had not reported financial COI.

CONCLUSIONS

Industry-funded trials for depression appear to subtly favour pharmacotherapy over psychotherapy. Disclosure of all financial ties with the pharmaceutical industry should be encouraged.

The effects of cognitive and behavioural therapies for anxiety disorders on depression: a meta-analysis

BACKGROUND

The effects of cognitive behavioural therapy of anxiety disorders on depression has been examined in previous meta-analyses, suggesting that these treatments have considerable effects on depression. In the current meta-analysis we examined whether the effects of treatments of anxiety disorders on depression differ across generalized anxiety disorder (GAD), social anxiety disorder (SAD) and panic disorder (PD). We also compared the effects of these treatments with the effects of cognitive and behavioural therapies of major depression (MDD).

METHOD

We searched PubMed, PsycINFO, EMBASE and the Cochrane database, and included 47 trials on anxiety disorders and 34 trials on MDD.

RESULTS

Baseline depression severity was somewhat lower in anxiety disorders than in MDD, but still mild to moderate in most studies. Baseline severity differed across the three anxiety disorders. The effect sizes found for treatment of the anxiety disorders ranged from g = 0.47 for PD, g = 0.68 for GAD and g = 0.69 for SAD. Differences between these effect sizes and those found in the treatment of MDD (g = 0.81) were not significant in most analyses and we found few indications that the effects differed across anxiety disorders. We did find that within-group effect sizes resulted in significantly (p < 0.001) larger effect sizes for depression (g = 1.50) than anxiety disorders (g = 0.73-0.91). Risk of bias was considerable in the majority of studies.

CONCLUSIONS

Patients participating in trials of cognitive behavioural therapy for anxiety disorders have high levels of depression. These treatments have considerable effects on depression, and these effects are comparable to those of treatment of primary MDD.

Personalized Psychotherapy for Adult Depression: A Meta-Analytic Review

Personalized medicine is aimed at identifying which characteristics of an individual predict the outcome of a specific treatment, in order to get a better match between the individual and the treatment received. We conducted a systematic review and meta-analysis of randomized trials comparing two psychotherapies directly in a group of depressed patients with a specific characteristic. We focused on the six most examined types of psychotherapy for adult depression. Our searches resulted in 41 studies with 2,741 patients who met inclusion criteria. These 41 studies examined 27 specific characteristics of patients. Power calculations indicated that we would need 4 studies for each characteristic to find a clinically relevant effect size set at g = 0.50 and 16 studies for an effect size of 0.24. Only 3 patient characteristics were found to have sufficient power and to significantly moderate treatment outcomes. Cognitive-behavioral therapy was found to be more effective than other therapies in older adults (g = 0.29), in patients with comorbid addictive disorders (g = 0.31), and in university students (g = 0.46). Risk of bias was considerable in most of the included studies. It was estimated that it will take another 326 years to have sufficient statistical power for showing an effect size of g = 0.50 of the 27 characteristics, and 1,372 years to show an effect size of 0.24. Although several dozens of studies have compared the effects of psychotherapies in specific target groups, we will need to develop more powerful alternatives to comparative outcome studies in order to identify personalized treatments for depression.