Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research

Scorn Not Its Simplicity: Examining the Effectiveness of Simple Generalist Treatment for Personality Disorders

Treatment guidelines for personality disorders have typically recommended specialized psychotherapeutic interventions. In this review, the author suggests that an intervention's effectiveness may be determined less by the specific method than by therapist competence, team culture, clinical process structure, and institutional context. The author argues that these elements determine variance in effectiveness between and within methods. Whereas initial studies of a specialized treatment may reflect the exceptional competencies of the treatment's developers and early adopters, in daily clinical practice, therapists with an average level of skill may struggle with the theoretical and methodological complexities of these treatments, which can hinder genuine connection with patients. This interference may particularly affect treatment outcomes when therapists encounter the intense emotions and interpersonal hypersensitivity experienced by patients with personality disorders. Most therapists would benefit from a set of simple generalist principles that determine the context for their work and offer a framework for dealing with clinical challenges while enabling them to be true to themselves and use their previously learned competencies. The Guideline-Informed Treatment for Personality Disorders is an enhanced common-factors approach that summarizes the core principles of effective treatment and can be feasibly implemented by most therapists.

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

Assessment of temporal instability in the applied ecology and conservation evidence base

Outcomes of meta-analyses are increasingly used to inform evidence-based decision making in various research fields. However, a number of recent studies have reported rapid temporal changes in magnitude and significance of the reported effects which could make policy-relevant recommendations from meta-analyses to quickly go out of date. We assessed the extent and patterns of temporal trends in magnitude and statistical significance of the cumulative effects in meta-analyses in applied ecology and conservation published between 2004 and 2018. Of the 121 meta-analyses analysed, 93% showed a temporal trend in cumulative effect magnitude or significance with 27% of the datasets exhibiting temporal trends in both. The most common trend was the early study effect when at least one of the first 5 years effect size estimates exhibited more than 50% magnitude difference to the subsequent estimate. The observed temporal trends persisted in majority of datasets once moderators were accounted for. Only 5 datasets showed significant changes in sample size over time which could potentially explain the observed temporal change in the cumulative effects. Year of publication of meta-analysis had no significant effect on presence of temporal trends in cumulative effects. Our results show that temporal changes in magnitude and statistical significance in applied ecology are widespread and represent a serious potential threat to use of meta-analyses for decision-making in conservation and environmental management. We recommend use of cumulative meta-analyses and call for more studies exploring the causes of the temporal effects.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

A meta-epidemiological study on the reported treatment effect of pregabalin in neuropathic pain trials over time

BACKGROUND

Pregabalin is a drug used to treat neuropathic pain, and its use has increased substantially since 2007. Early trials found a strong treatment effect on pain for post-herpetic neuralgia and diabetic neuropathy. However more recent studies have failed to replicate these results.

METHODS

This meta-epidemiological study aimed to assess change in the reported effectiveness of pregabalin in neuropathic pain trials over time, and if a change is present, determine any associated factors.

DATA SOURCES

We performed electronic searches for published trials in Medline, Embase and Cochrane Central Register of Controlled Trials databases; and unpublished trials on ClinicalTrials.gov, the EU Clinical Trials Register, and the Australia New Zealand Clinical Trials Registry with no restrictions.

STUDY SELECTION

We included randomized, placebo-controlled trials of pregabalin for treatment of neuropathic pain in adults.

DATA EXTRACTION AND SYNTHESIS

Two authors independently extracted study data: sample size and mean baseline, end-point and change in pain scores with measures of variance, trial end year, publication year, clinical indication, funding source, country of study, treatment duration, treatment dose, mean age and percentage male.

PRIMARY OUTCOME MEASURE

We defined treatment effect as the mean difference in pain scores between pregabalin and placebo groups at trial end-point and assessed for change over time using a random-effects meta-regression, adjusted for sample size, indication, treatment duration (weeks) and treatment dose.

RESULTS

We included 38 randomized published trials (9038 participants) and found that between 2003 and 2020, the reported treatment effect of pregabalin decreased by 0.4 points (95% CI: 0.3 to 0.6; p<0.001) on an 11-point pain scale per 5-year interval, from 1.3 points (95% CI: 1.0 to 1.5) in trials conducted in 2001-2005, to 0.3 (95% CI: -0.1 to 0.7) in trials conducted in 2016-2020. The reported treatment effect was lower than the minimal clinically important difference (MCID) of 1.7 points across all time periods, doses and most indications and was not found to be associated with study characteristics.

CONCLUSIONS

The reported treatment effect or analgesic efficacy of pregabalin from clinical trials has diminished over time. Clinical recommendations may need to be re-evaluated to account for recent evidence and to consider whether pregabalin therapy is indicated.

Of differing methods, disputed estimates and discordant interpretations: the meta-analytical multiverse of brain volume and IQ associations

Brain size and IQ are positively correlated. However, multiple meta-analyses have led to considerable differences in summary effect estimations, thus failing to provide a plausible effect estimate. Here we aim at resolving this issue by providing the largest meta-analysis and systematic review so far of the brain volume and IQ association (86 studies; 454 effect sizes from k = 194 independent samples; N = 26 000+) in three cognitive ability domains (full-scale, verbal, performance IQ). By means of competing meta-analytical approaches as well as combinatorial and specification curve analyses, we show that most reasonable estimates for the brain size and IQ link yield r-values in the mid-0.20s, with the most extreme specifications yielding rs of 0.10 and 0.37. Summary effects appeared to be somewhat inflated due to selective reporting, and cross-temporally decreasing effect sizes indicated a confounding decline effect, with three quarters of the summary effect estimations according to any reasonable specification not exceeding r = 0.26, thus contrasting effect sizes were observed in some prior related, but individual, meta-analytical specifications. Brain size and IQ associations yielded r = 0.24, with the strongest effects observed for more g-loaded tests and in healthy samples that generalize across participant sex and age bands.

Decline effects are rare in ecology

The scientific evidence base on any given topic changes over time as more studies are published. Currently, there is widespread concern about non-random, directional changes over time in the scientific evidence base associated with many topics. In particular, if studies finding large effects (e.g., large differences between treatment and control means) tend to get published quickly, while small effects tend to get published slowly, the net result will be a decrease over time in the estimated magnitude of the mean effect size, known as a "decline effect". If decline effects are common, then the published scientific literature will provide a biased and misleading guide to management decisions, and to the allocation of future research effort. We compiled data from 466 meta-analyses in ecology to look for evidence of decline effects. We found that decline effects are rare. Only ~5% of ecological meta-analyses truly exhibit a directional change in mean effect size over time arising for some reason other than random chance, usually but not always in the direction of decline. Most apparent directional changes in mean effect size over time are attributable to regression to the mean, consistent with primary studies being published in random order with respect to the effect sizes they report. Our results are good news: decline effects are the exception to the rule in ecology. Identifying and rectifying rare cases of true decline effects remains an important task, but ecologists should not overgeneralize from anecdotal reports of decline effects.

Meta-analyzing intelligence and religiosity associations: Evidence from the multiverse

Over the past century, a remarkable body of research about the relationship of intelligence and religiosity has accumulated. So far, the majority of studies that investigated this relationship showed a negative correlation, indicating lower cognitive abilities of individuals reporting stronger religious beliefs. Although the effect direction has been observed to be largely consistent across studies, the reported effect strength varied substantially across studies. Several potentially moderating variables such as different intelligence and religiosity assessment methods, educational status of samples, and participant sex have been proposed as likely candidates for explaining systematic differences in effect strengths. However, the effects of these moderators are to date unclear. Consequently, we focused in investigating effects of these moderating variables on the intelligence and religiosity link in an update of prior meta-analytical investigations in n = 89 (k = 105; N = 201,457) studies. Random-effects analyses showed a small but robust negative association between intelligence and religiosity r = -.14 (p < .001; 95% CI [-.17, -.12]). Effects were stronger for (i) psychometric intelligence tests than for proxy measures such as grade point averages and (ii) general population and college samples than pre-college samples. Moreover, we provide evidence from combinatorial, multiverse, and specification curve analyses that further corroborates the robustness of the investigated association. Out of 192 reasonable specifications all 135 (70.4%) significant summary effects were negative. In all, our results show small but robust negative associations between religiosity and intelligence that are differentiated in strength but generalize in terms of direction over moderating variables.

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson's correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.