Estimation of an overall standardized mean difference in random‐effects meta‐analysis if the distribution of random effects departs from normal

Stage 2 Registered Report: Anomalous perception in a Ganzfeld condition - A meta-analysis of more than 40 years investigation

This meta-analysis is an investigation into anomalous perception (i.e., conscious identification of information without any conventional sensorial means). The technique used for eliciting an effect is the Ganzfeld condition (a form of sensory homogenization that eliminates distracting peripheral noise). The database consists of studies published between January 1974 and December 2020 inclusive. The overall effect size estimated both with a frequentist and a Bayesian random-effect model, were in close agreement yielding an effect size of approximately .08 (.04 -.12). This result passed four publication bias tests and seems not contaminated by questionable research practices. Trend analysis carried out with a cumulative meta-analysis and a meta-regression model with year of publication as a covariate, did not indicate sign of decline of this effect size. The moderators' analyses show that the selected participants' effect size was almost three-times that obtained by non-selected participants and that tasks that simulate telepathic communication show a two-fold effect size for tasks requiring the participants to guess a target. The Stage 1 Registered Report can be accessed here: https://doi.org/10.12688/f1000research.24868.3.

Accuracy and precision of fixed and random effects in meta-analyses of randomized control trials for continuous outcomes

Meta-analyses of treatment effects in randomized control trials are often faced with the problem of missing information required to calculate effect sizes and their sampling variances. Particularly, correlations between pre- and posttest scores are frequently not available. As an ad-hoc solution, researchers impute a constant value for the missing correlation. As an alternative, we propose adopting a multivariate meta-regression approach that models independent group effect sizes and accounts for the dependency structure using robust variance estimation or three-level modeling. A comprehensive simulation study mimicking realistic conditions of meta-analyses in clinical and educational psychology suggested that imputing a fixed correlation 0.8 or adopting a multivariate meta-regression with robust variance estimation work well for estimating the pooled effect but lead to slightly distorted between-study heterogeneity estimates. In contrast, three-level meta-regressions resulted in largely unbiased fixed effects but more inconsistent prediction intervals. Based on these results recommendations for meta-analytic practice and future meta-analytic developments are provided.

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

BACKGROUND

Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses.

METHODS

In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality.

RESULTS

Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate.

CONCLUSIONS

The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Heterogeneity estimation in meta-analysis of standardized mean differences when the distribution of random effects departs from normal: A Monte Carlo simulation study

BACKGROUND

Advantages of meta-analysis depend on the assumptions underlying the statistical procedures used being met. One of the main assumptions that is usually taken for granted is the normality underlying the population of true effects in a random-effects model, even though the available evidence suggests that this assumption is often not met. This paper examines how 21 frequentist and 24 Bayesian methods, including several novel procedures, for computing a point estimate of the heterogeneity parameter ([Formula: see text]) perform when the distribution of random effects departs from normality compared to normal scenarios in meta-analysis of standardized mean differences.

METHODS

A Monte Carlo simulation was carried out using the R software, generating data for meta-analyses using the standardized mean difference. The simulation factors were the number and average sample size of primary studies, the amount of heterogeneity, as well as the shape of the random-effects distribution. The point estimators were compared in terms of absolute bias and variance, although results regarding mean squared error were also discussed.

RESULTS

Although not all the estimators were affected to the same extent, there was a general tendency to obtain lower and more variable [Formula: see text] estimates as the random-effects distribution departed from normality. However, the estimators ranking in terms of their absolute bias and variance did not change: Those estimators that obtained lower bias also showed greater variance. Finally, a large number and sample size of primary studies acted as a bias-protective factor against a lack of normality for several procedures, whereas only a high number of studies was a variance-protective factor for most of the estimators analyzed.

CONCLUSIONS

Although the estimation and inference of the combined effect have proven to be sufficiently robust, our work highlights the role that the deviation from normality may be playing in the meta-analytic conclusions from the simulation results and the numerical examples included in this work. With the aim to exercise caution in the interpretation of the results obtained from random-effects models, the tau2() R function is made available for obtaining the range of [Formula: see text] values computed from the 45 estimators analyzed in this work, as well as to assess how the pooled effect, its confidence and prediction intervals vary according to the estimator chosen.

On the Q statistic with constant weights for standardized mean difference

Cochran's Q statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value is also used in several popular estimators of the between-study variance, τ 2 . Those applications generally have not considered the implications of its use of estimated variances in the inverse-variance weights. Importantly, those weights make approximating the distribution of Q (more explicitly, Q IV ) rather complicated. As an alternative, we investigate a new Q statistic, Q F , whose constant weights use only the studies' effective sample sizes. For the standardized mean difference as the measure of effect, we study, by simulation, approximations to distributions of Q IV and Q F , as the basis for tests of heterogeneity and for new point and interval estimators of τ 2 . These include new DerSimonian-Kacker-type moment estimators based on the first moment of Q F , and novel median-unbiased estimators. The results show that: an approximation based on an algorithm of Farebrother follows both the null and the alternative distributions of Q F reasonably well, whereas the usual chi-squared approximation for the null distribution of Q IV and the Biggerstaff-Jackson approximation to its alternative distribution are poor; in estimating τ 2 , our moment estimator based on Q F is almost unbiased, the Mandel - Paule estimator has some negative bias in some situations, and the DerSimonian-Laird and restricted maximum likelihood estimators have considerable negative bias; and all 95% interval estimators have coverage that is too high when τ 2 = 0 , but otherwise the Q-profile interval performs very well.

The association of telomere length with substance use disorders: a systematic review and meta-analysis of observational studies

BACKGROUND AND AIMS

Several recent studies have investigated the relationship between telomere length and substance use disorders with inconsistent results. We aimed to assess this association and to identify moderators of the relationship.

METHODS

Systematic review and meta-analysis. Selection criteria were observational studies reporting telomere length in people with a substance use disorder compared with a control group. Studies focused solely on nicotine addiction, employing other study designs, and non-human studies were excluded. Study selection and data extraction were independently conducted by two researchers following a standardized protocol and included studies until December 2019. Standardized mean differences were used as the effect size index [d; 95% confidence interval (CI)] and random-effects models were used for the meta-analysis. Cochran's Q-statistic, I² index, visual inspection of the forest plot and a 95% prediction interval were applied to verify study heterogeneity. Subgroup analyses and meta-regressions were conducted to explore heterogeneity. Small study effects were examined using the 'funnel plot', the Egger test, Duval & Tweedie's trim-and-fill method and the precision-effect test-precision-effect estimate with standard error (PET-PEESE) method. The risk of bias and the quality of evidence were assessed.

RESULTS

Ten studies (12 analysis units with 2671 cases and 4532 controls) met the selection criteria. An overall effect size of moderate magnitude was found (d₊  = -0.63; 95% CI = -1.00 and -0.26; P = 0.0008). A potential small study effect was detected, as well as large heterogeneity between studies (Q-statistic P < 0.001, I²  = 97.3%). Selection of controls, reporting laboratory quality control procedures and total sample size significantly affected the effect size. The quality of the evidence was very low, based on risk of bias analysis and the grading of recommendations assessment, development and evaluation (GRADE) system.

CONCLUSIONS

People with substance use disorders appear to have shorter telomere length than controls; however, this finding should be interpreted with caution due to the poor quality of the evidence.

Meta-analysis using flexible random-effects distribution models

BACKGROUND

In meta-analysis, the normal distribution assumption has been adopted in most systematic reviews of random-effects distribution models due to its computational and conceptual simplicity. However, this restrictive model assumption is possibly unsuitable and might have serious influences in practices.

METHODS

We provide two examples of real-world evidence that clearly show that the normal distribution assumption is explicitly unsuitable. We propose new random-effects meta-analysis methods using five flexible random-effects distribution models that can flexibly regulate skewness, kurtosis and tailweight: skew normal distribution, skew t-distribution, asymmetric Subbotin distribution, Jones-Faddy distribution, and sinh-arcsinh distribution. We also developed a statistical package, flexmeta, that can easily perform these methods.

RESULTS

Using the flexible random-effects distribution models, the results of the two meta-analyses were markedly altered, potentially influencing the overall conclusions of these systematic reviews.

CONCLUSIONS

The restrictive normal distribution assumption in the random-effects model can yield misleading conclusions. The proposed flexible methods can provide more precise conclusions in systematic reviews.

Estimation in meta-analyses of response ratios

BACKGROUND

For outcomes that studies report as the means in the treatment and control groups, some medical applications and nearly half of meta-analyses in ecology express the effect as the ratio of means (RoM), also called the response ratio (RR), analyzed in the logarithmic scale as the log-response-ratio, LRR.

METHODS

In random-effects meta-analysis of LRR, with normal and lognormal data, we studied the performance of estimators of the between-study variance, τ², (measured by bias and coverage) in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect in the log scale, λ. We obtained additional empirical evidence from two examples.

RESULTS

The results of our extensive simulations showed several challenges in using LRR as an effect measure. Point estimators of τ² had considerable bias or were unreliable, and interval estimators of τ² seldom had the intended 95% coverage for small to moderate-sized samples (n<40). Results for estimating λ differed between lognormal and normal data.

CONCLUSIONS

For lognormal data, we can recommend only SSW, a weighted average in which a study's weight is proportional to its effective sample size, (when n≥40) and its companion interval (when n≥10). Normal data posed greater challenges. When the means were far enough from 0 (more than one standard deviation, 4 in our simulations), SSW was practically unbiased, and its companion interval was the only option.

The association of HTR2A polymorphisms with obsessive-compulsive disorder and its subtypes: A meta-analysis

BACKGROUND

Genetic risk factors that contribute to obsessive-compulsive disorder (OCD) have yet to be elucidated. Historically, serotonergic dysfunction has been implicated. Evidence from the literature points towards the serotonin receptor 2A gene (HTR2A) as a primary candidate. Our meta-analysis investigated whether polymorphisms in HTR2A are associated with OCD or its subtypes, based on sex and age of onset.

METHODS

Studies employing case-control or family-based designs were systematically searched, and those meeting eligibility underwent quality assessment, resulting in 18 studies. A random-effects meta-analysis using standard inverse-variance weighting to compute odds ratio (OR) was conducted. To examine sensitivity, results were also obtained using a more conservative statistical method.

RESULTS

Three HTR2A variants were identified: T102C, G-1438A, and C516T. T102C and G-1438A were analyzed together due to strong linkage disequilibrium, where the 102T allele co-occurs with -1438A allele. Results reported as OR [95%CI] showed that the T/A allele were significantly associated with OCD, 1.14 [1.01, 1.29]. After stratification, results remained significant for females, 1.20 [1.00, 1.45], and early-onset OCD, 1.27 [1.02, 1.58], but not males, 1.06 [0.91, 1.23]. No associations were found for late-onset OCD, 0.98 [0.70, 1.37], or C516T, 1.22 [0.14, 10.37], but conclusions cannot be drawn from two studies.

LIMITATIONS

Associations no longer reached significance with the conservative statistical approach. HTR2A alone cannot explain OCD complexity and limited samples reporting genetic data according to subtypes.

CONCLUSIONS

These results suggest a possible association of HTR2A polymorphisms with OCD, but further investigations considering sex and age of onset with larger samples is needed.

Stage 1 Registered Report: Anomalous perception in a Ganzfeld condition - A meta-analysis of more than 40 years investigation

This meta-analysis is an investigation into anomalous perception (i.e., conscious identification of information without any conventional sensorial means). The technique used for eliciting an effect is the ganzfeld condition (a form of sensory homogenization that eliminates distracting peripheral noise). The database consists of peer-reviewed studies published between January 1974 and June 2020 inclusive. The overall effect size will be estimated using a frequentist and a Bayesian random-effect model. Moderators analyses will be used to examine the influence of level of experience of participants, the type of task and the peer-review level. Publication bias will be estimated by using four different tests. Trend analysis will be conducted with a cumulative meta-analysis and a meta-regression model with Year of publication as covariate.

Stage 1 Registered Report: Anomalous perception in a Ganzfeld condition - A meta-analysis of more than 40 years investigation

This meta-analysis is an investigation into anomalous perception (i.e., conscious identification of information without any conventional sensorial means). The technique used for eliciting an effect is the ganzfeld condition (a form of sensory homogenization that eliminates distracting peripheral noise). The database consists of peer-reviewed studies published between January 1974 and June 2020 inclusive. The overall effect size will be estimated using a frequentist and a Bayesian random-effect model. Moderators analyses will be used to examine the influence of level of experience of participants, the type of task and the peer-review level. Publication bias will be estimated by using four different tests. Trend analysis will be conducted with a cumulative meta-analysis and a meta-regression model with Year of publication as covariate.

Stage 1 Registered Report: Anomalous perception in a Ganzfeld condition - A meta-analysis of more than 40 years investigation

This meta-analysis is an investigation into anomalous perception (i.e., conscious identification of information without any conventional sensorial means). The technique used for eliciting an effect is the ganzfeld condition (a form of sensory homogenization that eliminates distracting peripheral noise). The database consists of peer-reviewed studies published between January 1974 and June 2020 inclusive. The overall effect size will be estimated using a frequentist model and a Bayesian random model. Moderator analysis will be used to examine the influence of level of experience of participants and the type of task. Publication bias will be estimated by using three different tests. Trend analysis will be conducted on the cumulative database.

The association of telomere length with substance use disorders: systematic review and meta-analysis protocol

BACKGROUND

The present protocol was designed for a systematic review and meta-analysis aimed at determining the association of telomere length with substance use disorders with the exclusion of nicotine addiction, and to identify potential moderators of the effect of telomere length. Such methodological information may provide guidance to improve the quality of future research on this important topic.

METHODS

Potential studies will be identified through electronic databases (PubMed/MEDLINE, EMBASE, PsycINFO, and Web of Science) up from inception onwards. The inclusion criteria will include published or unpublished observational studies (cohort, case-control, and cross-sectional studies) reporting telomere length in adult patients with substance use disorder compared with a control group. Non-human studies or other study designs such as reviews, case-only, family-based, and/or population studies with only healthy participants will be excluded, as well as those focused solely on nicotine addiction. The main outcome will be telomere length in adults with substance use disorder (primary) and, specifically, in those with alcohol use disorder (secondary). Two investigators will independently evaluate the preselected studies for possible inclusion and will extract data following a standardized protocol. Disagreements will be resolved by consensus. The risk of bias of all included studies will be assessed using the Newcastle-Ottawa Quality Assessment Scale for non-randomized studies. Data will be converted into standardized mean differences as effect size index, and random-effects models will be used for the meta-analysis. Cochran's Q statistic, I² index, and visual inspection of the forest plot will be used to verify study heterogeneity. Subgroup analyses and meta-regressions will be conducted to ascertain heterogeneity. Several sensitivity analyses will be conducted to address the influence of potential confounding factors. Publication bias will be examined using the "funnel plot" method with Duval and Tweedie's trim-and-fill method and Egger test.

DISCUSSION

This systematic review will assess the association of telomere length with substance use disorders aside from nicotine addiction.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number CRD42019119785.