Interpreting and evaluating meta-analysis

An anchored experimental design and meta-analysis approach to address batch effects in large-scale metabolomics

Untargeted metabolomics studies are unbiased but identifying the same feature across studies is complicated by environmental variation, batch effects, and instrument variability. Ideally, several studies that assay the same set of metabolic features would be used to select recurring features to pursue for identification. Here, we developed an anchored experimental design. This generalizable approach enabled us to integrate three genetic studies consisting of 14 test strains of Caenorhabditis elegans prior to the compound identification process. An anchor strain, PD1074, was included in every sample collection, resulting in a large set of biological replicates of a genetically identical strain that anchored each study. This enables us to estimate treatment effects within each batch and apply straightforward meta-analytic approaches to combine treatment effects across batches without the need for estimation of batch effects and complex normalization strategies. We collected 104 test samples for three genetic studies across six batches to produce five analytical datasets from two complementary technologies commonly used in untargeted metabolomics. Here, we use the model system C. elegans to demonstrate that an augmented design combined with experimental blocks and other metabolomic QC approaches can be used to anchor studies and enable comparisons of stable spectral features across time without the need for compound identification. This approach is generalizable to systems where the same genotype can be assayed in multiple environments and provides biologically relevant features for downstream compound identification efforts. All methods are included in the newest release of the publicly available SECIMTools based on the open-source Galaxy platform.

Negative Affect and Political Sensitivity in Crossed Categorization: Self-Reports versus EMG

We experimentally examined the effects of negative integral affect on preferences among the double in-group (ii), crossed (io and oi), and double out-group (oo) targets of the crossed categorization paradigm. We used insults from members of politically sensitive vs. non-sensitive out-group categories of a crossed target (Oi) to induce affect. Dependent measures included self-reports and a psycho-physiological measure of affect (facial electromyography, EMG). Under no insult, participants conformed to social desirability pressure and favorably evaluated targets with a politically sensitive out-group membership, whereas facial EMG measures indicated greater negativity toward those same targets. Negativity of self-report and facial EMG measures converged, however, when members of a politically sensitive out-group category had provided hostility-justifying insults.

Adolescent Gender Differences in Residential Versus Outpatient Mental Health Treatment: A Meta-Analysis

Level-of-care decisions for adolescents in need of mental health services have been found to be inconsistent. This study sought to establish a research base for more objective and reliable level-of-care decisions related to gender differences in clinical characteristics of male and female patients aged 12–19 years receiving mental health treatment in a residential setting or outpatient mental health services. A meta-analytic approach was used to analyze 20 studies and 49 independent effect sizes across six constructs: internalizing, externalizing, psychosocial stressors, risk to self and others, total pathology, and substance use. The results indicated that male participants had higher levels of externalizing at the outpatient level, and female participants had higher levels of all remaining constructs at both levels of care. This indicated that female patients often presented to both levels of care with greater psychopathology.

The overall impact of emotional intelligence on nursing students and nursing

Healthcare employers often criticize the lack of emotional competency and critical thinking skills demonstrated by newly licensed nurses. The purpose of this study was to evaluate whether emotional intelligence (EI) training for nurses improves critical thinking and emotional competence enough to justify including EI in nursing curricula. A meta-analysis was conducted inclusive of EI related nursing abilities and traits such as leadership, health, reflection, ethical behavior, nursing student performance, and job retention/satisfaction. Studies of EI constructs, test instruments, and contrary viewpoints were also examined. The analysis included 395 EI studies of approximately 65,300 participants. All the studies reported a positive correlation with EI ranging from weak to strong with a moderate cumulative effect size of r = 0.3022 across all studies. This study may contribute to positive social change by reducing employers time and cost for training newly licensed nurses, thereby decreasing the overall cost of health care to the public.

Consensus-based recommendations for investigating clinical heterogeneity in systematic reviews

BACKGROUND

Critics of systematic reviews have argued that these studies often fail to inform clinical decision making because their results are far too general, that the data are sparse, such that findings cannot be applied to individual patients or for other decision making. While there is some consensus on methods for investigating statistical and methodological heterogeneity, little attention has been paid to clinical aspects of heterogeneity. Clinical heterogeneity, true effect heterogeneity, can be defined as variability among studies in the participants, the types or timing of outcome measurements, and the intervention characteristics. The objective of this project was to develop recommendations for investigating clinical heterogeneity in systematic reviews.

METHODS

We used a modified Delphi technique with three phases: (1) pre-meeting item generation; (2) face-to-face consensus meeting in the form of a modified Delphi process; and (3) post-meeting feedback. We identified and invited potential participants with expertise in systematic review methodology, systematic review reporting, or statistical aspects of meta-analyses, or those who published papers on clinical heterogeneity.

RESULTS

Between April and June of 2011, we conducted phone calls with participants. In June 2011 we held the face-to-face focus group meeting in Ann Arbor, Michigan. First, we agreed upon a definition of clinical heterogeneity: Variations in the treatment effect that are due to differences in clinically related characteristics. Next, we discussed and generated recommendations in the following 12 categories related to investigating clinical heterogeneity: the systematic review team, planning investigations, rationale for choice of variables, types of clinical variables, the role of statistical heterogeneity, the use of plotting and visual aids, dealing with outlier studies, the number of investigations or variables, the role of the best evidence synthesis, types of statistical methods, the interpretation of findings, and reporting.

CONCLUSIONS

Clinical heterogeneity is common in systematic reviews. Our recommendations can help guide systematic reviewers in conducting valid and reliable investigations of clinical heterogeneity. Findings of these investigations may allow for increased applicability of findings of systematic reviews to the management of individual patients.

Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature

BACKGROUND

While there is some consensus on methods for investigating statistical and methodological heterogeneity, little attention has been paid to clinical aspects of heterogeneity. The objective of this study is to summarize and collate suggested methods for investigating clinical heterogeneity in systematic reviews.

METHODS

We searched databases (Medline, EMBASE, CINAHL, Cochrane Library, and CONSORT, to December 2010) and reference lists and contacted experts to identify resources providing suggestions for investigating clinical heterogeneity between controlled clinical trials included in systematic reviews. We extracted recommendations, assessed resources for risk of bias, and collated the recommendations.

RESULTS

One hundred and one resources were collected, including narrative reviews, methodological reviews, statistical methods papers, and textbooks. These resources generally had a low risk of bias, but there was minimal consensus among them. Resources suggested that planned investigations of clinical heterogeneity should be made explicit in the protocol of the review; clinical experts should be included on the review team; a set of clinical covariates should be chosen considering variables from the participant level, intervention level, outcome level, research setting, or others unique to the research question; covariates should have a clear scientific rationale; there should be a sufficient number of trials per covariate; and results of any such investigations should be interpreted with caution.

CONCLUSIONS

Though the consensus was minimal, there were many recommendations in the literature for investigating clinical heterogeneity in systematic reviews. Formal recommendations for investigating clinical heterogeneity in systematic reviews of controlled trials are required.

Relations among posttraumatic stress disorder, comorbid major depression, and HPA function: a systematic review and meta-analysis

Exposure to traumatic stress is associated with increased risk for posttraumatic stress disorder (PTSD) and alterations of hypothalamic-pituitary-adrenocortical (HPA) function. Research linking traumatic stress with HPA function in PTSD has been inconsistent, however, in part due to (a) the inclusion of trauma-exposed individuals without PTSD (TE) in control groups and (b) a failure to consider comorbid major depressive disorder (MDD) and moderating variables. This meta-analysis of 47 studies (123 effect sizes, N=6008 individuals) revealed that daily cortisol output was lower for PTSD (d=-.36, SE=.15, p=.008) and PTSD+MDD (d=-.65, SE=.25, p=.008) groups relative to no trauma controls (NTC); TE and NTC groups did not differ significantly from each other. Afternoon/evening cortisol was lower in TE (d=-.25, SE=.09, p=.007) and PTSD (d=-.27, SE=.12, p=.021) groups and higher in PTSD+MDD groups (d=.49, SE=.24, p=.041) relative to NTC. Post-DST cortisol levels were lower in PTSD (d=-.40, SE=.12, p<.001), PTSD+MDD (d=-.65, SE=.14, p<.001), and TE groups (d=-.53, SE=.14, p<.001) relative to NTC. HPA effect sizes were moderated by age, sex, time since index event, and developmental timing of trauma exposure. These findings suggest that enhanced HPA feedback function may be a marker of trauma-exposure rather than a specific mechanism of vulnerability for PTSD, whereas lower daily cortisol output may be associated with PTSD in particular.

Hypothalamic-pituitary-adrenal axis dysregulation in depressed children and adolescents: a meta-analysis

Research findings on the hypothalamic-pituitary-adrenal (HPA) axis and pediatric depression reflect a variety of methodological approaches that tap different facets of HPA-axis functions. Partly owing to the methodological heterogeneity of studies, descriptive reviews of this area have produced inconsistent conclusions. Therefore, we conducted formal meta-analyses of pertinent studies in order to advance our understanding of HPA-axis dysregulation in pediatric depression. We examined: (a) 17 published studies of HPA-axis response to the dexamethasone suppression test (DST) in depressed youth (DST; N=926) and (b) 17 studies of basal HPA-axis functioning (N=1332). We also examined descriptively studies that used corticotropin-releasing hormone (CRH) infusion, and those that used psychological probes of the HPA-axis. The global standardized mean effect size difference in HPA-axis response to the DST between depressed and non-depressed youth was 0.57, z=4.18, p<0.01. The global standardized mean difference effect size in basal HPA-axis functioning was 0.20, z=4.53, p<0.01. Age, sex, timing of sampling, dexamethasone dosage, or type of control group was not a significant source of variability for the DST or basal studies. In addition, when compared to non-depressed peers, depressed youth have a normative response to CRH infusion but an overactive response to psychological stressors. In conclusion, the HPA-axis system tends to be dysregulated in depressed youth, as evidenced by atypical responses to the DST, higher baseline cortisol values, and an overactive response to psychological stressors. This pattern of dysregulation suggests anomalies within the axis's negative feedback system and CRH production, but intact pituitary and adrenal sensitivity.

Self-efficacy and smoking cessation: a meta-analysis

According to relapse models, self-efficacy (SE), or confidence in one's ability to abstain, should predict the outcome of an attempt to quit smoking. We reviewed 54 studies that prospectively examined this relationship. The relationship between SE and future smoking depended upon the population studied and the timing of the SE assessment. The relationship between SE and future smoking was modest when SE was assessed prior to a quit attempt; SE scores were .21 standard deviation units (SD) higher for those not smoking at follow-up than for those who were smoking. The relationship was stronger (.47 SD) when SE was assessed post-quit. However, this effect was diminished when only abstainers at the time of the SE assessment were included in analysis (.28 SD). Controlling for smoking status at the time of SE assessment substantially reduced the relationship between SE and future smoking. Although SE has a reliable association with future abstinence, it is less robust than expected. Many studies may overestimate the relationship by failing to appropriately control for smoking behavior at the time of the SE assessment. (PsycINFO Database Record (c) 2009 APA, all rights reserved).

Five ways of being "theoretical": applications to provider-patient communication research

OBJECTIVE

Analyzes the term "theoretical" as it applies to the area of provider-patient communication research, in order to understand better at a conceptual level what the term may mean for authors and critics.

METHODS

Based on literature on provider-patient communication.

RESULTS

Offers, and discusses, five definitions of the term "theoretical" as it applies to empirical research and its exposition: (1) grounding, (2) referencing, (3) design and analysis, (4) interpretation, and (5) impact. Each of these definitions embodies a different standard for evaluating the theoretical aspects of research.

CONCLUSION

Although it is often said that research on provider-patient communication is not "theoretical" enough, the term is ambiguous and often applied vaguely. A multidimensional analysis reveals that there are several distinct ways in which empirical research can be strong or weak theoretically.

PRACTICE IMPLICATIONS

Researchers, educators, editors, and reviewers could use the "Five Ways" framework to appraise the theory-relevant strengths and weaknesses of empirical research and its exposition.

Smooth pursuit in schizophrenia: a meta-analytic review of research since 1993

Abnormal smooth pursuit eye-tracking is one of the most replicated deficits in the psychophysiological literature in schizophrenia [Levy, D. L., Holzman, P. S., Matthysse, S., & Mendell, N. R. (1993). Eye tracking dysfunction and schizophrenia: A critical perspective. Schizophrenia Bulletin, 19, 461-505]. We used meta-analytic procedures to quantify patient-control differences in eye-tracking and to evaluate potential moderators of effect size including patient and target characteristics and characteristics of the control population (matched or not). The magnitude of patient-control differences in pursuit depended on the measure. Global measures had large effect sizes. Among specific measures, maintenance gain and leading saccades yielded large effect sizes, with gain also yielding the narrowest confidence interval. Effect sizes associated with specific measures of smooth pursuit vs. specific measures of intrusive saccades did not clearly implicate one system over the other. Patient demographics and target characteristics generally had little influence on effect sizes. However, studies that failed to sex-match patients and controls tended to have smaller effect sizes for maintenance gain and catch-up saccade rate. Average effect sizes and confidence limits for global measures of pursuit and for maintenance gain place these measures alongside the very strongest neurocognitive measures in the literature [Heinrichs, R. W. (2004). Meta-analysis, and the science of schizophrenia: Variant evidence or evidence of variants? Neuroscience and Biobehavioral Reviews, 28, 379-394] for distinguishing between patients with schizophrenia and controls.

Impact of EMG-triggered neuromuscular stimulation of the wrist and finger extensors of the paretic hand after stroke: a systematic review of the literature

OBJECTIVE

To assess whether EMG-triggered neuromuscular electrical stimulation (EMG-NMES) applied to the extensor muscles of the forearm improves hand function after stroke.

DESIGN

Systematic review of randomized controlled trials.

METHODS

A computer-aided literature search up to June 2006 identified articles comparing EMG-NMES of the upper extremity with usual care. Methodological quality was rated on the Physiotherapy Evidence Database scale (PEDro), and the Hedges' g model was used to calculate the summary effect sizes (SES) using fixed or random models depending on heterogeneity.

RESULTS

Eight studies, selected out of 192 hits and presenting 157 patients, were included in quantitative and qualitative analyses. The methodological quality ranged from 2 to 6 points. The meta-analysis revealed non-significant effect sizes in favour of EMG-NMES for reaction time, sustained contraction, dexterity measured with the Box and Block manipulation test, synergism measured with the Fugl-Meyer Motor Assessment Scale and manual dexterity measured with the Action Research Arm test.

CONCLUSION

No statistically significant differences in effects were found between EMG-NMES and usual care. Most studies had poor methodological quality, low statistical power and insufficient treatment contrast between experimental and control groups. In addition, all studies except two investigated the effects of EMG-NMES in the chronic phase after stroke, whereas the literature suggests that an early start, within the time window in which functional outcome of the upper limb is not fully defined, is more appropriate.

Examining the impact of cell phone conversations on driving using meta-analytic techniques

OBJECTIVE

The performance costs associated with cell phone use while driving were assessed meta-analytically using standardized measures of effect size along five dimensions.

BACKGROUND

There have been many studies on the impact of cell phone use on driving, showing some mixed findings.

METHODS

Twenty-three studies (contributing 47 analysis entries) met the appropriate conditions for the meta-analysis. The statistical results from each of these studies were converted into effect sizes and combined in the meta-analysis.

RESULTS

Overall, there were clear costs to driving performance when drivers were engaged in cell phone conversations. However, subsequent analyses indicated that these costs were borne primarily by reaction time tasks, with far smaller costs associated with tracking (lane-keeping) performance. Hands-free and handheld phones revealed similar patterns of results for both measures of performance. Conversation tasks tended to show greater costs than did information-processing tasks (e.g., word games). There was a similar pattern of results for passenger and remote (cell phone) conversations. Finally, there were some small differences between simulator and field studies, though both exhibited costs in performance for cell phone use.

CONCLUSION

We suggest that (a) there are significant costs to driver reactions to external hazards or events associated with cell phone use, (b) hands-free cell phones do not eliminate or substantially reduce these costs, and (c) different research methodologies or performance measures may underestimate these costs.

APPLICATION

Potential applications of this research include the assessment of performance costs attributable to different types of cell phones, cell phone conversations, experimental measures, or methodologies.

A meta-analysis of the relationship of child sexual abuse to HIV risk behavior among women

OBJECTIVE

This study is a meta-analysis of the literature exploring the relationship between child sexual abuse (CSA) and HIV risk behavior among women. Four outcome variables were tested: unprotected sex; sex with multiple partners; sex trading; and adult sexual revictimization.

METHOD

Forty-six studies met the inclusion criteria and were included in the analysis, and separate meta-analyses were performed for each of the four dependent variables described above.

RESULTS

Using the correlation coefficient r as an effect size estimate, results indicate an average effect size of .05 for the unprotected sex meta-analysis (N = 16 studies), .13 for the sex with multiple partners meta-analysis (N = 23 studies), .12 for the sex trading meta-analysis (N = 23 studies), and .17 for the adult sexual revictimization meta-analysis (N = 21 studies). We conducted a test of three potential moderator variables (source of sample, definition of CSA based on type of contact, and definition of CSA based on maximum age of victim). Results did not support the hypothesis that these three variables explain a significant amount of variability in effect sizes with one exception: Studies that define CSA more broadly to include adolescent victims (e.g., victims up to 17 years of age) had larger and more homogenous effect sizes for the sex trading meta-analysis than those that defined CSA as having occurred at younger ages.

CONCLUSIONS

These findings suggest a small positive relationship between CSA and subsequent HIV risk behavior among women that varies as a function of how CSA and HIV risk behavior are operationalized.

A meta-analysis of adolescent psychosocial smoking prevention programs published between 1978 and 1997 in the United States

Psychosocial smoking prevention studies have shown inconsistent results and theory-driven programs have been related to program success. This meta-analysis was used as a judgment tool for resolving these issues by estimating average program effects and investigating the relative efficacy of program types. The present study examined 65 adolescent psychosocial smoking prevention programs (1978 to 1997) among students in Grades 6 to 12 in the United States. Three program modalities (social influence, cognitive behavior, life skill) and two program settings (exclusively school based, school-community-incorporated) were identified as major a priori classifications. Knowledge had the highest effect sizes (.53) at short-term (< or = 1 year) but rapidly decreased (.19) at long-term (> 1 year). Behavioral effect was the most meaningful, being persistent over a 3-year period (.19 at < or = 1 year; .18 at 1 to 3 years). Adolescent smoking reduction rates were increased by using either cognitive behavior or life skills program modalities, and/or a school-community-incorporated program setting.

Meta-analysis: recent developments in quantitative methods for literature reviews

We describe the history and current status of the meta-analytic enterprise. The advantages and historical criticisms of meta-analysis are described, as are the basic steps in a meta-analysis and the role of effect sizes as chief coins of the meta-analytic realm. Advantages of the meta-analytic procedures include seeing the "landscape" of a research domain, keeping statistical significance in perspective, minimizing wasted data, becoming intimate with the data summarized, asking focused research questions, and finding moderator variables. Much of the criticism of meta-analysis has been based on simple misunderstanding of how meta-analyses are actually carried out. Criticisms of meta-analysis that are applicable are equally applicable to traditional, nonquantitative, narrative reviews of the literature. Much of the remainder of the chapter deals with the processes of effect size estimation, the understanding of the heterogeneity of the obtained effect sizes, and the practical and scientific importance of the effect sizes obtained.

Aerobic exercise and bone density at the hip in postmenopausal women: a meta-analysis

BACKGROUND

The effects of aerobic exercise on bone density at the hip in postmenopausal women in the absence of estrogen replacement therapy are not currently known. The purpose of this study was to examine the effects of aerobic exercise on bone density at the hip in postmenopausal women.

METHODS

Using the meta-analytic approach, studies dealing with the effects of aerobic exercise on bone density at the hip in postmenopausal women were searched for using computerized literature searches (MEDLINE, January 1978 to December 1995) as well as cross-referencing from retrieved review articles and original investigations.

RESULTS

A total of 18 effect sizes were derived from six studies. Using a fixed-effects model and bootstrap resampling (5,000 iterations) overall changes in bone density at the hip yielded an average effect size of 0.43 (95% CI = 0.04 to 0.81), equivalent to an overall change of approximately 2.42% (exercise = 2.13%; nonexercise = -0.29%). Statistically significant differences were observed when effect sizes were partitioned by country in which studies were conducted (United States, mean = 1.03, 95% CI = 0.48 to 1.68; other countries, mean = 0.18, 95% CI = -0.27 to 0.54; Qb = 5.44, P = 0.04) and calcium intake (> or =1,000 mg/day, mean = 0.83, 95% CI = 0.49 to 1.23; <1,000 mg/day = -0.23, 95% CI = -0.85 to 0.21; Qb = 10.64, P = 0.002).

CONCLUSIONS

The overall results of this study suggest that site-specific aerobic exercise has a moderately positive effect on bone density at the hip in postmenopausal women. However, a need exists for additional, well-designed studies before a final recommendation can be made regarding the efficacy of aerobic exercise as a nonpharmacologic intervention for optimizing bone density at the hip in postmenopausal women.

Effectiveness of interventions to improve patient compliance: a meta-analysis

OBJECTIVES

This article summarizes the results of 153 studies published between 1977 and 1994 that evaluated the effectiveness of interventions to improve patient compliance with medical regimens.

METHODS

The compliance interventions were classified by theoretical focus into educational, behavioral, and affective categories within which specific intervention strategies were further distinguished. The compliance indicators broadly represent five classes of compliance-related assessments: (1) health outcomes (eg, blood pressure and hospitalization), (2) direct indicators (eg, urine and blood tracers and weight change), (3) indirect indicators (eg, pill count and refill records), (4) subjective report (eg, patients' or others' reports), and (5) utilization (appointment making and keeping and use of preventive services). An effect size (ES) r, defined as Fisher's Z transformation of the Pearson correlation coefficient, representing the association between each intervention (intervention versus control) and compliance measure was calculated. Both an unweighted and weighted r were calculated because of large sample size variation, and a combined probability across studies was calculated.

RESULTS

The interventions produced significant effects for all the compliance indicators (combined Z values more than 5 and less than 32), with the magnitude of effects ranging from small to large. The largest effects (unweighted) were evident for refill records and pill counts and in blood/urine and weight change studies. Although smaller in magnitude, compliance effects were evident for improved health outcomes and utilization. Chronic disease patients, including those with diabetes and hypertension, as well as cancer patients and those with mental health problems especially benefited from interventions.

CONCLUSIONS

No single strategy or programmatic focus showed any clear advantage compared with another. Comprehensive interventions combining cognitive, behavioral, and affective components were more effective than single-focus interventions.

Aerobic exercise and lumbar spine bone mineral density in postmenopausal women: a meta-analysis

PURPOSE

To use the meta-analytic approach to examine the effects of aerobic exercise on lumbar spine bone mineral density in postmenopausal women.

DATA SOURCES

Studies were retrieved from computer searches (MEDLINE, Current Contents) as well as cross-referencing from the bibliographies of retrieved studies and review articles.

STUDY SELECTION

A total of 17 effect sizes consisting of 330 subjects (192 exercise, 138 nonexercise) were included from the 10 studies that met the criteria for inclusion: (1) aerobic activity as the only exercise intervention, (2) postmenopausal women as subjects, (3) comparative nonexercise group included, (4) changes in lumbar spine bone mineral density reported, (5) studies published in English-language journals between January 1975 and December 1994.

DATA EXTRACTION

The primary outcome measure of interest was the treatment effect delta3, defined as the percent change in lumbar spine bone mineral density, calculated by subtracting the percent change in the exercise group from the percent change in the nonexercise group.

RESULTS

Using a fixed effects model, bootstrap analysis (10,000 replications) showed that significant changes in lumbar spine bone mineral density occurred (mean +/- SD = 2.83 +/- 0.77%, 95% confidence interval (CI) = 1.33 to 4.35%). This overall average increase was caused primarily by the loss of lumbar spine bone mineral density in the nonexercise group relative to the exercise group (mean +/- SD, exercise = .32 +/- 2.46%, 95% CI = -0.94 to 1.58; nonexercise = -2.51 +/- 2.69%, 95% CI = -4.60 to -0.96).

CONCLUSION

The results of this meta-analysis suggest that aerobic exercise helps to maintain lumbar spine bone mineral density in postmenopausal women.