Meta-analysis of clinical trials as a scientific discipline. II: Replicate variability and comparison of studies that agree and disagree

Systematic Reviews in Hematopoietic Cell Transplantation and Cellular Therapy: Considerations and Guidance from the American Society for Transplantation and Cellular Therapy, European Society for Blood and Marrow Transplantation, and Center for International Blood and Marrow Transplant Research Late Effects and Quality of Life Working Committee

Systematic reviews apply rigorous methodologies to address a prespecified, clearly formulated clinical research question. The conclusion that results is often cited to more robustly inform decision making by clinicians, third-party payers, and managed care organizations about the clinical question of interest. Although systematic reviews provide a rigorous standard, they may be infeasible when the task is to create general disease-focused guidelines comprising multiple clinical practice questions versus a single major clinical practice question. Collaborating transplantation and cellular therapy society committees also recognize that the quantity and or quality of reference sources may be insufficient for a meaningful systematic review. As the conduct of systematic reviews has evolved over time in terms of grading systems, reporting requirements, and use of technology, here we provide current guidance on methodologies, resources for reviewers, and approaches to overcome challenges in conducting systematic reviews in transplantation and cellular therapy.

Systematic reviews in hematopoietic cell transplantation and cellular therapy: considerations and guidance from the American Society for Transplantation and Cellular Therapy, European Society for Blood and Marrow Transplantation, and the Center for International Blood and Marrow Transplant Research late effects and quality of life working committee

Systematic reviews apply rigorous methodologies to address a pre-specified, clearly formulated clinical research question. The conclusion that results is often cited to more robustly inform decision-making by clinicians, third-party payers and managed care organizations about the clinical question of interest. While systematic reviews provide a rigorous standard, they may be unfeasible when the task is to create general disease-focused guidelines comprised of multiple clinical practice questions versus a single major clinical practice question. Collaborating transplantation and cellular therapy societal committees also recognize that the quantity and or quality of reference sources may be insufficient for a meaningful systematic review. As the conduct of systematic reviews has evolved over time in terms of grading systems, reporting requirements and use of technology, here we provide current guidance in methodologies, resources for reviewers, and approaches to overcome challenges in conducting systematic reviews in transplantation and cellular therapy.

Multivariate Methods for Meta-Analysis of Genetic Association Studies

Multivariate meta-analysis of genetic association studies and genome-wide association studies has received a remarkable attention as it improves the precision of the analysis. Here, we review, summarize and present in a unified framework methods for multivariate meta-analysis of genetic association studies and genome-wide association studies. Starting with the statistical methods used for robust analysis and genetic model selection, we present in brief univariate methods for meta-analysis and we then scrutinize multivariate methodologies. Multivariate models of meta-analysis for a single gene-disease association studies, including models for haplotype association studies, multiple linked polymorphisms and multiple outcomes are discussed. The popular Mendelian randomization approach and special cases of meta-analysis addressing issues such as the assumption of the mode of inheritance, deviation from Hardy-Weinberg Equilibrium and gene-environment interactions are also presented. All available methods are enriched with practical applications and methodologies that could be developed in the future are discussed. Links for all available software implementing multivariate meta-analysis methods are also provided.

Reconciling disparate data to determine the right answer: A grounded theory of meta analysts' reasoning in meta-analysis

While the systematic review process is intended to maximize objectivity and limit researchers' biases, examples remain of discordant recommendations from meta-analyses. Current guidelines to explore discrepancies assume the variation is produced by methodological differences and thus focus only on the study process. Because heterogeneity of interpretation also occurs when experts examine the same data, our purpose was to examine if there are reasoning differences, ie, in how information is processed and valued. We created simulated meta-analyses based on idealized randomized studies (ie, perfect studies with no bias) to ensure differences in interpretations could only be due to reasoning. We recruited published meta-analysts using purposeful variables. We conducted 3 audio-recorded interviews per participant using structured and semi-structured interviews, with paraphrasing and reflective listening to enhance and verify responses. Recruitment and analysis of transcripts and field notes followed the principles of grounded theory (eg, theoretical saturation, constant comparative analysis). Results show the complexity of meta-analytic reasoning. At each step of the process, participants attempted to reconcile disparate forms of knowledge to determine a right answer (moral concern) and accurately draw a treatment effect (epistemological concern). The reasoning processes often shifted between considering the meta-analysis as if the data were whole, and as if the data were discrete components (individual studies). These findings highlight paradigmatic tensions regarding the epistemological premises of meta-analysis, resembling previous historical investigations of the functioning of scientific communities. In understanding why different meta-analysts interpret data differently, it may be unrealistic to expect objective homogenous recommendations based on meta-analyses.

Clinical Benefit of Gastric Staple Line Reinforcement (SLR) in Gastrointestinal Surgery: a Meta-analysis

Background

The objective of this study was to assess whether the use of staple line reinforcement (SLR) reduces staple line complications (SLC). Mechanical staple lines are essential for gastrointestinal surgery such as bariatric surgery. However, SLC, such as bleeding and leakage, still occur. The purposes of this study were to provide quantitative evidence on the relative efficacy of gastric SLR and to compare the rates of effectiveness of three commonly used methods.

Methods

A search of the medical literature in English language journals identified studies from Jan 1, 2000, to Dec 31, 2013, using the following reinforcement types: (1) no reinforcement, (2) oversewing, (3) a biocompatible glycolide copolymer, and (4) bovine pericardium after gastric bypasses and sleeve gastrectomies. Types of reinforcement were compared using a random-effects model.

Results

This meta-analysis reviewed 16,967 articles, extracting data on 56,309 patients concerning leak and 41,864 patients concerning bleeding. Over 40 % of patients had no reinforcement, resulting in the highest leak rate (2.75 %) and bleed rate (3.45 %). Overall, reinforcing with bovine pericardium had the lowest leak (1.28 %) and bleed (1.23 %) rates. Suture oversewing was better than no reinforcement but not as effective as bovine pericardium for leak (2.45 %) and bleed (2.69 %) rates. Buttressing with a biocompatible glycolide copolymer resulted in the second highest leak rate (2.61 %) and a bleed rate of 2.48 % but had significantly lower bleed rates than no reinforcement.

Conclusions

SLR provided superior results for patients compared to no reinforcement for reducing SLC. Buttressing with bovine pericardium resulted in the most favorable outcomes. The effectiveness of different methods used to reinforce the staple line in gastric surgery does not appear to be equal.

Why it is important to include unpublished data in systematic reviews

It is known that studies with statistically significant results have a higher probability to be published (publication bias). Therefore, studies with no statistically significant differences (or not favoring the investigational drug) may not be found in commonly accessed databases and remain unpublished. Moreover, unpublished data may also refer to information that are not included in study reports published in scientific journals but that may be important to estimate study outcomes. Retrieving unpublished evidence represents a compelling challenge for researchers, and in the present paper we explore how to do it.

Meta-analysis of haplotype-association studies: comparison of methods and empirical evaluation of the literature

Background

Meta-analysis is a popular methodology in several fields of medical research, including genetic association studies. However, the methods used for meta-analysis of association studies that report haplotypes have not been studied in detail. In this work, methods for performing meta-analysis of haplotype association studies are summarized, compared and presented in a unified framework along with an empirical evaluation of the literature.

Results

We present multivariate methods that use summary-based data as well as methods that use binary and count data in a generalized linear mixed model framework (logistic regression, multinomial regression and Poisson regression). The methods presented here avoid the inflation of the type I error rate that could be the result of the traditional approach of comparing a haplotype against the remaining ones, whereas, they can be fitted using standard software. Moreover, formal global tests are presented for assessing the statistical significance of the overall association. Although the methods presented here assume that the haplotypes are directly observed, they can be easily extended to allow for such an uncertainty by weighting the haplotypes by their probability.

Conclusions

An empirical evaluation of the published literature and a comparison against the meta-analyses that use single nucleotide polymorphisms, suggests that the studies reporting meta-analysis of haplotypes contain approximately half of the included studies and produce significant results twice more often. We show that this excess of statistically significant results, stems from the sub-optimal method of analysis used and, in approximately half of the cases, the statistical significance is refuted if the data are properly re-analyzed. Illustrative examples of code are given in Stata and it is anticipated that the methods developed in this work will be widely applied in the meta-analysis of haplotype association studies.

Systematic review and cost analysis comparing use of chlorhexidine with use of iodine for preoperative skin antisepsis to prevent surgical site infection

OBJECTIVE

To compare use of chlorhexidine with use of iodine for preoperative skin antisepsis with respect to effectiveness in preventing surgical site infections (SSIs) and cost.

METHODS

We searched the Agency for Healthcare Research and Quality website, the Cochrane Library, Medline, and EMBASE up to January 2010 for eligible studies. Included studies were systematic reviews, meta-analyses, or randomized controlled trials (RCTs) comparing preoperative skin antisepsis with chlorhexidine and with iodine and assessing for the outcomes of SSI or positive skin culture result after application. One reviewer extracted data and assessed individual study quality, quality of evidence for each outcome, and publication bias. Meta-analyses were performed using a fixed-effects model. Using results from the meta-analysis and cost data from the Hospital of the University of Pennsylvania, we developed a decision analytic cost-benefit model to compare the economic value, from the hospital perspective, of antisepsis with iodine versus antisepsis with 2 preparations of chlorhexidine (ie, 4% chlorhexidine bottle and single-use applicators of a 2% chlorhexidine gluconate [CHG] and 70% isopropyl alcohol [IPA] solution), and also performed sensitivity analyses.

RESULTS

Nine RCTs with a total of 3,614 patients were included in the meta-analysis. Meta-analysis revealed that chlorhexidine antisepsis was associated with significantly fewer SSIs (adjusted risk ratio, 0.64 [95% confidence interval, [0.51-0.80]) and positive skin culture results (adjusted risk ratio, 0.44 [95% confidence interval, 0.35-0.56]) than was iodine antisepsis. In the cost-benefit model baseline scenario, switching from iodine to chlorhexidine resulted in a net cost savings of $16-$26 per surgical case and $349,904-$568,594 per year for the Hospital of the University of Pennsylvania. Sensitivity analyses showed that net cost savings persisted under most circumstances.

CONCLUSIONS

Preoperative skin antisepsis with chlorhexidine is more effective than preoperative skin antisepsis with iodine for preventing SSI and results in cost savings.

Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review

OBJECTIVE

To evaluate the quality of reviews about etanercept (ETN) and infliximab (IFX), two biologic treatments for rheumatoid arthritis (RA).

STUDY DESIGN

A comprehensive, systematic review, including searches of MEDLINE, EMBASE, and other electronic databases and hand-searches for published and unpublished literature. Two raters independently examined each article and identified systematic reviews as those including either a description of: (1) sources for identification and data retrieval; or (2) search strategy. They applied the quality of reporting of meta-analyses (QUOROM) instrument to systematic reviews.

RESULTS

Of 3,620 total citations, 281 were identified as reviews. Of these, 26 (9%) qualified as systematic rather than narrative. Overall, few reviews described selection of sources, critical appraisal, or quantitative summary or synthesis. Systematic reviews most often failed to explain validity assessment. Several articles did not disclose authors' participation in industry-funded clinical trials. Most reviews published in high impact factor and rheumatology journals did not meet many quality standards. Significant associations existed between review type (narrative vs. systematic) and reported funding (P=0.05), conflicts of interest (P=0.005), and country of publication (P<0.0001).

CONCLUSION

More than 90% of the published reviews were narrative and did not report methods and conflicts of interest in sufficient detail, raising concerns about selection and reporting bias.

The interpretation of systematic reviews with meta-analyses: an objective or subjective process?

Background

Discrepancies between the conclusions of different meta-analyses (quantitative syntheses of systematic reviews) are often ascribed to methodological differences. The objective of this study was to determine the discordance in interpretations when meta-analysts are presented with identical data.

Methods

We searched the literature for all randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period. We organized the articles chronologically and grouped them in packages. The first package included the first RCT, and a summary of the review articles published prior to first RCT. The second package contained the second and third RCT, a meta-analysis based on the data, and a summary of all review articles published prior to the third RCT. Similar packages were created for the 5^th RCT, 10^th RCT, 20^th RCT and 23^rd RCT (all articles). We presented the packages one at a time to eight different reviewers and asked them to answer three clinical questions after each package based solely on the information provided. The clinical questions included whether 1) they believed magnesium is now proven beneficial, 2) they believed magnesium will eventually be proven to be beneficial, and 3) they would recommend its use at this time.

Results

There was considerable disagreement among the reviewers for each package, and for each question. The discrepancies increased when the heterogeneity of the data increased. In addition, some reviewers became more sceptical of the effectiveness of magnesium over time, and some reviewers became less sceptical.

Conclusion

The interpretation of the results of systematic reviews with meta-analyses includes a subjective component that can lead to discordant conclusions that are independent of the methodology used to obtain or analyse the data.

Quantitative RT-PCR detection of colorectal tumor cells in peripheral blood--a systematic review

BACKGROUND

Twenty percent to 40% of patients with node-negative colorectal cancer die of metastatic disease. Detection of cancer cell dissemination has been proposed as a tool to select patients at highest risk for recurrence. In this review, we summarize the evidence for detection with quantitative reverse transcriptase-polymerase chain reaction (RT-PCR) assays of circulating tumor cells (CTCs) in peripheral blood of colorectal cancer patients.

MATERIALS AND METHODS

Pubmed and Embase were systematically searched for all English publications relevant to circulating cancer cells, peripheral blood, quantitative RT-PCR (q-RT-PCR), and colorectal cancer. Cross-references and the "related articles" function were used to broaden the search. Manuscripts reporting on the results of nonquantitative RT-PCR assays were excluded. The study methodology, CTCs detection rates in peripheral blood, and prognostic value were reviewed.

RESULTS

Twelve manuscripts on qRT-PCR were retrieved. Stage dependence was found for detection of CTCs in four of 10 studies. From univariate analysis performed for disease-free survival and overall survival in 4 of 12 studies, there was evidence (P < 0.05) for an effect of the detection of CTCs with qRT-PCR. None of the included trials identified detection of CTCs in peripheral blood as an independent predictor of survival.

CONCLUSION

Quantification of CTCs in peripheral blood holds promise in predicting stage and outcome in colorectal cancer patients. At present, evidence from the literature is too scarce to integrate quantitative RT-PCR assays to detect CTCs into the management of colorectal cancer.

Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study

OBJECTIVE

To determine whether financial ties to one drug company are associated with favourable results or conclusions in meta-analyses on antihypertensive drugs.

DESIGN

Retrospective cohort study.

SETTING

Meta-analyses published up to December 2004 that were not duplicates and evaluated the effects of antihypertensive drugs compared with any comparator on clinical end points in adults. Financial ties were categorised as one drug company compared with all others.

MAIN OUTCOME MEASURES

The main outcomes were the results and conclusions of meta-analyses, with both outcomes separately categorised as being favourable or not favourable towards the study drug. We also collected data on characteristics of meta-analyses that the literature suggested might be associated with favourable results or conclusions.

RESULTS

124 meta-analyses were included in the study, 49 (40%) of which had financial ties to one drug company. On univariate logistic regression analyses, meta-analyses of better methodological quality were more likely to have favourable results (odds ratio 1.16, 95% confidence interval 1.07 to 1.27). Although financial ties to one drug company were not associated with favourable results, such ties constituted the only characteristic significantly associated with favourable conclusions (4.09, 1.30 to 12.83). When controlling for other characteristics of meta-analyses in multiple logistic regression analyses, meta-analyses that had financial ties to one drug company remained more likely to report favourable conclusions (5.11, 1.54 to 16.92).

CONCLUSION

Meta-analyses on antihypertensive drugs and with financial ties to one drug company are not associated with favourable results but are associated with favourable conclusions.

Communicating the uncertainty of harms and benefits of medical interventions

BACKGROUND

There is growing interest in shared medical decision making among patients, physicians, and policy makers. This requires patients to interpret increasing amounts of medical information, much of which is uncertain. Little is known about the optimal approaches to or outcomes of communicating uncertainty about the risks and benefits of treatments.

METHODS

The authors reviewed the literature on various issues related to uncertainty in decision making: conceptualizing uncertainty, identifying its potential sources, assessing uncertainty, potential methods of communicating uncertainty, potential outcomes of communicating uncertainty, and current practices and recommendations by expert groups on communicating uncertainty.

RESULTS

There are multiple sources of uncertainty in most medical decisions. There are conceptual differences in how researchers define uncertainty and its sources, as well as in its measurement. The few studies that have assessed alternate means of communicating uncertainty dealt mostly with presenting uncertainty about probabilities. Both patients' and physicians' interpretation of and responses to uncertainty may depend on their personal characteristics and values and may be affected by the manner in which uncertainty is communicated.

CONCLUSIONS

Research has not yet identified best practices for communicating uncertainty to patients about harms and benefits of treatment. More conceptual, qualitative, and quantitative studies are needed to explore fundamental questions about how people process, interpret, and respond to various types of uncertainty inherent in clinical decisions.

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review

OBJECTIVE

To compare the methodological quality and conclusions in Cochrane reviews with those in industry supported meta-analyses and other meta-analyses of the same drugs.

DESIGN

Systematic review comparing pairs of meta-analyses that studied the same two drugs in the same disease and were published within two years of each other.

DATA SOURCES

Cochrane Database of Systematic Reviews (2003, issue 1), PubMed, and Embase.

DATA EXTRACTION

Two observers independently extracted data and used a validated scale to judge the methodological quality of the reviews.

RESULTS

175 of 1596 Cochrane reviews had a meta-analysis that compared two drugs. Twenty four meta-analyses that matched the Cochrane reviews were found: eight were industry supported, nine had undeclared support, and seven had no support or were supported by non-industry sources. On a 0-7 scale, the median quality score was 7 for Cochrane reviews and 3 for other reviews (P < 0.01). Compared with industry supported reviews and reviews with undeclared support, Cochrane reviews had more often considered the potential for bias in the review--for example, by describing the method of concealment of allocation and describing excluded patients or studies. The seven industry supported reviews that had conclusions recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P = 0.02), although the estimated treatment effect was similar on average (z = 0.46, P = 0.64). Reviews with undeclared support and reviews with not for profit support or no support had conclusions that were similar in cautiousness to the Cochrane reviews.

CONCLUSIONS

Industry supported reviews of drugs should be read with caution as they were less transparent, had few reservations about methodological limitations of the included trials, and had more favourable conclusions than the corresponding Cochrane reviews.

Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study

OBJECTIVE

To appraise multiple systematic reviews on the same clinical topic, focusing on predictors and correlates of quality of reporting of meta-analysis (QUOROM) scores.

DESIGN

Case study.

SETTING

Reviews providing at least individual quantitative estimates on role of acetylcysteine in the prevention of contrast associated nephropathy.

DATA SOURCES

PubMed, the database of abstracts of reviews of effects, and the Cochrane database of systematic reviews (updated March 2005).

MAIN OUTCOME MEASURES

Funding, compliance with the QUOROM checklist, scores on the Oxman and Guyatt quality index, and authors' recommendations.

RESULTS

10 systematic reviews, published August 2003 to March 2005, were included. Nine pooled events despite heterogeneity and five recommended routine use of acetylcysteine, whereas the remaining studies called for further research. Compliance with the 18 items on the QUOROM checklist was relatively high (median 16, range 11 to 17), although shorter manuscripts had significantly lower scores (R = 0.73; P = 0.016). Reviewers who reported previous not for profit funding were more likely to score higher on the Oxman and Guyatt quality index. No association was found between QUOROM and Oxman and Guyatt scores (R = -0.06; P = 0.86), mainly because of greater emphasis of the Oxman and Guyatt scores on the appraisal of bias in selection and validity assessment (inadequate in five reviews).

CONCLUSIONS

Multiple systematic reviews on the same clinical topic varied in quality of reporting and recommendations. Longer manuscripts and previous not for profit funding were associated with higher quality.

Autism spectrum disorders in Prader???Willi and Angelman syndromes: a systematic review

Autism spectrum disorders (ASDs) have been linked with maternally derived duplications/triplications of chromosome 15q11-13 and therefore might occur more frequently in people with Prader-Willi syndrome (PWS) when due to uniparental disomy (UPD), than in other forms of chromosomal abnormality involving this region [i.e. deletion (DEL) forms of PWS and DEL+UPD forms of Angelman's syndrome -(AS)]. Twelve studies regarding ASD in PWS and AS were reviewed. It was noteworthy that among the genetically confirmed UPD and DEL cases of PWS and AS, the rate of ASD was 25.3% (38/150; range 0-36.5%) in PWS and 1.9% in AS (2/104; range 0-100%) (Fisher's exact P<0.0001). Among the subset of cases with confirmed UPD or DEL, the rate of ASD in the UPD cases of PWS was significantly higher (20/53) than in the remaining combined samples (i.e. DEL PWS+UPD AS+DEL AS cases; 20/201) (Fisher's exact P<0.0001). ASD in UPD PWS cases (20/53) compared with DEL PWS cases (18/97) was also statistically significant (Fisher's exact P=0.0176). Thus, the limited available evidence supported the prediction that overexpression of maternally imprinted genes in 15q11-13 confers a risk for ASD. Further research will be required to confirm these findings.

Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary

OBJECTIVE

To assess whether language of publication restrictions impact the estimates of an intervention's effectiveness, whether such impact is similar for conventional medicine and complementary medicine interventions, and whether the results are influenced by publication bias and statistical heterogeneity.

STUDY DESIGN AND SETTING

We set out to examine the extent to which including reports of randomized controlled trials (RCTs) in languages other than English (LOE) influences the results of systematic reviews, using a broad dataset of 42 language-inclusive systematic reviews, involving 662 RCTs, including both conventional medicine (CM) and complementary and alternative medicine (CAM) interventions.

RESULTS

For CM interventions, language-restricted systematic reviews, compared with language-inclusive ones, did not introduce biased results, in terms of estimates of intervention effectiveness (random effects ration of odds rations ROR=1.02; 95% CI=0.83-1.26). For CAM interventions, however, language-restricted systematic reviews resulted in a 63% smaller protective effect estimate than language-inclusive reviews (random effects ROR=1.63; 95% CI=1.03-2.60).

CONCLUSION

Language restrictions do not change the results of CM systematic reviews but do substantially alter the results of CAM systematic reviews. These findings are robust even after sensitivity analyses, and do not appear to be influenced by statistical heterogeneity and publication bias.

Systematic reviews involving complementary and alternative medicine interventions had higher quality of reporting than conventional medicine reviews

OBJECTIVE

To compare the quality of systematic reviews reported in English and in languages other than English, and to determine whether there are differences between conventional medicine (CM) and complementary and alternative medicine (CAM) reports.

STUDY DESIGN AND SETTING

We used the Oxman and Guyatt (OG) scale to assess the quality of reporting in 130 systematic reviews: 50 were language-restricted, 32 were language-inclusive but only English-language (EL) trials contained (inclusive-EL), and 48 were language-inclusive and included trials published in languages other than English (inclusive-LOE). Of the 130 reviews, 105 addressed CM interventions and 25 addressed CAM interventions.

RESULTS

Comparison of the systematic reviews showed that the quality of reporting and reporting characteristics are not affected by inclusion or exclusion of LOE; however, the quality of reporting of systematic reviews involving CAM interventions is higher than that of reviews focusing on CM interventions.

CONCLUSION

Informal comparison of the OG scale with the data collected on quality assessments showed that the OG scale performs well overall but may not identify important differences in comprehensiveness of the search strategy and avoidance of bias in study selection. Further research is required to determine the best methods for assessing quality of systematic reviews and whether the effect of language restrictions is dependent on the type of intervention (CM or CAM).

Consumer food safety knowledge, practices, and demographic differences: findings from a meta-analysis

Risk communication and consumer education to promote safer handling of food can be the best way of managing the risk of foodborne illness at the consumer end of the food chain. Thus, an understanding of the overall status of food handling knowledge and practices is needed. Although traditional qualitative reviews can be used for combining information from several studies on specific food handling behaviors, a structured approach of meta-analysis can be more advantageous in a holistic assessment. We combined findings from 20 studies using meta-analysis methods to estimate percentages of consumers engaging in risky behaviors, such as consumption of raw food, poor hygiene, and cross-contamination, separated by various demographic categories. We estimated standard errors to reflect sampling error and between-study random variation. Then we evaluated the statistical significance of differences in behaviors across demographic categories and across behavioral measures. There were considerable differences in behaviors across demographic categories, possibly because of socioeconomic and cultural differences. For example, compared with women, men reported greater consumption of raw or undercooked foods, poorer hygiene, poorer practices to prevent cross-contamination, and less safe defrosting practices. Mid-age adults consumed more raw food (except milk) than did young adults and seniors. High-income individuals reported greater consumption of raw foods, less knowledge of hygiene, and poorer cross-contamination practices. The highest raw ground beef and egg consumption and the poorest hygiene and cross-contamination practices were found in the U.S. Mountain region. Meta-analysis was useful for identifying important data gaps and demographic groups with risky behaviors, and this information can be used to prioritize further research.

An application of meta-analysis in food safety consumer research to evaluate consumer behaviors and practices

Meta-analysis provides a structured method for combining results from several studies and accounting for and differentiating between study variables. Numerous food safety consumer research studies often focus on specific behaviors among different subpopulations but fail to provide a holistic picture of consumer behavior. Combining information from several studies provides a broader understanding of differences and trends among demographic subpopulations, and thus, helps in developing effective risk communication messages. In the illustrated example, raw/undercooked ground beef consumption and hygienic practices were evaluated according to gender, ethnicity, and age. Percentages of people engaging in each of the above behaviors (referred to as effect sizes) were combined using weighted averages of these percentages. Several measures, including sampling errors, random variance between studies, sample sizes of studies, and homogeneity of findings across studies, were used in the meta-analysis. The statistical significance of differences in behaviors across demographic segments was evaluated using analysis of variance. The meta-analysis identified considerable variability in effect sizes for raw/undercooked ground beef consumption and poor hygienic practices. More males, African Americans, and adults between 30 and 54 years (mid-age) consumed raw/undercooked ground beef than other demographic segments. Males, Caucasians, and Hispanics and young adults between 18 and 29 years were more likely to engage in poor hygienic practices. Compared to traditional qualitative review methods, meta-analysis quantitatively accounts for interstudy differences, allows greater consideration of data from studies with smaller sample sizes, and offers ease of analysis as newer data become available, and thus, merits consideration for its application in food safety consumer research.

Visual acuity following cataract surgeries in relation to preoperative appropriateness ratings

The authors initiated this study to consider if the formal preoperative assessment of appropriate or inappropriate utilization of cataract surgery by an expert panel could predict postoperative improvement or decline in visual acuity. They evaluated the association between ratings of appropriateness, as determined by the RAND-UCLA method, and measurements of postoperative visual acuity using Fisher's exact tests for tables greater than 2 x 2. For 768 patients, improvement of at least 2 Snellen chart lines occurred in 89% of surgeries rated as appropriate or appropriate and crucial, 68% rated as uncertain, and 36% rated as inappropriate (P < 0.0001, Fisher's exact test). These results provide evidence supporting the validity of the RAND-UCLA method to assess the appropriateness of surgery.

A meta-analysis of peritoneal lavage for acute pancreatitis

BACKGROUND AND AIM

It has been postulated that continuous irrigation of the peritoneal cavity with crystalloid solutions in patients with acute pancreatitis can improve mortality and morbidity. The aim of the study is to perform a meta-analysis of available randomized prospective clinical trials, to evaluate whether lavage influences mortality and morbidity in patients with acute pancreatitis.

METHODS

We performed a computer search of Medline for all available literature on the use of lavage in patients with acute pancreatitis. A meta-analysis was conducted on eight randomized, prospective, clinical trials (a total of 333 patients) evaluating continuous peritoneal lavage in patients with acute pancreatitis. The end-points were mortality and morbidity (i.e. pancreatic necrosis, peripancreatic fluid collections, intra-abdominal abscess formation, septicemia, organ system failure).

RESULTS

Continuous lavage did not improve either mortality (weighted mean difference 1.6%, 95% CI -6.7% to 9.9%, not significant (n.s.)) or morbidity (weighted mean difference 6.2%, 95% CI -3.2% to 15.6%, n.s.) when compared with control patients.

CONCLUSIONS

The use of continuous peritoneal lavage in patients with acute pancreatitis has not been found to be associated with any significant improvement in mortality or morbidity.

Treatment outcomes for students with learning disabilities: how important are internal and external validity?

This study analyzed the magnitude of experimental intervention outcomes as a function of violations in internal and external validity for studies that included students with learning disabilities. The results indicated that treatment outcomes were significantly affected by the following violations: teacher effects, establishing criterion levels of instructional performance, reliance on experimental measures, using different measures between pretest and posttest, using a sample heterogenous in age, and using incorrect units of analysis. Furthermore, the underreporting of information related to ethnicity, locale of the study, psychometric data, and teacher applications positively inflated the magnitude of treatment outcomes. A weighted hierarchical regression analysis revealed that composite scores of the aforementioned high-risk variables accounted for 16% of the total variance in effect size. The implications for interpreting intervention research to practice are discussed.

What contributions do languages other than English make on the results of meta-analyses?

Including only a portion of all available evidence may introduce systematic errors into the meta-analytic process and threaten its validity. We set out to examine whether language restricted meta-analyses, compared to language inclusive meta-analyses, provide different estimates of the effectiveness of interventions evaluated in randomized trials. We identified and retrieved all 79 meta-analyses from several disease areas in which explicit eligibility criteria regarding trial selection were reported. General characteristics and quality of reporting of the meta-analyses were assessed using a validated instrument. We explored the effects of language of publication of the randomized trials on the quantitative results using logistic regression analyses. Language restricted meta-analyses, compared to language inclusive meta-analyses, did not differ with respect to the estimate of benefit of the effectiveness of an intervention (ROR = 0.98; 95% CI: 0.81-1.17). These results were also robust after a series of sensitivity analyses. This study provides no evidence that language restricted meta-analyses lead to biased estimates of intervention effectiveness. We encourage others to replicate this study using different sampling frames, clinical topics and interventions.

Systematic review and meta-analysis in anatomic pathology

Systematic reviews and meta-analyses are techniques of data retrieval and analysis which complement traditional narrative reviews. They are widely used in clinical medicine and are finding an increasing role in anatomical pathology. Performing high quality systematic review and meta-analysis requires the accumulation of large numbers of cases from well planned and executed studies and is facilitated if data is presented in a standardized manner. Techniques which allow data from individual patients included in a variety of different studies are now being developed indicating that in future research papers may require a more detailed description of results than in the past. This need may be met by posting anonymised data on the Internet. Systematic reviews and meta-analyses are never complete since data are continually contributed and analyses constantly updated. As with any research paper, the results of these techniques require careful evaluation and the role of the expert reviewer is enlarged by these methodologies.

What is the quality of the implemented meta-analytic procedures in chronic pain treatment meta-analyses?

OBJECTIVE

Meta-analysis (MA) is the application of quantitative techniques for the purposes of summarizing data from individual studies. This type of review has many advantages over traditional reviews. However, different investigators performing MAs on the same data set have reached different conclusions. These reliability problems have been attributed to differences in the quality of the implemented meta-analytic procedures. We, therefore, examined the chronic pain treatment meta-analytic literature for MA procedure quality and for the consistency of conclusions. DESIGN, SETTING, PARTICIPANTS, OUTCOME MEASURES: Chronic pain treatment MAs were isolated according to inclusion/exclusion criteria. Data from these MAs were abstracted into structured tables. Table format reflected eight meta-analytic procedures identified previously as being important to MA implementation quality. These were: adequacy of retrieval, publication bias, inclusion/exclusion criteria, abstraction of data, quality, homogeneity/heterogeneity, independence, and statistical versus clinical interpretation. Each meta-analytic procedure was then independently rated by two raters. Rating results were then analyzed by procedure for each individual MA for percentage scores out of 100%, and mean scores. For MAs addressing the same topic area (pain facility treatment, antidepressant treatment, manipulation treatment) direction of effect size was noted. Mean effect sizes were calculated for these subgroups.

RESULTS

Sixteen chronic pain treatment MAs fulfilled inclusion/exclusion criteria. Mean procedure ratings indicated that four procedures may not be implemented adequately. These were publication bias, abstraction of data, quality, and homogeneity/heterogeneity. There was wide MA implementation score variability, with 37.5% scoring less than 50%. The effect sizes of the MA subgroups demonstrated replicate nonvariability.

CONCLUSIONS

Some meta-analytic procedures could be interpreted to be implemented inadequately in some chronic pain treatment MAs. There is wide variability between individual chronic pain treatment MAs on adequacy of implementation of these procedures. However, the effect sizes of the different MA subgroups demonstrated consistency. This finding indicates that for these MA subgroups, MA results are consistent between authors. In addition, chronic pain MAs, as compared with other groups of MAs, appear to address some of the procedures in a more adequate fashion. Future chronic pain MAs should concentrate on improving the quality of their methods with particular emphasis on the above four procedures. Because of potential validity problems with these results, these data cannot and should not be used to make administrative decisions about previous MAs.

Serendipity in diagnostic imaging: magnetic resonance imaging of the breast

BACKGROUND

Magnetic resonance imaging (MRI) of the breast has been proposed as a noninvasive diagnostic test for evaluation of suspicious ("index") lesions noted on mammography and/or clinical breast examination (CBE). However, women may have incidental ("serendipitous") lesions detected by MRI that are not found on mammography or CBE. To understand better whether or not biopsy procedures should be performed to evaluate serendipitous lesions, we estimated the breast cancer risk for women with this type of lesion.

METHODS

A decision analysis model was used to estimate the positive predictive value (i.e., the chance that a woman with a serendipitous lesion has cancer) of MRI for serendipitous lesions in women who had an abnormal mammogram and/or CBE suspicious for cancer (where a biopsy procedure is recommended). We restricted the analysis to data from women whose index lesions were noncancerous and used meta-analysis of published medical literature to determine the likelihood ratios (measures of how test results change the probability of having cancer) for MRI and the combination of CBE and mammography. The positive predictive value of MRI was calculated using the U.S. population prevalence of cancer (derived from registry data) and the likelihood ratios of the diagnostic tests.

RESULTS

Under a wide variety of assumptions, the positive predictive value of MRI was extremely low for serendipitous lesions. For instance, assuming sensitivity and specificity values for MRI of 95.6% and 68.6%, respectively, approximately four of 1000 55- to 59-year-old women with serendipitous lesions would be expected to have cancer (positive predictive value = 0.44%, 95% confidence interval = 0.24%-0.67%).

CONCLUSION

In women with a suspicious lesion discovered by mammography and/or CBE that is found to be benign, serendipitous breast lesions detected by MRI are extremely unlikely to represent invasive breast cancer. Immediate biopsy of such serendipitous lesions may, therefore, not be required.

The reproducibility of a method to identify the overuse and underuse of medical procedures

BACKGROUND

To assess the overuse and underuse of medical procedures, various methods have been developed, but their reproducibility has not been evaluated. This study estimates the reproducibility of one commonly used method.

METHODS

We performed a parallel, three-way replication of the RAND-University of California at Los Angeles appropriateness method as applied to two medical procedures, coronary revascularization and hysterectomy. Three nine-member multidisciplinary panels of experts were composed for each procedure by stratified random sampling from a list of experts nominated by the relevant specialty societies. Each panel independently rated the same set of clinical scenarios in terms of the appropriateness of the relevant procedure on a risk-benefit scale ranging from 1 to 9. Final ratings were used to classify the procedure in each scenario as necessary or not necessary (to evaluate underuse) and inappropriate or not inappropriate (to evaluate overuse). Reproducibility was measured by overall agreement and by the kappa statistic. The criteria for underuse and overuse derived from these ratings were then applied to real populations of patients who had undergone coronary revascularization or hysterectomy.

RESULTS

The rates of agreement among the three coronary-revascularization panels were 95, 94, and 96 percent for inappropriate-use scenarios and 93, 92, and 92 percent for necessary-use scenarios. Agreement among the three hysterectomy panels was 88, 70, and 74 percent for inappropriate-use scenarios. Scenarios involving necessary use of hysterectomy were not assessed. The three-way kappa statistic to detect overuse was 0.52 for coronary revascularization and 0.51 for hysterectomy. The three-way kappa statistic to detect underuse of coronary revascularization was 0.83. Application of individual panels' criteria to real populations of patients resulted in a 100 percent variation in the proportion of cases classified as inappropriate and a 20 percent variation in the proportion of cases classified as necessary.

CONCLUSIONS

The appropriateness method is far from perfect. Appropriateness criteria may be useful in comparing levels of appropriate procedures among populations but should not by themselves be used to direct care for individual patients.

How to use a review article: prophylactic endoscopic sclerotherapy for esophageal varices. Evidence Based Critical Care Medicine Group

OBJECTIVE

To assess the validity of a meta-analysis about sclerotherapy for the primary prevention of bleeding from esophageal varices, to interpret the results, and discuss whether they apply in practice.

DATA SOURCES

Critical appraisal techniques for systematic reviews.

DATA EXTRACTION

Systematic reviews are distinct from narrative reviews in that they answer specific clinical questions, and have explicit and reproducible methods for searching, selecting, and appraising the primary studies, to create the most valid synthesis of the evidence.

DATA SYNTHESIS

Meta-analyses are systematic reviews containing a critical appraisal and statistical summary of individual study results and their confidence limits, whereas qualitative systematic reviews provide a narrative executive summary of study results.

CONCLUSIONS

High-quality systematic reviews are being used increasingly to guide practice, strengthening the link between research results and improved health outcomes. Understanding their strengths and limitations helps us to use them appropriately in practice.

Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away

This paper examines how threats to the validity of meta-analysis have been dealt with by clinical researchers employing this approach to literature review. Three validity threats were identified--mixing of dissimilar studies, publication bias, and inclusion of poor quality studies. Approaches to addressing these threats were evaluated for their effectiveness and popularity by surveying 32 published meta-analyses in clinical psychology. Distrust of meta-analysis, however, was found to transcend these validity threats. Other explanations for why this popular research strategy continues to receive widespread criticism were considered. Suggestions were made for how meta-analysis might better address these concerns.

Do authors of review articles use systematic methods to identify, assess and synthesize information?

PURPOSE

Review articles are an important source of summary information for practising clinicians to assist them in remaining current with the rapidly expanding medical literature. Consequently it is essential that these be of the highest quality. In this study we evaluate, according to published criteria, the methodological quality of review articles (R) including meta-analysis (MA) appearing in a major cancer journal, Journal of Clinical Oncology (JCO), 1983-1995.

METHODS

A hand-search of JCO was performed, from the first issue January 1983 through December 1995, to identify R, defined as publications that describe and comment on studies relevant to a specific topic or clinical intervention. Only those dealing with aspects of treatment of human cancer were considered further. Methodological quality was first assessed using 8 criteria proposed by Mulrow, rated independently by two medical oncologists as: specified, unclear or not specified. MA, including studies of dose intensity, were further analyzed according to 23 more detailed criteria proposed by Sacks et al. and rated as adequate, partial or no/unknown compliance.

RESULTS

Of 176 review articles, 122 dealt with aspects of treatment of cancer. Compliance with four of Mulrow's eight criteria was generally good, in that 99% clearly stated a purpose, all attempted qualitative synthesis of data. 95% presented a summary and 76% considered future directions. However, in the 106 qualitative reviews (QR), authors rarely gave information on methods of data identification (11.3%), data selection (10.4%) and assessment of validity (8.4%). Structured abstracts seemed to improve the focus and clarity of QR and there was a minor improvement in deficient areas in the later time cohort (1990-1995). Based on 'adequate' compliance with each of the 23 criteria identified by Sacks et al., six dose intensity studies scored 7-12, seven literature data MA scored 10-15 and three individual patient data MA scored 16-18. The highest scores were in the sections relating to prospective design, combinability and statistical analysis. Factors relating to control of bias, sensitivity analysis and application of results were addressed less consistently.

CONCLUSIONS

With the exception of MA, the majority of authors contributing reviews to a major cancer journal, JCO, did not use systematic methods to identify, assess and synthesize information. Initiatives such as the Cochrane Collaboration Cancer Network can support and educate clinicians who wish to perform systematic reviews, but quality of reviews would also improve if author, editors and readers systematically applied any of the sets of criteria now available in the literature.

[Methods of investigation in clinical cardiology. VIII. Meta-analysis and systematic reviews in cardiology]

Over the last 20 years, the development of meta-analysis has been aimed at obtaining objective synthesis of the available results on specific research questions. The main achievements of meta-analysis include the application of techniques to perform systematic literature searches and to obtain unbiased selection of studies, data extraction and pooled estimates of effect. This paper discusses the methodologic steps to follow when conducting a meta-analysis, with emphasis on study selections, data collection and statistical methods to combine the results from individual studies. We also present a set of guided questions as an aid to critically evaluate the conclusions of published meta-analyses. The application of meta-analytic techniques to cardiology is illustrated using a meta-analysis of the randomized controlled trials of angioplasty versus bypass surgery in the management of patients with ischemic heart disease.

Meta-analysis and endocrinology

It is likely that more studies that use meta-analysis will be published in the endocrinologic literature. Major strengths of meta-analysis are the systematic ascertainment of research on a given topic and the explicit delineation of reasons for accepting or rejecting studies as a basis for drawing conclusions. The tendency of meta-analysis to focus on a single estimate of effect and to ignore heterogeneity are problems both with the conduct of meta-analysis and the way in which it is interpreted. Meta-analysis cannot overcome bias in the original studies. It is difficult to perform a good meta-analysis and easy to perform a bad one. The critical reader should not be overawed by the results of a meta-analysis. Reading a meta-analysis should not substitute for careful reading of the primary studies on which the meta-analysis is based. Meta-analysis should not be used to stifle the conduct of original research.

Meta-analysis and prospective meta-analysis in childhood leukemia clinical research

In this paper, we consider the role of meta-analysis and 'prospective meta-analysis' studies in childhood acute lymphocytic leukemia (ALL). In this issue, Valsecchi and Masera [1] give a thoughtful discourse, generally favorable to this approach. This article presents the opposite point of view. The aims of our article are to present the implications in clinical, rather than biostatistical terms, and to provide an extensive literature review of the subject of meta-analysis. We conclude that treatment assessments, resulting from meta-analysis of closed studies (retrospective) should be met with healthy skepticism. Trials requiring international resources should be true intergroup trials with a single coordinating center, rather than prospective meta-analysis, unless it is a question grafted onto each group's own research agenda. For example, each group might ask its own systemic control question, but a CNS protection question is asked collectively.