Combining studies using effect sizes and quality scores: application to bone loss in postmenopausal women

An examination of psychometric properties of study quality assessment scales in meta-analysis: Rasch measurement model applied to the firefighter cancer literature

Most existing quality scales have been developed with minimal attention to accepted standards of psychometric properties. Even for those that have been used widely in medical research, limited evidence exists supporting their psychometric properties. The focus of our current study is to address this gap by evaluating the psychometrics properties of two existing quality scales that are frequently used in cancer observational research: (1) Item Bank on Risk of Bias and Precision of Observational Studies developed by the Research Triangle Institute (RTI) International and (2) Newcastle-Ottawa Quality Assessment Scale (NOQAS). We used the Rasch measurement model to evaluate the psychometric properties of two quality scales based on the ratings of 49 studies that examine firefighters' cancer incidence and mortality. Our study found that RTI and NOQAS have an acceptable item reliability. Two raters were consistent in their assessment, demonstrating high interrater reliability. We also found that NOQAS has more items that show better fit than the RTI scale. The NOQAS produced lower study quality scores with a smaller variation, suggesting that NOQAS items are much easier to rate. Our findings accord with a previous study, which conclude that the RTI scale was harder to apply and thus produces more heterogenous quality scores than NOQAS. Although both RTI and NOQAS showed high item reliability, NOQAS items are better fit to the underlying construct, showing higher validity of internal structure and stronger psychometric properties. The current study adds to our understanding of the psychometric properties of NOQAS and RTI scales for future meta-analyses of observational studies, particularly in the firefighter cancer literature.

A 7-Step Guideline for Qualitative Synthesis and Meta-Analysis of Observational Studies in Health Sciences

Objectives: To provide a step-by-step, easy-to-understand, practical guide for systematic review and meta-analysis of observational studies. Methods: A multidisciplinary team of researchers with extensive experience in observational studies and systematic review and meta-analysis was established. Previous guidelines in evidence synthesis were considered. Results: There is inherent variability in observational study design, population, and analysis, making evidence synthesis challenging. We provided a framework and discussed basic meta-analysis concepts to assist reviewers in making informed decisions. We also explained several statistical tools for dealing with heterogeneity, probing for bias, and interpreting findings. Finally, we briefly discussed issues and caveats for translating results into clinical and public health recommendations. Our guideline complements "A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research" and addresses peculiarities for observational studies previously unexplored. Conclusion: We provided 7 steps to synthesize evidence from observational studies. We encourage medical and public health practitioners who answer important questions to systematically integrate evidence from observational studies and contribute evidence-based decision-making in health sciences.

Bias Assessment in Outcomes Research: The Role of Relative Versus Absolute Approaches

OBJECTIVES

Bias assessment tools vary in content and detail, and the method used for assessment may produce different assessment results in a study if not carefully considered. Therefore, taking an approach to the assessment of studies that produces a similar result regardless of the tool used for assessment (tool independence) is important.

METHODS

A preexisting study that used 25 different quality scales was assessed to examine tool dependence of 2 common approaches to bias assessments-absolute value judgments (defined as the qualitative risk of bias judgment based on a threshold across studies) and relative ranks (defined as the relative probability toward bias of a study relative to the best assessed study). Agreement between each of the 25 scales and a composite scale (that includes all unique safeguards across all scales) was computed (using the intraclass correlation coefficient [ICC]; consistency). Tool dependence was considered present when the ICCs were inconsistent across the 25 scales for the same study.

RESULTS

We found that using relative ranks for tools with different numbers and types of items produced consistent results, with only small differences in the agreement for the various tools with the composite tool, whereas consistency (measured by the ICC) varied considerably when using absolute judgments. Inconsistency is problematic because it means that the assessment result is linked to the scale and not to the study.

CONCLUSIONS

Tool independence is an important attribute of a bias assessment tool. On the basis of this study, the use of relative ranks retains tool independence and therefore produces consistent ranks for the same study across tools.

Comparison of bias adjustment methods in meta-analysis suggests that quality effects modeling may have less limitations than other approaches

BACKGROUND

The quality of primary research is commonly assessed before inclusion in meta-analyses. Findings are discussed in the context of the quality appraisal by categorizing studies according to risk of bias. The impact of appraised risk of bias on study outcomes is typically judged by the reader; however, several methods have been developed to quantify this risk of bias assessment and incorporate it into the pooled results of meta-analysis, a process known as bias adjustment. The advantages, potential limitations, and applicability of these methods are not well defined.

STUDY DESIGN AND SETTING

Comparative evaluation of the applicability of the various methods and their limitations are discussed using two examples from the literature. These methods include weighting, stratification, regression, use of empirically based prior distributions, and elicitation by experts.

RESULTS

Use of the two examples from the literature suggest that all methods provide similar adjustment. Methods differed mainly in applicability and limitations.

CONCLUSION

Bias adjustment is a feasible process in meta-analysis with several strategies currently available. Quality effects modelling was found to be easily implementable with fewer limitations in comparison to other methods.

Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis

BACKGROUND

Manual therapy (MT) and exercise have been extensively used to treat people with musculoskeletal conditions such as temporomandibular disorders (TMD). The evidence regarding their effectiveness provided by early systematic reviews is outdated.

PURPOSE

The aim of this study was to summarize evidence from and evaluate the methodological quality of randomized controlled trials that examined the effectiveness of MT and therapeutic exercise interventions compared with other active interventions or standard care for treatment of TMD.

DATA SOURCES

Electronic data searches of 6 databases were performed, in addition to a manual search.

STUDY SELECTION

Randomized controlled trials involving adults with TMD that compared any type of MT intervention (eg, mobilization, manipulation) or exercise therapy with a placebo intervention, controlled comparison intervention, or standard care were included. The main outcomes of this systematic review were pain, range of motion, and oral function. Forty-eight studies met the inclusion criteria and were analyzed.

DATA EXTRACTION

Data were extracted in duplicate on specific study characteristics.

DATA SYNTHESIS

The overall evidence for this systematic review was considered low. The trials included in this review had unclear or high risk of bias. Thus, the evidence was generally downgraded based on assessments of risk of bias. Most of the effect sizes were low to moderate, with no clear indication of superiority of exercises versus other conservative treatments for TMD. However, MT alone or in combination with exercises at the jaw or cervical level showed promising effects.

LIMITATIONS

Quality of the evidence and heterogeneity of the studies were limitations of the study.

CONCLUSIONS

No high-quality evidence was found, indicating that there is great uncertainty about the effectiveness of exercise and MT for treatment of TMD.

Non-pharmacological cancer pain interventions in populations with social disparities: a systematic review and meta-analysis

INTRODUCTION

Global advances in pain relief have improved the quality of life of cancer populations. Yet, variation in cancer pain outcomes has been found in populations with social disparities compared to mainstream groups. Populations with social disparities bear an inequitable distribution of resources such as ethnic minorities, low income individuals, and women in vulnerable circumstances.

RESEARCH PURPOSE

A systematic review and meta-analysis of the effect of non-pharmacological cancer pain interventions in cancer populations with social disparities of income, ethnicity, or gender.

METHODS

Randomized controlled trials, controlled trials, and before and after studies were targeted through comprehensive multidatabase searches. Two reviewers independently screened titles/abstracts for potentially relevant studies and reviewed the full text of relevant articles for inclusion. Data were extracted from included studies by one reviewer and verified by another reviewer. Four reviewers independently completed quality assessment. Studies were grouped by intervention. Effects were evaluated for heterogeneity and pooled.

RESULTS

The search found 5219 potential records. Full text of 26 reports was evaluated. Three randomized controlled trials (RCTs) met inclusion criteria, targeting ethnic minorities and underserved populations and/or women. Interventions included education, coaching, and online support groups. Studies found no significant differences in pain reduction between intervention and control groups or between ethnic minorities and their counterparts. A high risk of bias was found in all studies. Meta-analysis found no statistically significant difference on pain intensity among underserved groups, ethnic minorities, or between ethnic minorities and white counterparts.

CONCLUSION

Results show the need to examine supportive care interventions particularly in populations with social disparities.

Non-pharmacological cancer pain interventions in populations with social disparities: a systematic review and meta-analysis

INTRODUCTION

Global advances in pain relief have improved the quality of life of cancer populations. Yet, variation in cancer pain outcomes has been found in populations with social disparities compared to mainstream groups. Populations with social disparities bear an inequitable distribution of resources such as ethnic minorities, low income individuals, and women in vulnerable circumstances.

RESEARCH PURPOSE

A systematic review and meta-analysis of the effect of non-pharmacological cancer pain interventions in cancer populations with social disparities of income, ethnicity, or gender.

METHODS

Randomized controlled trials, controlled trials, and before and after studies were targeted through comprehensive multidatabase searches. Two reviewers independently screened titles/abstracts for potentially relevant studies and reviewed the full text of relevant articles for inclusion. Data were extracted from included studies by one reviewer and verified by another reviewer. Four reviewers independently completed quality assessment. Studies were grouped by intervention. Effects were evaluated for heterogeneity and pooled.

RESULTS

The search found 5219 potential records. Full text of 26 reports was evaluated. Three randomized controlled trials (RCTs) met inclusion criteria, targeting ethnic minorities and underserved populations and/or women. Interventions included education, coaching, and online support groups. Studies found no significant differences in pain reduction between intervention and control groups or between ethnic minorities and their counterparts. A high risk of bias was found in all studies. Meta-analysis found no statistically significant difference on pain intensity among underserved groups, ethnic minorities, or between ethnic minorities and white counterparts.

CONCLUSION

Results show the need to examine supportive care interventions particularly in populations with social disparities.

Traumatic Injury and Multiple Sclerosis: A Systematic Review and Meta-Analysis

A systematic review/meta-analysis of literature addressing a possible association between traumatic injury and onset of multiple sclerosis was conducted. Medline, Embase, Cochrane DSR, Ovid HealthStar, CINAHL, ISI Web of Science and Scopus were searched for analytical studies from 1950 to 2011. Two investigators independently reviewed articles for inclusion, assessing their quality using the Newcastle-Ottawa Scale. Of the 13 case-control studies included, 8 were moderate quality and 5 low; of the 3 cohort studies 2 were high and 1 moderate. Meta-analysis including moderate and low quality case-control studies produced a modest but significant odds ratio: 1.41 (95% confidence interval: 1.03, 1.93). However, when low quality studies were excluded, the resulting odds ratio was non-significant. Cohort studies produced a non-significant standardized incidence ratio of 1.00 (95% confidence interval: 0.86, 1.16). These findings support the conclusion that there is no association between traumatic injury and multiple sclerosis onset; more high quality cohort studies would help to confirm this observation.

Evidence synthesis for medical decision making and the appropriate use of quality scores

Meta-analyses today continue to be run using conventional random-effects models that ignore tangible information from studies such as the quality of the studies involved, despite the expectation that results of better quality studies reflect more valid results. Previous research has suggested that quality scores derived from such quality appraisals are unlikely to be useful in meta-analysis, because they would produce biased estimates of effects that are unlikely to be offset by a variance reduction within the studied models. However, previous discussions took place in the context of such scores viewed in terms of their ability to maximize their association with both the magnitude and direction of bias. In this review, another look is taken at this concept, this time asserting that probabilistic bias quantification is not possible or even required of quality scores when used in meta-analysis for redistribution of weights. The use of such a model is contrasted with the conventional random effects model of meta-analysis to demonstrate why the latter is inadequate in the face of a properly specified quality score weighting method.

A meta-analysis of the impact of short-term sleep deprivation on cognitive variables

A substantial amount of research has been conducted in an effort to understand the impact of short-term (<48 hr) total sleep deprivation (SD) on outcomes in various cognitive domains. Despite this wealth of information, there has been disagreement on how these data should be interpreted, arising in part because the relative magnitude of effect sizes in these domains is not known. To address this question, we conducted a meta-analysis to discover the effects of short-term SD on both speed and accuracy measures in 6 cognitive categories: simple attention, complex attention, working memory, processing speed, short-term memory, and reasoning. Seventy articles containing 147 cognitive tests were found that met inclusion criteria for this study. Effect sizes ranged from small and nonsignificant (reasoning accuracy: g = -0.125, 95% CI [-0.27, 0.02]) to large (lapses in simple attention: g = -0.776, 95% CI [-0.96, -0.60], p < .001). Across cognitive domains, significant differences were observed for both speed and accuracy; however, there were no differences between speed and accuracy measures within each cognitive domain. Of several moderators tested, only time awake was a significant predictor of between-studies variability, and only for accuracy measures, suggesting that heterogeneity in test characteristics may account for a significant amount of the remaining between-studies variance. The theoretical implications of these findings for the study of SD and cognition are discussed.

Meta-analysis of heterogeneous clinical trials: an empirical example

Meta-analysis of heterogeneous clinical trials is currently sub-optimal. This is because there has been no improvement in the method of weighted averaging for such studies since the DL method in 1986. This article presents the argument for the use of situation specific weights to integrate results from such trials. An empirical example is given with data from a meta-analysis done 10 years earlier. Previously reported data on 21 studies that looked at the effect of working conditions on preterm births were re-analyzed. Several methods were used to estimate the overall effect sizes. Study specific scores were included in the weighting process when combining studies and it was shown that this model not only was more conservative than the model of DL but also retains the legitimacy of the pooled effect size. The inclusion of appropriate study specific scores in an appropriate meta-analysis model permits the quantification of the variation between studies based on something tangible as opposed to the random adjustments made by the random effects model to the pooled effect size. It is important that such differences are recognized by the wider research community so that meta-analyses remain a valid tool for synthesizing research.

Effectiveness of interferential current therapy in the management of musculoskeletal pain: a systematic review and meta-analysis

BACKGROUND

Interferential current (IFC) is a common electrotherapeutic modality used to treat pain. Although IFC is widely used, the available information regarding its clinical efficacy is debatable.

PURPOSE

The aim of this systematic review and meta-analysis was to analyze the available information regarding the efficacy of IFC in the management of musculoskeletal pain.

DATA SOURCES

Randomized controlled trials were obtained through a computerized search of bibliographic databases (ie, CINAHL, Cochrane Library, EMBASE, MEDLINE, PEDro, Scopus, and Web of Science) from 1950 to February 8, 2010.

DATA EXTRACTION

Two independent reviewers screened the abstracts found in the databases. Methodological quality was assessed using a compilation of items included in different scales related to rehabilitation research. The mean difference, with 95% confidence interval, was used to quantify the pooled effect. A chi-square test for heterogeneity was performed.

DATA SYNTHESIS

A total of 2,235 articles were found. Twenty studies fulfilled the inclusion criteria. Seven articles assessed the use of IFC on joint pain; 9 articles evaluated the use of IFC on muscle pain; 3 articles evaluated its use on soft tissue shoulder pain; and 1 article examined its use on postoperative pain. Three of the 20 studies were considered to be of high methodological quality, 14 studies were considered to be of moderate methodological quality, and 3 studies were considered to be of poor methodological quality. Fourteen studies were included in the meta-analysis.

CONCLUSION

Interferential current as a supplement to another intervention seems to be more effective for reducing pain than a control treatment at discharge and more effective than a placebo treatment at the 3-month follow-up. However, it is unknown whether the analgesic effect of IFC is superior to that of the concomitant interventions. Interferential current alone was not significantly better than placebo or other therapy at discharge or follow-up. Results must be considered with caution due to the low number of studies that used IFC alone. In addition, the heterogeneity across studies and methodological limitations prevent conclusive statements regarding analgesic efficacy.

Meta-analysis of rare and adverse event data

Potentially beneficial drugs may have side effects. Such adverse events may be serious but rare, making their effect difficult to quantify, or even their identification problematic within randomized controlled trials. Meta-analysis provides a means of combining such outcome data across a series of studies and provides a more powerful analysis than is possible from a single study. However, meta-analysis of adverse event data presents some unique methodological challenges. Procedural issues pertaining to study selection and the inclusion of both randomized and observational data are discussed. The statistical problem of combining rare outcomes is also considered at length. Model choice, continuity corrections, exact statistics, Bayesian methods and sensitivity analysis are all covered. The application of results from adverse event meta-analyses to clinical and economic decision models is also considered. A number of examples are presented to illustrate the key points.

Systematic reviews of oral complications from cancer therapies, Oral Care Study Group, MASCC/ISOO: methodology and quality of the literature

BACKGROUND

Oral complications are commonly experienced by patients undergoing cancer therapies. The Oral Care Study Group of the Multinational Association of Supportive Care in Cancer/International Society of Oral Oncology (MASCC/ISOO) has completed nine systematic reviews including Bisphosphonate Osteonecrosis of the Jaw, Odontogenic/Periodontal Infection, Dysgeusia, Oral Fungal Infection, Osteoradionecrosis, Trismus, Oral Pain, Oral Viral Infection, and Xerostomia.

METHODS

The aims of these reviews were to determine the prevalence of each oral complication, relationship with quality of life, economic impact, and formulation of guidelines based on the quality of the literature. The present article described the details of the methodology and statistical analysis utilized in these nine systematic reviews. Additionally, a summary of the quality of the literature from these oral complications is presented.

CONCLUSION

Oral complications associated with cancer therapies are common among cancer patients. The systematic reviews by the Oral Care Study Group of MASCC/ISOO provide a thorough assessment of the available literature for these oral complications.

A quality-effects model for meta-analysis

We introduce a quality-effects approach that combines evidence from a series of trials comparing 2 interventions. This approach incorporates the heterogeneity of effects in the analysis of the overall interventional efficacy. However, unlike the random-effects model based on observed between-trial heterogeneity, we suggest adjustment based on measured methodological heterogeneity between studies. We propose a simple noniterative procedure for computing the combined effect size under this model and suggest that this could represent a more convincing alternative to the random effects model.

Cost-effectiveness of influenza vaccination in working-age cancer patients

BACKGROUND

Despite recommendations to immunize all patients at an increased risk of influenza complications, the vaccine utilization among high-risk nonelderly adults remains low and its cost-effectiveness is unclear. In the current study, the authors analyzed the cost-effectiveness of influenza vaccination in working-age (ages 20-64 years) cancer patients.

METHODS

The authors developed a decision-analytic model, from the societal perspective, using epidemiologic, vaccine effectiveness, resource utilization, cost, survival, and utility data from published sources, supplemented with data collected from the authors' own institutional accounting system. Two strategies were compared: influenza vaccination of working-age cancer patients and no vaccination. The base-case patient was assumed to be a 51-year-old cancer patient (the mean age for the National Cancer Institute's Surveillance, Epidemiology, and End Results [SEER] population of working-age patients within 5 years of cancer diagnosis).

RESULTS

The effectiveness of the influenza vaccine was 6.02 quality-adjusted life-years (QALYs) at a cost of $30.10. The effectiveness of the no vaccination strategy was 6.01 QALYs at a cost of $27.86. Compared with the no vaccination strategy, the incremental cost-effectiveness ratio of vaccinating working-age cancer patients would be $224.00 per QALY gained. Using the benchmark of $50,000 per QALY, the model was only sensitive to changes in cancer survival (threshold of 2.8 months).

CONCLUSIONS

The influenza vaccine is cost-effective for working-age cancer patients with a life expectancy of >or=3 months. All working-age cancer patients who are within 5 years of cancer diagnosis and have a life expectancy of at least 3 months should be vaccinated against influenza.

Perspectives on cancer therapy-induced mucosal injury: pathogenesis, measurement, epidemiology, and consequences for patients

BACKGROUND

A frequent complication of anticancer treatment, oral and gastrointestinal (GI) mucositis, threatens the effectiveness of therapy because it leads to dose reductions, increases healthcare costs, and impairs patients' quality of life. The Multinational Association of Supportive Care in Cancer and the International Society for Oral Oncology assembled an international multidisciplinary panel of experts to create clinical practice guidelines for the prevention, evaluation, and treatment of mucositis.

METHODS

The panelists examined medical literature published from January 1966 through May 2002, presented their findings at two separate conferences, and then created a writing committee that produced two articles: the current study and another that codifies the clinical implications of the panel's findings in practice guidelines.

RESULTS

New evidence supports the view that oral mucositis is a complex process involving all the tissues and cellular elements of the mucosa. Other findings suggest that some aspects of mucositis risk may be determined genetically. GI proapoptotic and antiapoptotic gene levels change along the GI tract, perhaps explaining differences in the frequency with which mucositis occurs at different sites. Studies of mucositis incidence in clinical trials by quality and using meta-analysis techniques produced estimates of incidence that are presented herein for what to our knowledge may be a broader range of cancers than ever presented before.

CONCLUSIONS

Understanding the pathobiology of mucositis, its incidence, and scoring are essential for progress in research and care directed at this common side-effect of anticancer therapies.

Meta-analyses in systematic reviews of randomized controlled trials in perinatal medicine: comparison of fixed and random effects models

There is a need for empirical work comparing the random effects model with the fixed effects model in the calculation of a pooled relative risk in the meta-analysis in systematic reviews of randomized controlled trials. Such comparisons are particularly important when trial results are heterogeneous. We considered 84 independent meta-analyses in which each trial included a set of different women/newborns. These meta-analyses were included in systematic reviews published in the Cochrane Library's pregnancy and childbirth module. Twenty-one of these 84 meta-analyses demonstrated statistical heterogeneity at p<0.10. The random effects model estimates showed wider confidence intervals, particularly in those meta-analyses showing heterogeneity in the trial results. The summary relative risk for the random effects model tended to show a larger protective treatment effect than the fixed effects model in the heterogeneous meta-analyses. In this set of meta-analyses, statistical evaluation of publication bias cannot be shown to account for heterogeneity. Our empirical conclusion is that there may be opposing effects if the random effects model is used in the meta-analysis of clinical trials showing heterogeneity in the results: stronger treatment effects reflected in the summary relative risk, but wider confidence intervals about this summary measure.

An illustrated guide to the methods of meta-analysis

Meta-analysis is now accepted as a necessary tool for the evaluation of health care. Such analyses have been carried out in virtually every area of medicine to evaluate a wide spectrum of health care interventions and policies. This paper has three broad aims: (1) to describe the basic principles of meta-analysis, using a meta-analysis of interventions intended to reduce hospital re-admission rates for illustration; (2) to consider threats to the internal validity of meta-analysis, and the measures which can be taken to minimize their impact; and (3) to present an overview of more specialist and developing methods for synthesizing data, with the intention of outlining the directions meta-analysis may take in the future. The methods used to synthesize studies, which take 'weighted averages' of effect sizes have been refined to a high degree, while the methods for dealing with threats to the validity of meta-analyses such as publication bias, and variations in quality of the primary studies, are at a less advanced stage. However, many consider this standard 'weighted average' approach to meta-analysis not to be 'state of the art' in at least some situations, where the use of more sophisticated methods, generally to explain variation in estimates from different studies and synthesize a broader base of evidence, would be advantageous. Currently, approaches which attempt to do this are mainly still in the experimental stage and, unfortunately, ideas which sound natural and appealing are often difficult to implement in practice. Clearly, it will be some time before they are used routinely, but significant steps have been made.

Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis

PURPOSE

This study estimates the inter-rater and test-retest reliability of Chalmers' quality score scale in the context of bone mass loss and fracture rate in postmenopausal women.

METHODS

An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss and fracture rate in postmenopausal women. Twenty articles were randomly selected and four raters independently assessed the quality of each article with Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names, journal, year of publication and source of funding. Raters were also asked to assess all 20 articles one more time, two months after the first evaluation. Intraclass (ICC) and test-retest correlation coefficients were calculated.

RESULTS

The overall inter-rater ICC was 0.66 [0.55, 0.79](95%). The overall test-retest reliability of Chalmers' scale was 0.81 [0.67, 0. 98](95%). When ratings were stratified according to articles' blinding status, blinded assessments generated a smaller inter-rater ICC than non-blinded assessments: 0.30 [0.17, 0.53](95%) vs. 0.80 [0. 71, 0.90](95%). In addition, analyzing sub-scales separately generated different estimates of reliability.

CONCLUSIONS

This study shows that the reliability of the quality scale developed by Chalmers substantially varies between sub-scales, and is highly dependent on articles' blinding status. The possibility of bias in rating non-blinded articles can not be ruled out. The reliability of the scale can also be dependent on the outcome studied.