Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement

Quality Assessment of Systematic Reviews Performed As An Undergraduate thesis in A Peruvian University: A Systematic Review

Objective

To determine the quality of systematic reviews submitted as a thesis in the Medical School of Ricardo Palma University.

Methods

We conducted a systematic review. We included systematic reviews submitted as theses from Ricardo Palma University, and we excluded narrative reviews, editorials, clinical experiments, and those with incomplete data. We performed a structured search on EMBASE, PubMed, Scopus, and Institutional Repository from the Ricardo Palma University and RENATI. The risk of bias assessment was performed through the AMSTAR-2 and the modified AMSTAR-2 tools. The primary outcome was review quality. A qualitative synthesis of the information was performed.

Results

One thousand four hundred eighty-seven theses were identified, and exclusion criteria were applied, whereby 11 theses were selected for review and thorough consultation. Of the 11 selected theses, and through the AMSTAR-2 and modified AMSTAR-2 tools, the findings reached were that 90.9% of the included theses presented critically low quality that was not modified even when the quality was reevaluated after its publication as a scientific article.

Conclusion

The systematic reviews presented as undergraduate thesis in the Medical School of Ricardo Palma University showed low and critically low quality. Improvement in systematic review training is required for both students and institutional advisors.

Identification of application and interpretation errors that can occur in pairwise meta-analyses in systematic reviews of interventions: a systematic review

OBJECTIVES

To generate a bank of items describing application and interpretation errors that can arise in pairwise meta-analyses in systematic reviews of interventions.

STUDY DESIGN AND SETTING

MEDLINE, Embase, and Scopus were searched to identify studies describing types of errors in meta-analyses. Descriptions of errors and supporting quotes were extracted by multiple authors. Errors were reviewed at team meetings to determine if they should be excluded, reworded, or combined with other errors, and were categorized into broad categories of errors and subcategories within.

RESULTS

Fifty articles met our inclusion criteria, leading to the identification of 139 errors. We identified 25 errors covering data extraction/manipulation, 74 covering statistical analyses, and 40 covering interpretation. Many of the statistical analysis errors related to the meta-analysis model (eg, using a two-stage strategy to determine whether to select a fixed or random-effects model) and statistical heterogeneity (eg, not undertaking an assessment for statistical heterogeneity).

CONCLUSION

We generated a comprehensive bank of possible errors that can arise in the application and interpretation of meta-analyses in systematic reviews of interventions. This item bank of errors provides the foundation for developing a checklist to help peer reviewers detect statistical errors.

The Evidence for Screening Older Adults for Nutrition Risk in Primary Care: An Umbrella Review

It is not known if nutrition risk screening of older adults should be a standard practice in primary care. The evidence in support of nutrition risk screening of older adults in primary care was examined and critically analyzed using an umbrella review. The peer reviewed and grey literature were searched for clinical practice guidelines (CPGs) and systematic reviews (SRs). Titles and abstracts were independently screened by the two authors. Resources were excluded if they did not apply to older adults, did not discuss nutrition/malnutrition risk screening, or were in settings other than primary care. Full texts were independently screened by both authors, resulting in the identification of six CPGs and three SRs that met the review criteria. Guidelines were appraised with the AGREE II tool and SRs with the AMSTAR 2 tool. The quality of the CPGs was high, while the quality of the SRs was low. The CPGs and SRs acknowledged a lack of high-quality research on the benefits of regular nutrition risk screening for older adults in primary care; however, CPGs recommended annual screening for older adults in primary care practices or other community settings. High-quality research investigating nutrition risk screening of older adults in primary care is needed.

Machine learning in pancreas surgery, what is new? literature review

Background

Machine learning (ML) is an inquiry domain that aims to establish methodologies that leverage information to enhance performance of various applications. In the healthcare domain, the ML concept has gained prominence over the years. As a result, the adoption of ML algorithms has become expansive. The aim of this scoping review is to evaluate the application of ML in pancreatic surgery.

Methods

We integrated the preferred reporting items for systematic reviews and meta-analyses for scoping reviews. Articles that contained relevant data specializing in ML in pancreas surgery were included.

Results

A search of the following four databases PubMed, Cochrane, EMBASE, and IEEE and files adopted from Google and Google Scholar was 21. The main features of included studies revolved around the year of publication, the country, and the type of article. Additionally, all the included articles were published within January 2019 to May 2022.

Conclusion

The integration of ML in pancreas surgery has gained much attention in previous years. The outcomes derived from this study indicate an extensive literature gap on the topic despite efforts by various researchers. Hence, future studies exploring how pancreas surgeons can apply different learning algorithms to perform essential practices may ultimately improve patient outcomes.

Efficacy of Surgical Masks Versus N95 Respirators for the Prevention of COVID-19 in Dental Settings: A Systematic Review

Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2. (SARS-CoV-2). It spreads mainly through saliva droplets or nasal discharge. Dentists are among the professionals with the greatest risk of contracting and transmitting COVID-19. We compared the efficacy of surgical masks versus N95 respirators in preventing COVID-19 infection in dental settings. PubMed, Scopus, Web of Science, and Cochrane Library databases were searched. Search terms corresponded to a predefined PICOS (patient/population, intervention, comparison, and outcomes) question. The risk of bias was evaluated using AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews-2), ROBIS (Risk of Bias in Systematic Reviews), and Health Evidence tools. A total of 191 articles were screened, and nine of them were further evaluated for eligibility, of which five articles (fulfilled the selection criteria) and were included in this study. Two studies concluded that surgical masks could provide equivalent protection to N95 respirators. Another study found that N95 respirators were superior to surgical masks. The fourth study found that better protection can be achieved when using surgical masks by the aerosol source than when the recipient uses an N95 respirator, while the last study concluded that surgical masks or N95 respirators alone do not provide full protection. Thus, according to this systematic review, N95 respirators provide better protection against COVID-19 infection compared to surgical masks.

User experience of applying AMSTAR 2 to appraise systematic reviews of healthcare interventions: a commentary

BACKGROUND

'A Measurement Tool to Assess Systematic Reviews, version 2' (AMSTAR 2) is a validated 16-item scale designed to appraise systematic reviews (SRs) of healthcare interventions and to rate the overall confidence in their results. This commentary aims to describe the challenges with rating of the individual items and the application of AMSTAR 2 from the user perspective.

DISCUSSION

A group of six experienced users (methodologists working in different clinical fields for at least 10 years) identified and discussed the challenges in rating of each item and the general use of AMSTAR 2 to appraise SRs. A group discussion was used to develop recommendations on how users could deal with the identified challenges. We identified various challenges with the content of items 2-16 and with the derivation of the overall confidence ratings on AMSTAR 2. These challenges include the need (1) to provide additional definitions (e.g., what constitutes major deviations from SR protocol on item 2), (2) to choose a rating strategy for multiple conditions on single items (e.g., how to rate item 5 if studies were selected in duplicate, but consensus between two authors was not reported), and (3) to determine rules for deriving the confidence ratings (e.g., what items are critical for such ratings). Based on these challenges we formulated specific recommendations for items 2-16 that AMSTAR 2 users could consider before applying the tool. Our commentary adds to the existing literature by providing the first in-depth examination of the AMSTAR 2 tool from the user perspective. The identified challenges could be addressed by additional decision rules including definitions for ambiguous items and guidance for rating of complex items and derivation of confidence ratings. We recommend that a team consensus regarding such decision rules is required before appraisal procedure begins.

TRIAL REGISTRATION

Not applicable.

The added value of family-centered rounds in the hospital setting: A systematic review of systematic reviews

BACKGROUND

Family engagement in care for adult inpatients may improve shared decision making in the hospital and the competence and preparedness of informal caregivers to take over the care at home. An important strategy to involve family members in hospital care processes is to include them in (ward) rounds, also called 'family-centered rounds'(FCRs).

OBJECTIVES

Summarize the evidence regarding the added value of FCRs from the perspectives of patients, family, and healthcare professionals.

METHODS

A review protocol was registered a priori with PROSPERO (number CRD42022320915). The electronic databases PubMed, CINAHL, and PsycInfo were searched for English-written systematic reviews with a focus on FCRs. The results and methods were presented in line with the PRISMA guidelines, and the methodological quality of the included reviews was assessed using the adapted version of the AMSTAR tool.

RESULTS

Of the 207 initial records, four systematic reviews were identified covering a total of 67 single studies, mainly performed in critical and pediatric care. Added values of FCR were described at review level, with references to single studies. All four systematic reviews reported an improvement in satisfaction among patients, family, and healthcare professionals, whereby satisfaction is linked to improved communication and interaction, improved situational understanding, inclusion of family in the decision-making process, and improved relationships within the care situation.

CONCLUSION

Although only limited research has been conducted on the value of FCRs in the adult non-critical care setting, and despite the existence of a variety of outcome measures, the results available from the pediatric and acute care setting are positive. The findings of the sole study in an adult non-critical patient population are in line with these results. Further research in adult non-critical care is required to verify its effects in this setting.

Evaluation of right ventricle systolic function after tetralogy of Fallot repair: A systematic review comparing cardiac magnetic resonance and global longitudinal strain

BACKGROUND

Most patients who undergo tetralogy of Fallot (TOF) repair experience late right ventricle (RV) dysfunction due to pulmonary valve regurgitation (PVR). Cardiac magnetic resonance (CMR) is the gold standard method for evaluating RV during follow-up. Global longitudinal strain (GLS) has been introduced as a novel method for the assessment of RV dysfunction. We aimed to compare the feasibility of GLS and CMR for assessing RV function after TOF repair.

METHODS

We systematically reviewed the English literature using PubMed, SciELO and Google Scholar for articles published between January 1, 2015, and December 31, 2020. Articles evaluating RV function comparing by GLS and CMR after TOF repair were included.

RESULTS

Nine studies including 465 patients were analyzed. Most patients were men (280; 60%), the male:female ratio was 1.5:1, and the age range was .8 to 57.7 years. The mean follow-up time was 6 to 32 months. The correlation between RV GLS and RV ejection fraction (EF) by CMR was negative for the articles and varied from moderate to strong (r = -.45, r = -.60, r = -.76).

CONCLUSION

Right ventricle GLS can be considered for routine follow-up of TOF repair patients, even though CMR remains the noninvasive gold standard method. Using a single parameter may not allow comparison of the accuracy of 3D RV EF by using CMR and GLS. Further studies with a larger number of patients undergoing TOF repair are required to evaluate the correlation between these examinations.

A systematic meta-review of patient-level predictors of psychological therapy outcome in major depressive disorder

BACKGROUND

Psychological therapies are effective for treating major depressive disorder, but current clinical guidelines do not provide guidance on the personalization of treatment choice. Established predictors of psychotherapy treatment response could help inform machine learning models aimed at predicting individual patient responses to different therapy options. Here we sought to comprehensively identify known predictors.

METHODS

EMBASE, Medline, PubMed, PsycINFO were searched for systematic reviews with or without meta-analysis published until June 2020 to identify individual patient-level predictors of response to psychological treatments. 3113 abstracts were identified and 300 articles assessed. We qualitatively synthesized our findings by predictor category (sociodemographic; symptom profile; social support; personality features; affective, cognitive, and behavioural; comorbidities; neuroimaging; genetics) and treatment type. We used the AMSTAR 2 to evaluate the quality of included reviews.

RESULTS

Following screening and full-text assessment, 27 systematic reviews including 12 meta-analyses were eligible for inclusion. 74 predictors emerged for various psychological treatments, primarily cognitive behavioural therapy, interpersonal therapy, and mindfulness-based cognitive therapy.

LIMITATIONS

A paucity of studies examining predictors of psychological treatment outcome, as well as methodological heterogeneities and publication biases limit the strength of the identified predictors.

CONCLUSIONS

The synthesized predictors could be used to supplement clinical decision-making in selecting psychological therapies based on individual patient characteristics. These predictors could also be used as a priori input features for machine learning models aimed at predicting a given patient's likelihood of response to different treatment options for depression, and may contribute toward the development of patient-specific treatment recommendations in clinical guidelines.

The Development of Artificial Intelligence in Hernia Surgery: A Scoping Review

Background

Artificial intelligence simulates human intelligence in machines that have undergone programming to make them think like human beings and imitate their activities. Artificial intelligence has dominated the medical sector to perform various patient diagnosis activities and improve communication between professionals and patients. The main goal of this study is to perform a scoping review to evaluate the development of artificial intelligence in all forms of hernia surgery except the diaphragm and upside-down hernia.

Methods

The study used the Preferred Reporting Items for Systematic and Meta-analyses for Scoping Review (PRISMA-ScR) to guide the structuring of the manuscript and fulfill all the requirements of every subheading. The sources used to gather data are the PubMed, Cochrane, and EMBASE databases, IEEE and Google and Google Scholar search engines. AMSTAR tool is the most appropriate for assessing the methodological quality of the included studies.

Results

The study exclusively included twenty articles, whereby seven focused on artificial intelligence in inguinal hernia surgery, six focused on abdominal hernia surgery, five on incisional hernia surgery, and two on AI in medical imaging and robotics in hernia surgery.

Conclusion

The outcomes of this study reveal a significant literature gap on artificial intelligence in hernia surgery. The results also indicate that studies focus on inguinal hernia surgery more than any other types of hernia surgery since the articles addressing the topic are more. The study implies that more research is necessary for the field to develop and enjoy the benefits associated with AI. Thus, this situation will allow the integration of AI in activities like medical imaging and surgeon training.

Traduction franco-canadienne de l’Assessment of Systematic Reviews Revised (AMSTAR 2) : validation transculturelle et fidélité interjuges

Objective: Produce a French-Canadian translation of AMSTAR 2, affirm its content validity, and examine interrater reliability. Methods: Based on Vallerand's methodological approach, we conducted forward and parallel inverse-translations. Subsequently, an expert panel evaluated the translations to create a preliminary experimental French-Canadian version. A second expert panel examined this version and proposed additional modifications. Twenty future health professionals then rated the second experimental version for ambiguity on a scale (from 1 to 7). The principal co-investigators then reviewed the problematic elements and proposed a pre-official version. To ascertain content validity, a final back-translation was conducted resulting in the official version. Four judges evaluated 13 systematic reviews using the official French-Canadian version of AMSTAR 2. The Kappa coefficient was used to evaluate interrater reliability. Results: This rigorous adaptation enabled the development of a Franco-Canadian version of AMSTAR 2. Its application demonstrated low ambiguity (mean 1.15; SD 0.26) as well as good overall interrater reliability (total κ > 0.64) across all items. Conclusion: The French-Canadian version of AMSTAR 2 can now support francophone clinicians, educators, and managers in Canada as they undertake evidence-based practice.

A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

BACKGROUND

AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions.

METHODS

Three reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet's agreement coefficient (AC1 for unweighted analysis and AC2 for weighted analysis).

RESULTS

Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher.

CONCLUSION

ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.

Effects of Virtual Reality Training on Upper Limb Function and Balance in Stroke Patients: Systematic Review and Meta-Meta-Analysis

Background

Virtual reality (VR) training is a promising intervention strategy that has been utilized in health care fields like stroke rehabilitation and psychotherapy. Current studies suggest that VR training is effective in improving the locomotor ability of stroke patients.

Objective

This is the first meta-meta-analysis of the effects of VR on motor function in stroke patients. This study aimed to systematically summarize and quantify the present meta-analyses results of VR training and produce high-quality meta-meta-analysis results to obtain a more accurate prediction.

Methods

We searched 4 online databases (Web of Science, Scopus, PubMed, and Chinese National Knowledge Infrastructure) for meta-analysis studies. After accounting for overlap, 10 studies (accounting for almost 550 stroke patients) were obtained. Based on the meta-meta-analysis of these patients, this study quantified the impact of VR training on stroke patients’ motor performance, mainly including upper limb function, balance, and walking ability. We combined the effects under the random effect model and pooled the estimates as standardized mean differences (SMD).

Results

The results of the meta-meta-analysis showed that VR training effectively improves upper limb function (SMD 4.606, 95% CI 2.733-6.479, P<.05) and balance (SMD 2.101, 95% CI 0.202-4.000, P<.05) of stroke patients. However, the results showed considerable heterogeneity and thus, may need to be treated with caution. Due to the limited research, a meta-meta-analysis of walking ability was not performed.

Conclusions

These findings represent a comprehensive body of high-quality evidence that VR training is more effective at improving upper limb function and balance of stroke patients.

Methodological Quality Analysis of Systematic Review for the Treatment of Rotator Cuff Disease

Objective  To evaluate the methodological quality of systematic reviews for the surgical and nonsurgical treatment of individuals with rotator cuff syndrome; to compare, through the Assessing the methodological quality of systematic reviews (AMSTAR) instrument, the quality of studies found in the Cochrane Library, PubMed (Publisher Medline ), EMBASE andQinsightdatabases. Methods  This is a descriptive and comparative cross-sectional study, in which two independent authors analyzed, through the AMSTAR instrument, the methodological quality of Cochrane and non-Cochrane systematic reviews on the treatment of individuals diagnosed with rotator cuff syndrome. Results  A total of 76 systematic reviews were evaluated by the AMSTAR instrument. The overall mean score was 6.1 (±2.1) and the mean per database was 9.1 (±0.9) for the Cochrane reviews and 5.7 (±1.8) for the non-Cochrane reviews. The lowest-scoring item of AMSTAR was 11, related to the display of the conflict of interests of the publication. In a comparative analysis of the final variable score, there was a statistical difference between the Cochrane and non-Cochrane studies. Conclusion  According to the present study, systematic reviews using the Cochrane methodology have a better methodological quality compared to non-Cochrane studies on the treatment of rotator cuff dysfunctions.

Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement - a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools

Background

Systematic Reviews (SRs) can build the groundwork for evidence-based health care decision-making. A sound methodological quality of SRs is crucial. AMSTAR (A Measurement Tool to Assess Systematic Reviews) is a widely used tool developed to assess the methodological quality of SRs of randomized controlled trials (RCTs). Research shows that AMSTAR seems to be valid and reliable in terms of interrater reliability (IRR), but the test retest reliability (TRR) of AMSTAR has never been investigated. In our study we investigated the TRR of AMSTAR to evaluate the importance of its measurement and contribute to the discussion of the measurement properties of AMSTAR and other quality assessment tools.

Methods

Seven raters at three institutions independently assessed the methodological quality of SRs in the field of occupational health with AMSTAR. Between the first and second ratings was a timespan of approximately two years. Answers were dichotomized, and we calculated the TRR of all raters and AMSTAR items using Gwet’s AC1 coefficient. To investigate the impact of variation in the ratings over time, we obtained summary scores for each review.

Results

AMSTAR item 4 (Was the status of publication used as an inclusion criterion?) provided the lowest median TRR of 0.53 (moderate agreement). Perfect agreement of all reviewers was detected for AMSTAR-item 1 with a Gwet’s AC1 of 1, which represented perfect agreement. The median TRR of the single raters varied between 0.69 (substantial agreement) and 0.89 (almost perfect agreement). Variation of two or more points in yes-scored AMSTAR items was observed in 65% (73/112) of all assessments.

Conclusions

The high variation between the first and second AMSTAR ratings suggests that consideration of the TRR is important when evaluating the psychometric properties of AMSTAR.. However, more evidence is needed to investigate this neglected issue of measurement properties. Our results may initiate discussion of the importance of considering the TRR of assessment tools. A further examination of the TRR of AMSTAR, as well as other recently established rating tools such as AMSTAR 2 and ROBIS (Risk Of Bias In Systematic reviews), would be useful.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01231-y.

Remission and low disease activity in psoriatic arthritis publications: a systematic literature review with meta-analysis

OBJECTIVES

Remission (REM) or low disease activity (LDA) is the treatment target in psoriatic arthritis (PsA). The objective of this study was to assess the reporting and prevalence of REM/LDA in published studies of PsA.

METHODS

This was a systematic literature review of all clinical papers published in PubMed, EMBASE or Cochrane database in English between 2012 and 2019 in the field of PsA. Data were collected regarding reporting of REM/LDA by very low disease activity/minimal disease activity (VLDA/MDA), Disease Activity index for Psoriatic Arthritis (DAPSA), or Disease Activity Score 28 joints (DAS28). The pooled rates of REM and LDA by each definition were calculated by random effect meta-analysis.

RESULTS

In all, 258 publications (corresponding to 114 651 patients), of which 81 (31%) were randomized controlled trials, were analysed: patients' mean age was 49.4 ( 4.4) years; with a mean disease duration of 8.5 ( 3.8) years. REM/LDA was reported in 91/258 (35.3%) publications. VLDA/MDA was used in 61/91 (67.0%) studies, DAPSA in 27/91 (29.6%) and DAS28 in 28/91 (30.7%), with 40/91 (43.9%) papers reporting several of these definitions. The pooled prevalence (lower-upper limits) of REM was 13.1% (10.9-15.4), 23.1% (16.8-30.1) and 42.1% (33.9-50.4) using VLDA, DAPSA-REM and DAS28, respectively. For LDA the pooled prevalence was 36.3% (32.3-40.5), 52.8% (41.8-63.6) and 60.4% (52.5-68.0) using MDA, DAPSA-LDA and DAS28, respectively.

CONCLUSION

REM/LDA status was reported in only1/3 of recent studies on PsA, with important variations in the frequency of these outcomes according to the definition used: 13.1-42.1% for REM, and 36.3-60.4% for LDA. This highlights the need for consensus.

The methodological quality of systematic reviews on the treatment of adult major depression needs improvement according to AMSTAR 2: A cross-sectional study

Background

Several standards have been developed to assess methodological quality of systematic reviews (SR). One widely used tool is the AMSTAR. A recent update - AMSTAR 2 - is a 16 item evaluation tool that enables a detailed assessment of SR that include randomised (RCT) or non-randomised studies (NRS) of healthcare interventions.

Methods

A cross-sectional study of SR on pharmacological or psychological interventions in major depression in adults was conducted. SR published during 2012–2017 were sampled from MEDLINE, EMBASE and the Cochrane Database of SR. Methodological quality was assessed using AMSTAR 2. Potential predictive factors associated with quality were examined.

Results

In rating overall confidence in the results of 60 SR four reviews were rated “high”, two were “moderate”, one was “low” and 53 were “critically low”. The mean AMSTAR 2 percentage score was 45.3% (standard deviation 22.6%) in a wide range from 7.1% to 93.8%. Predictors of higher quality were: type of review (higher quality in Cochrane Reviews), SR including only randomized trials and higher journal impact factor.

Limitations

AMSTAR 2 is not intended to be used for the generation of a percentage score.

Conclusions

According to AMSTAR 2 the overall methodological quality of SR on the treatment of adult major depression needs improvement. Although there is a high need for summarized information in the field of mental health, this work demonstrates the need to critically assess SR before using their findings. Better adherence to established reporting guidelines for SR is needed.

Treating schizophrenia: the quality of evidence behind treatment recommendations and how it can improve

OBJECTIVE

To assess the methodological and reporting quality of systematic reviews that comprise the American Psychiatric Association (APA) Practice Guideline for the Treatment of Patients with Schizophrenia and to determine the extent to which results from Cochrane systematic reviews published after guideline development would alter or confirm current recommendations.

PARTICIPANTS

Systematic reviews that underpinned recommendations in the APA guidelines and Cochrane systematic reviews.

MAIN OUTCOME

Three independent reviewers scored all systematic reviews referenced in the guideline for quality and reporting using AMSTAR and PRISMA checklist, respectively. Items in both tools were individually graded and compared to identify consistently low-performing areas within the systematic reviews. Post hoc analysis of the Cochrane systematic reviews since the latest revision of APA's guidelines were performed to determine whether their findings were congruent with recent recommendations.

RESULTS

The mean score of the 57 reviews on the PRISMA checklist was 70%. The mean AMSTAR score was 6.8, correlating with a moderate quality score. Post hoc analysis revealed that 171 Cochrane reviews had been published since the APA guideline release. Only half of the reviews of pharmacological interventions confirmed current recommendations.

CONCLUSIONS AND RELEVANCE

The methodological quality of the systematic reviews included in the APA guideline was deficient in key areas. Our study brings to light the importance of using high-quality evidence in the development of clinical practice guidelines. An updated APA guideline (last updated in 2009) is necessary to provide the highest quality treatment recommendations for clinicians in the management of schizophrenia.

TRIAL REGISTRATION NUMBER

UMIN-CTR, UMIN000023099.

Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools

BACKGROUND

Systematic reviews play a crucial role in clinical decision making and resource allocation and are expected to be unbiased and consistent. The aim of this study is a review of systematic reviews on the use of prophylactic mesh to prevent parastomal hernia (PH) formation using ROBIS and AMSTAR tools to assess the risk of bias and methodological quality.

METHODS

We included systematic reviews with or without meta-analysis of which the objective was to assess the use of a prophylactic mesh to prevent PH. A systematic search of the literature in five databases from inception until December 2017 was conducted. For each systematic review, methodologic quality and risk of bias were assessed using the AMSTAR and ROBIS tools, respectively. We estimated the inter-rater reliability for individual domains and for the overall methodological quality and risk of bias using Fleiss' k.

RESULTS

We identified 14 systematic reviews that met the inclusion criteria. Using the AMSTAR scale with a cutoff value, six reviews showed high methodologic quality and eight were of low quality. Using the ROBIS tool, the overall risk of bias was low in 50% of the reviews analyzed. In the remaining studies, the risk of bias was unclear.

CONCLUSIONS

The global evidence in favor of the use of a prophylactic mesh for preventing PH is not uniform regarding quality and risk of bias. Surgeons cannot be equally confident in the results of all systematic reviews published on this topic.

Assessing the Quality of Systematic Reviews in Healthcare Using AMSTAR and AMSTAR2

Abstract. The current study assessed the consistency between A Measurement Tool to Assess Systematic Reviews (AMSTAR) and its updated version (AMSTAR2) applied to the same systematic reviews in healthcare. Data from k = 10 systematic reviews were coded by two raters using AMSTAR and AMSTAR2. AMSTAR and AMSTAR2 perfectly agreed on a subset of nine individual items and strongly correlated based on the total scores (percentage scores: ρ = .84, p = .002, k = 10; absolute scores: Spearman ρ = .83, p = .003, k = 10). The overall review quality was medium to high on AMSTAR, while the overall confidence in the results was low to critically low on AMSTAR2 for the same systematic reviews. AMSTAR2 can identify the sources of strengths, weaknesses, and biases in systematic reviews. However, the interpretation of the overall confidence in results of systematic reviews requires additional guidelines for users.

Neighbourhood community life and health: A systematic review of reviews

Neighbourhood community life has been widely recognized as an important determinant of population health. This systematic review of reviews provides an overview of the evidence for the ecological correlation between neighbourhood community life and population health. Nine databases were searched from 2008 to 2018 in order to identify systematic reviews of studies examining the association between neighbourhood community life and population health in urban neighbourhoods within the Organisation for Economic Co-operation and Development countries. Two reviewers completed selection and data extraction, then assessed the methodological quality of reviews using the Measurement Tool to Assess Systematic Reviews. We identified three high quality reviews and five of moderate quality. The reviews vary in quality of methodology, concepts, and measures. Most of the reviews examined the influence of social cohesion, social capital, and social interactions on health. Reviews found evidence supporting a consistently favourable correlation between social cohesion and physical activity, as well as a favourable trend in the relationship between social cohesion and healthy weight. They also found evidence of a favourable trend in the correlation between social capital and healthy weight. Reviews identified studies supporting a consistently favourable correlation between social interaction and depression. We identify evidence of a positive association between neighbourhood community life and several population health outcomes. Future research should define and conceptualize neighbourhood community life factors and health indicators to improve the comparison between studies and the process of evidence synthesis. This will also enable policy makers to take appropriate decisions.

Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists

Introduction: Systematic reviews (SRs) are the backbone of evidence-based health care, but no gold standard exists to assess their methodological quality. Although the AMSTAR tool is accepted for analyzing the quality of SRs, the ROBIS instrument was recently developed. This study compared the capacity of both instruments to capture the quality of SRs of interventions for improving vaccination coverage.Methods: We conducted a comprehensive literature search in the Cochrane Library and PubMed. Two reviewers independently screened the search output, assessed study eligibility, and extracted data from eligible SRs; resolving differences through consensus. We conducted Principal Component Analysis (PCA) in Stata 14 to determine similarities and differences between AMSTAR and ROBIS.Results: A total of 2322 records were identified through the search and 75 full-text publications were assessed for eligibility, of which 57 met inclusion criteria. Using AMSTAR, we found 32%, 60% and 9% of SRs to have high, moderate and low quality, respectively. With ROBIS, we judged 74%, 14% and 12% of SRs to have low, unclear and high risk of bias. PCA showed that SRs with low risk of bias in ROBIS clustered together with SRs having high-quality in AMSTAR, and SRs with high risk of bias in ROBIS clustered with low-quality SRs in AMSTAR.Conclusions: Our findings suggest that there is an association between methodological quality and risk of bias in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.

A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool

OBJECTIVES

The objectives of this study were to determine the interrater reliability (IRR) of assessment of multiple systematic reviews (AMSTAR) 2 for reviews of pharmacological or psychological interventions for the treatment of major depression, to compare it to that of AMSTAR and risk of bias in systematic reviews (ROBIS), and to assess the convergent validity between the appraisal tools.

STUDY DESIGN AND SETTING

Two groups of four raters were each assigned one of two samples of 30 systematic reviews. All eight raters applied AMSTAR 2 to their sample. Each group also applied either AMSTAR or ROBIS. Fleiss' kappa and Gwet's AC₁ were calculated, and agreement between the tools was assessed.

RESULTS

The median kappa values as a measure of IRR indicated a moderate agreement for AMSTAR 2 (median = 0.51), a substantial agreement for AMSTAR (median = 0.62), and a fair agreement for ROBIS (median = 0.27). Validity results showed a positive association for AMSTAR and AMSTAR 2 (r = 0.91) as well as ROBIS and AMSTAR 2 (r = 0.84). For the overall rating, AMSTAR 2 showed a high concordance with ROBIS and a lower concordance with AMSTAR.

CONCLUSION

The IRR of AMSTAR 2 was found to be slightly lower than the IRR of AMSTAR and higher than the IRR of ROBIS. Validity measurements indicate that AMSTAR 2 is closely related to both ROBIS and AMSTAR.

A review of methods used for hazard identification and risk assessment of environmental hazards

BACKGROUND

Approximately one quarter of all deaths globally are attributed to living or working in an unhealthy environment, with household and ambient air pollution, along with exposures to ultraviolet radiation and chemicals amongst the leading causes. At present there are no international standards for assessing the risks of these environmental hazards. The use of heterogeneous methods to identify health risks from environmental hazards may reduce the level of confidence the public has in the conclusions that are made.

OBJECTIVES

To describe and compare the processes and methods used by national and international organisations that conduct hazard identification and/or risk assessment (HI/RA) of environmental hazards and to identify knowledge gaps to inform the development of future methods.

METHODS

We searched the websites of 19 organisations (ten national, five international and four World Health Organization (WHO) units) and extracted data from all relevant, publicly available resources which described the processes and methods used in HI/RA of environmental hazards. We contacted each organisation for any additional information.

RESULTS

Five organisations were excluded from further analysis: three made recommendations but did not conduct HI/RA; one used heterogenous methods across their reviews for HI; and one WHO unit did not have any published guidelines. Of the 14 organisations analysed, five (36%) describe the process for establishing the questions to be answered in the assessments. Only one (7%) organisation uses systematic review methods, although five (36%) state that they use such methods. Ten (71%) assess the scientific quality of the included studies, however only three (21%) use explicit criteria. Only three (21%) organisations assess the quality of the body of evidence using explicit criteria. Four (29%) organisations describe the process for making the final HI conclusions and three (38%) the final RA conclusions. Eight (57%) have a conflict of interest policy and seven (50%) organisations describe a process for managing them. The US Office of Health Assessment and Translation and the World Health Organisation meet the most criteria for describing their processes and methods.

CONCLUSIONS

The processes and methods used by organisations conducting HI/RA of environmental hazards are inconsistent. There is a need for empirically based tools and methods to be adopted for the evaluation and synthesis of evidence, and the formulation of conclusions across all organisations that conduct HI or RA. These tools and methods will lead to increased transparency, comparability and validity of the assessments.

Quality criteria, instruments, and requirements for nursing documentation: A systematic review of systematic reviews

AIM

To obtain an overview of existing evidence on quality criteria, instruments, and requirements for nursing documentation.

DESIGN

Systematic review of systematic reviews.

DATA SOURCES

We systematically searched the databases PubMed and CINAHL for the period 2007-April 2017. We also performed additional searches.

REVIEW METHODS

Two reviewers independently selected the reviews using a stepwise procedure, assessed the methodological quality of the selected reviews, and extracted the data using a predefined extraction format. We performed descriptive synthesis.

RESULTS

Eleven systematic reviews were included. Several quality criteria were described referring to the importance of following the nursing process and using standardized nursing terminologies. In addition, some evidence-based instruments were described for assessing the quality of nursing documentation, such as the D-Catch. Furthermore, several requirements for formats and systems of electronic nursing documentation were found that refer to the importance of user-friendliness and development in consultation with nursing staff.

CONCLUSION

Aligning documentation with the nursing process, using standard terminologies, and using user-friendly formats and systems appear to be important for high-quality nursing documentation. The lack of evidence-based quality indicators presents a challenge in the pursuit of high-quality nursing documentation.

IMPACT

There is uncertainty in nursing practice about which criteria have to be met to achieve high-quality documentation. Aligning documentation with the nursing process, using standard terminologies, and using user-friendly formats and systems appear to be important. These findings can help nursing staff and care organizations enhance the quality of nursing documentation.

Pharmacist-participated medication review in different practice settings: Service or intervention? An overview of systematic reviews

Introduction

Medication review (MR) is a pharmacy practice conducted in different settings that has a positive impact on patient health outcomes. In this context, systematic reviews on MR have restricted the assessment of this practice using criteria such as methodological quality, practice settings, and patient outcomes. Therefore, expanding research on this subject is necessary to facilitate the understanding of the effectiveness of MR and the comparison of its results.

Aim

To examine the panorama of systematic reviews on pharmacist-participated MR in different practice settings.

Methods

A literature search was undertaken in Biblioteca Virtual em Saúde (BVS), Embase, PubMed, Scopus, The Cochrane Library, and Web of Science databases through January 2018 using keywords for "medication review", "systematic review", and "pharmacist". Two independents investigators screened titles, abstracts, full texts; assessed methodological quality; and, extracted data from the included reviews.

Results

Seventeen systematic reviews were included, of which sixteen presented low to moderate methodological quality. Most of reviews were conducted in Europe (n = 7), included controlled primary studies (n = 16), elderly patients (n = 9), and long-term care facilities (n = 8). Seven reviews addressed MR as an intervention and thirteen reviews cited collaboration between physicians and pharmacists in the practice of MR. In addition, thirteen terminologies for MR were used and the main objective was to identify and solve drug-related problems and/or optimize the drug use (n = 11).

Conclusion

There is considerable heterogeneity in practice settings, population, definitions, terminologies, and approach of MR as well as poor description of patient care process in the systematic reviews. These facts may limit the comparison, summarization and understanding of the results of MR. Furthermore, the methodological quality of most systematic reviews was below ideal. Thus, international agreement on the MR process is necessary to assess, compare and optimize the quality of care provided.

Methodological quality of systematic reviews on treatments for depression: a cross-sectional study

AIMS

Depression is one of the most common mental disorders and identifying effective treatment strategies is crucial for the control of depression. Well-conducted systematic reviews (SRs) and meta-analyses can provide the best evidence for supporting treatment decision-making. Nevertheless, the trustworthiness of conclusions can be limited by lack of methodological rigour. This study aims to assess the methodological quality of a representative sample of SRs on depression treatments.

METHODS

A cross-sectional study on the bibliographical and methodological characteristics of SRs published on depression treatments trials was conducted. Two electronic databases (the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effects) were searched for potential SRs. SRs with at least one meta-analysis on the effects of depression treatments were considered eligible. The methodological quality of included SRs was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using logistic regression analysis.

RESULTS

A total of 358 SRs were included and appraised. Over half of included SRs (n = 195) focused on non-pharmacological treatments and harms were reported in 45.5% (n = 163) of all studies. Studies varied in methods and reporting practices: only 112 (31.3%) took the risk of bias among primary studies into account when formulating conclusions; 245 (68.4%) did not fully declare conflict of interests; 93 (26.0%) reported an 'a priori' design and 104 (29.1%) provided lists of both included and excluded studies. Results from regression analyses showed: more recent publications were more likely to report 'a priori' designs [adjusted odds ratio (AOR) 1.31, 95% confidence interval (CI) 1.09-1.57], to describe study characteristics fully (AOR 1.16, 95% CI 1.06-1.28), and to assess presence of publication bias (AOR 1.13, 95% CI 1.06-1.19), but were less likely to list both included and excluded studies (AOR 0.86, 95% CI 0.81-0.92). SRs published in journals with higher impact factor (AOR 1.14, 95% CI 1.04-1.25), completed by more review authors (AOR 1.12, 95% CI 1.01-1.24) and SRs on non-pharmacological treatments (AOR 1.62, 95% CI 1.01-2.59) were associated with better performance in publication bias assessment.

CONCLUSION

The methodological quality of included SRs is disappointing. Future SRs should strive to improve rigour by considering of risk of bias when formulating conclusions, reporting conflict of interests and authors should explicitly describe harms. SR authors should also use appropriate methods to combine the results, prevent language and publication biases, and ensure timely updates.

Interventions during pregnancy to prevent preterm birth: an overview of Cochrane systematic reviews

BACKGROUND

Preterm birth (PTB) is a major factor contributing to global rates of neonatal death and to longer-term health problems for surviving infants. Both the World Health Organization and the United Nations consider prevention of PTB as central to improving health care for pregnant women and newborn babies. Current preventative clinical strategies show varied efficacy in different populations of pregnant women, frustrating women and health providers alike, while researchers call for better understanding of the underlying mechanisms that lead to PTB.

OBJECTIVES

We aimed to summarise all evidence for interventions relevant to the prevention of PTB as reported in Cochrane systematic reviews (SRs). We intended to highlight promising interventions and to identify SRs in need of an update.

METHODS

We searched the Cochrane Database of Systematic Reviews (2 November 2017) with key words to capture any Cochrane SR that prespecified or reported a PTB outcome. Inclusion criteria focused on pregnant women without signs of preterm labour or ruptured amniotic membranes. We included reviews of interventions for pregnant women irrespective of their risk status. We followed standard Cochrane methods.We applied GRADE criteria to evaluate the quality of SR evidence. We assigned graphic icons to classify the effectiveness of interventions as: clear evidence of benefit; clear evidence of harm; clear evidence of no effect or equivalence; possible benefit; possible harm; or unknown benefit or harm. We defined clear evidence of benefit and clear evidence of harm to be GRADE moderate- or high-quality evidence with a confidence interval (CI) that does not cross the line of no effect. Clear evidence of no effect or equivalence is GRADE moderate- or high-quality evidence with a narrow CI crossing the line of no effect. Possible benefit and possible harm refer to GRADE low-quality evidence with a clear effect (CI does not cross the line of no effect) or GRADE moderate- or high-quality evidence with a wide CI. Unknown harm or benefit refers to GRADE low- or very low-quality evidence with a wide CI.

MAIN RESULTS

We included 83 SRs; 70 had outcome data. Below we highlight key results from a subset of 36 SRs of interventions intended to prevent PTB.

OUTCOME

preterm birthClear evidence of benefitFour SRs reported clear evidence of benefit to prevent specific populations of pregnant women from giving birth early, including midwife-led continuity models of care versus other models of care for all women; screening for lower genital tract infections for pregnant women less than 37 weeks' gestation and without signs of labour, bleeding or infection; and zinc supplementation for pregnant women without systemic illness. Cervical cerclage showed clear benefit for women with singleton pregnancy and high risk of PTB only.Clear evidence of harmNo included SR reported clear evidence of harm.No effect or equivalenceFor pregnant women at high risk of PTB, bedrest for women with singleton pregnancy and antibiotic prophylaxis during the second and third trimester were of no effect or equivalent to a comparator.Possible benefitFour SRs found possible benefit in: group antenatal care for all pregnant women; antibiotics for pregnant women with asymptomatic bacteriuria; pharmacological interventions for smoking cessation for pregnant women who smoke; and vitamin D supplements alone for women without pre-existing conditions such as diabetes.Possible harmOne SR reported possible harm (increased risk of PTB) with intramuscular progesterone, but this finding is only relevant to women with multiple pregnancy and high risk of PTB. Another review found possible harm with vitamin D, calcium and other minerals for pregnant women without pre-existing conditions.

OUTCOME

perinatal deathClear evidence of benefitTwo SRs reported clear evidence of benefit to reduce pregnant women's risk of perinatal death: midwife-led continuity models of care for all pregnant women; and fetal and umbilical Doppler for high-risk pregnant women.Clear evidence of harmNo included SR reported clear evidence of harm.No effect or equivalenceFor pregnant women at high risk of PTB, antibiotic prophylaxis during the second and third trimester was of no effect or equivalent to a comparator.Possible benefitOne SR reported possible benefit with cervical cerclage for women with singleton pregnancy and high risk of PTB.Possible harmOne SR reported possible harm associated with a reduced schedule of antenatal visits for pregnant women at low risk of pregnancy complications; importantly, these women already received antenatal care in settings with limited resources.

OUTCOMES

preterm birth and perinatal deathUnknown benefit or harmFor pregnant women at high risk of PTB for any reason including multiple pregnancy, home uterine monitoring was of unknown benefit or harm. For pregnant women at high risk due to multiple pregnancy: bedrest, prophylactic oral betamimetics, vaginal progesterone and cervical cerclage were all of unknown benefit or harm.

AUTHORS' CONCLUSIONS

Implications for practiceThe overview serves as a map and guide to all current evidence relevant to PTB prevention published in the Cochrane Library. Of 70 SRs with outcome data, we identified 36 reviews of interventions with the aim of preventing PTB. Just four of these SRs had evidence of clear benefit to women, with an additional four SRs reporting possible benefit. No SR reported clear harm, which is an important finding for women and health providers alike.The overview summarises no evidence for the clinically important interventions of cervical pessary, cervical length assessment and vaginal progesterone because these Cochrane Reviews were not current. These are active areas for PTB research.The graphic icons we assigned to SR effect estimates do not constitute clinical guidance or an endorsement of specific interventions for pregnant women. It remains critical for pregnant women and their healthcare providers to carefully consider whether specific strategies to prevent PTB will be of benefit for individual women, or for specific populations of women.Implications for researchFormal consensus work is needed to establish standard language for overviews of reviews and to define the limits of their interpretation.Clinicians, researchers and funders must address the lack of evidence for interventions relevant to women at high risk of PTB due to multiple pregnancy.

Quality assessment of systematic reviews on vertical bone regeneration

The aim of this study was to evaluate and compare the quality of systematic reviews of vertical bone regeneration techniques, using two quality-assessment tools (AMSTAR and ROBIS). An electronic literature search was conducted to identify systematic reviews or meta-analyses that would evaluate at least one of the following outcomes: implant survival, success rates, complications or bone gain after vertical ridge augmentation. Methodological quality assessment was performed by two independent evaluators. Results were compared between reviewers, and reliability measures were calculated using the Holsti's method^® and Cohen's kappa. Seventeen systematic reviews were included, of which seven presented meta-analysis. Mean ±95% confidence interval AMSTAR score was 6.35 [4.74;7.97], with higher scores being correlated with a smaller risk of bias (Pearson's correlation coefficient=-0.84; P<0.01). Cohen's inter-examiner kappa showed substantial agreement for both checklists. From the available evidence, we ascertained that, regardless of the technique used, it is possible to obtain vertical bone gains. Implant success in regenerated areas was similar to implants placed in pristine bone with results equating between 61.5% and 100% with guided bone regeneration being considered the most predictable technique regarding bone stability, while distraction osteogenesis achieved the biggest bone gains with the highest risk of possible complications.

The management strategies of cancer-associated anorexia: a critical appraisal of systematic reviews

BACKGROUND

Cancer-related anorexia remains one of the most prevalent and troublesome clinical problems experienced by patients with cancer during and after therapy. To ensure high-quality care, systematic reviews (SRs) are seen as the best guide. Considering the methodology quality of SRs varies, we undertook a comprehensive overview, and critical appraisal of pertinent SRs.

METHODS

Eight databases (between the inception of each database and September 1, 2017) were searched for SRs on the management of cancer-related anorexia. Two researchers evaluated the methodological quality of each SR by using the Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist. Characteristics of the "high quality" SRs were abstracted, included information on relevant studies numbers, study design, population, intervention, control, outcome and result.

RESULTS

Eighteen SRs met the inclusion criteria. The R-AMSTAR scores of methodological quality ranged from 18 to 41 out of 44, with an average score of 30. Totally eight SRs scored ≥31 points, which showed high methodological quality, and would be used for data extraction to make summaries. Anamorelin had some positive effects to relieve cancer anorexia-cachexia syndrome (CACS) and improve the quality of life (QoL). Megestrol Acetate (MA) could improve appetite, and was associated with slight weight gain for CACS. Oral nutritional interventions were effective in increasing nutritional intake and improving some aspects of QoL in patients with cancer who were malnourished or at nutritional risk. The use of thalidomide, Eicosapentaenoic Acid, and minerals, vitamins, proteins, or other supplements for the treatment of cachexia in cancer were uncertain, and there was inadequate evidence to recommend it to clinical practices, the same situation in Chinese Herb Medicine and acupuncture (acupuncture and related therapies were effective in improving QoL) for treating anorexia in cancer patients, warranting further RCTs in these areas.

CONCLUSIONS

Anamorelin, MA, oral nutrition interventions, and acupuncture could be considered to be applied in patients with cancer-related anorexia. Future RCTs and SRs with high quality on the pharmaceutical or non-pharmaceutical interventions of anorexia in cancer patients are warranted.

How is AMSTAR applied by authors - a call for better reporting

BACKGROUND

The assessment of multiple systematic reviews (AMSTAR) tool is widely used for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of randomized controlled trials (RCTs). Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what it has been validated.

METHODS

We searched MEDLINE (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR. Full-text studies were sought for all retrieved hits and screened by one reviewer. A second reviewer verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or by consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals.

RESULTS

We identified 247 studies. They included a median of 17 reviews (interquartile range (IQR): 8 to 47) per study. AMSTAR was modified in 23% (57/247) of studies. In most studies, an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes = 1) being the most frequent option (102/200; 51%). More than one third of the authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis, we compared overviews of reviews (n = 154) with the methodological publications (n = 93). The overviews of reviews were much less likely to mention both limitations with respect to study designs (if other studies other than RCTs were included in the reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35).

CONCLUSIONS

Authors, peer reviewers, and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ensure to have a methodological expert in their review team.

How is AMSTAR applied by authors - a call for better reporting

Background

The assessment of multiple systematic reviews (AMSTAR) tool is widely used for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of randomized controlled trials (RCTs). Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what it has been validated.

Methods

We searched MEDLINE (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR. Full-text studies were sought for all retrieved hits and screened by one reviewer. A second reviewer verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or by consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals.

Results

We identified 247 studies. They included a median of 17 reviews (interquartile range (IQR): 8 to 47) per study. AMSTAR was modified in 23% (57/247) of studies. In most studies, an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes = 1) being the most frequent option (102/200; 51%). More than one third of the authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis, we compared overviews of reviews (n = 154) with the methodological publications (n = 93). The overviews of reviews were much less likely to mention both limitations with respect to study designs (if other studies other than RCTs were included in the reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35).

Conclusions

Authors, peer reviewers, and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ensure to have a methodological expert in their review team.

Efficacy and safety of aripiprazole for the treatment of schizophrenia: an overview of systematic reviews

PURPOSE

To conduct an overview to summarize the efficacy and safety of aripiprazole for the treatment of schizophrenia.

METHODS

A literature search was performed in PubMed, the Cochrane Library, LILACS, and the Centre for Reviews and Dissemination, for articles published until March 31, 2017. We included systematic reviews with meta-analyses of randomized controlled trials assessing the efficacy, and/or the safety of aripiprazole, for patients with schizophrenia. Two authors independently performed the study selection, data extraction, and quality assessment. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach and the Risk of Bias in Systematic Review (ROBIS) tool were used to appraise the quality of evidence and the risk of bias in the reviews, respectively.

RESULTS

Fourteen studies fulfilled the inclusion criteria. Aripiprazole showed efficacy similar to that of both typical and atypical antipsychotic drugs (except olanzapine and amisulpride). Aripiprazole caused significantly lower weight gain and alterations in glucose and cholesterol levels, as compared to clozapine, risperidone, and olanzapine. In addition, aripiprazole caused significantly fewer general extrapyramidal side effects, less use of antiparkinsonian drugs, and akathisia, compared with typical antipsychotic drugs and risperidone. The overall quality of evidence in the reviews ranged from "very low" to "moderate," principally because of the risk of bias of original trials, inconsistency, and imprecision in the outcomes. According to the ROBIS tool, there are four reviews with "high" risk of bias and five with "unclear" risk of bias.

CONCLUSIONS

Aripiprazole exhibited efficacy similar to that of other antipsychotic drugs and a better safety profile than that of typical (i.e., less some extrapyramidal side effects) and atypical (i.e., less metabolic changes) antipsychotic drugs.

Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability

OBJECTIVES

The objective of the study was to assess the inter-rater reliability (IRR) of AMSTAR and ROBIS in judging individual domains and overall methodological quality/risk of bias of systematic reviews, the concurrent validity of the tools, and the time required to apply them.

STUDY DESIGN AND SETTING

This is a cross-sectional study. Five raters independently read 31 systematic reviews and applied AMSTAR and ROBIS. Fleiss' k for multiple raters for individual domains and overall methodological quality/risk of bias was calculated. Similar domains assessed by both tools and final scores were matched to explore the concurrent validity, using the Kendall tau correlation.

RESULTS

IRR ranged from fair to perfect for AMSTAR and from moderate to substantial for ROBIS. Kappa for overall quality/risk of bias was 0.73 (95% confidence interval [CI] 0.65-0.81) for AMSTAR and 0.64 (95% CI 0.54-0.74) for ROBIS. We judged most of the reviews at intermediate quality with AMSTAR (53%), while judgments were split in high (53%) and low (47%) risk of bias with ROBIS. The correlation between judgments on similar domains ranged from moderate to high, while it was fair on the overall judgment (K = 0.35, 95% CI 0.21-0.49). The mean time to complete ROBIS was about double that for AMSTAR.

CONCLUSION

AMSTAR and ROBIS offer similar IRR but differ in their construct and applicability.

AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

The number of published systematic reviews of studies of healthcare interventions has increased rapidly and these are used extensively for clinical and policy decisions. Systematic reviews are subject to a range of biases and increasingly include non-randomised studies of interventions. It is important that users can distinguish high quality reviews. Many instruments have been designed to evaluate different aspects of reviews, but there are few comprehensive critical appraisal instruments. AMSTAR was developed to evaluate systematic reviews of randomised trials. In this paper, we report on the updating of AMSTAR and its adaptation to enable more detailed assessment of systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. With moves to base more decisions on real world observational evidence we believe that AMSTAR 2 will assist decision makers in the identification of high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.

"Assessing the methodological quality of systematic reviews in radiation oncology: A systematic review"

OBJECTIVE

The objective of our study was to evaluate the methodological quality of systematic reviews and meta-analyses in Radiation Oncology.

METHODS

A systematic literature search was conducted for all eligible systematic reviews and meta-analyses in Radiation Oncology from 1966 to 2015. Methodological characteristics were abstracted from all works that satisfied the inclusion criteria and quality was assessed using the critical appraisal tool, AMSTAR. Regression analyses were performed to determine factors associated with a higher score of quality.

RESULTS

Following exclusion based on a priori criteria, 410 studies (157 systematic reviews and 253 meta-analyses) satisfied the inclusion criteria. Meta-analyses were found to be of fair to good quality while systematic reviews were found to be of less than fair quality. Factors associated with higher scores of quality in the multivariable analysis were including primary studies consisting of randomized control trials, performing a meta-analysis, and applying a recommended guideline related to establishing a systematic review protocol and/or reporting.

CONCLUSIONS

Systematic reviews and meta-analyses may introduce a high risk of bias if applied to inform decision-making based on AMSTAR. We recommend that decision-makers in Radiation Oncology scrutinize the methodological quality of systematic reviews and meta-analyses prior to assessing their utility to inform evidence-based medicine and researchers adhere to methodological standards outlined in validated guidelines when embarking on a systematic review.

Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool

OBJECTIVES

No gold standard exists to assess methodological quality of systematic reviews (SRs). Although Assessing the Methodological Quality of Systematic Reviews (AMSTAR) is widely accepted for analyzing quality, the ROBIS instrument has recently been developed. This study aimed to compare the capacity of both instruments to capture the quality of SRs concerning psoriasis interventions.

STUDY DESIGN AND SETTING

Systematic literature searches were undertaken on relevant databases. For each review, methodological quality and bias risk were evaluated using the AMSTAR and ROBIS tools. Descriptive and principal component analyses were conducted to describe similarities and discrepancies between both assessment tools.

RESULTS

We classified 139 intervention SRs as displaying high/moderate/low methodological quality and as high/low risk of bias. A high risk of bias was detected for most SRs classified as displaying high or moderate methodological quality by AMSTAR. When comparing ROBIS result profiles, responses to domain 4 signaling questions showed the greatest differences between bias risk assessments, whereas domain 2 items showed the least.

CONCLUSION

When considering SRs published about psoriasis, methodological quality remains suboptimal, and the risk of bias is elevated, even for SRs exhibiting high methodological quality. Furthermore, the AMSTAR and ROBIS tools may be considered as complementary when conducting quality assessment of SRs.

Methodological quality of systematic reviews referenced in clinical practice guidelines for the treatment of opioid use disorder

INTRODUCTION

With efforts to combat opioid use disorder, there is an increased interest in clinical practice guidelines (CPGs) for opioid use disorder treatments. No literature exists examining the quality of systematic reviews used in opioid use disorder CPGs. This study aims to describe the methodological quality and reporting clarity of systematic reviews (SRs) used to create CPGs for opioid use disorder.

METHODS

From June to July 2016 guideline clearinghouses and medical literature databases were searched for relevant CPGs used in the treatment of opioid use disorder. Included CPGs must have been recognized by a national organization. SRs from the reference section of each CPG was scored by using AMSTAR (a measurement tool to assess the methodological quality of systematic reviews) tool and PRISMA (preferred reporting items for systematic reviews and meta-analyses) checklist.

RESULTS

Seventeen CPGs from 2006-2016 were included in the review. From these, 57 unique SRs were extracted. SRS comprised 0.28% to 17.92% of all references found in the CPGs. All SRs obtained moderate or high methodological quality score on the AMSTAR tool. All reviews met at least 70% of PRISMA criteria. In PRISMA, underperforming areas included accurate title labeling, protocol registration, and risk of bias. Underperforming areas in AMSTAR included conflicts of interest, funding, and publication bias. A positive correlation was found between AMSTAR and PRISMA scores (r = .79).

CONCLUSION

Although the SRs in the CPGs were of good quality, there are still areas for improvement. Systematic reviewers should consult PRISMA and AMSTAR when conducting and reporting reviews. It is important for CPG developers to consider methodological quality as a factor when developing CPG recommendations, recognizing that the quality of systematic reviews underpinning guidelines does not necessarily correspond to the quality of the guideline itself.

Selecting and implementing overview methods: implications from five exemplar overviews

BACKGROUND

Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed.

METHODS

Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus.

RESULTS

Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews. Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified.

CONCLUSIONS

Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.