How is AMSTAR applied by authors - a call for better reporting

Most systematic reviews reporting adherence to AMSTAR 2 had critically low methodological quality: a cross-sectional meta-research study

OBJECTIVES

To analyze the methodological quality and characteristics of systematic reviews (SRs) that reported they were conducted in line with the AMSTAR 2 (A MeaSurement Tool to Assess Systematic Reviews).

STUDY DESIGN AND SETTING

This was a cross-sectional meta-research study. We searched MEDLINE and Embase. We included full reports of SRs reporting the study was conducted, prepared, or designed in line with the AMSTAR 2. Eligible SRs were those published from January 1, 2018, until May 3, 2022. We assessed the methodological quality of the included SRs using AMSTAR 2.

RESULTS

We included a total of 45 records. There were 43 SRs and 2 SR protocols. Among them, most were SRs of interventions that included primary studies on humans. More than half had a meta-analysis. According to our overall AMSTAR 2 assessments of included SRs, 35 SRs were of critically low confidence, 7 SRs were of low confidence, and one SR was of high confidence. There were no SRs of moderate confidence.

CONCLUSION

Even when authors indicate in their manuscripts that the SR was conducted/prepared/designed in line with the AMSTAR 2, it does not necessarily imply it is of high or even moderate confidence according to AMSTAR 2. A self-assessment with AMSTAR 2 could be required for submission and carefully checked by the editors/peer reviewers.

Low risk of erectile dysfunction after nontransecting bulbar urethroplasty for urethral stricture: a systematic review and meta-analysis

BACKGROUND

Sexual function after urethroplasty may be a concern for patients, but there are still some controversies regarding the consequences of nontransecting bulbar urethroplasty (ntBU) in terms of erectile dysfunction (ED).

AIM

This meta-analysis aimed to compare the efficacy and safety of ntBU with that of transecting bulbar urethroplasty (tBU).

METHODS

The PubMed, Web of Science, Cochrane, and Embase databases were searched and reviewed up to October 31, 2022. Quality evaluation was performed using the Newcastle-Ottawa scale system and Cochrane tools for the nonrandomized and randomized studies, respectively. Baseline characteristics, preoperative information, and postoperative outcomes were collected.

OUTCOMES

Outcomes included success rate, ED, overall complication, and maximum urinary flow.

RESULTS

Thirteen studies comprising 1683 patients met the inclusion criteria, with 596 and 1087 patients undergoing ntBU and tBU, respectively. The results revealed that compared with the tBU group, the patients who underwent ntBU had a significantly lower incidence of ED, while there were no significant differences in the other perioperative outcomes. In subgroup analysis, the nontransecting anastomotic urethroplasty group had a lower incidence of ED than excision and primary anastomosis, and other perioperative outcomes were similar between the 2 groups.

CLINICAL IMPLICATIONS

The results of the study may help clinicians choose procedures that protect sexual function in the treatment of urethral stricture.

STRENGTHS AND LIMITATIONS

The strength of this study is that it is, to our knowledge, the first meta-analysis to evaluate the efficacy and safety of ntBU. A limitation is that most of the included studies were retrospective cohort studies.

CONCLUSION

ntBU preserves the high efficacy of its transecting counterpart while reducing postoperative ED.

AMSTAR 2 is only partially applicable to systematic reviews of non-intervention studies: a meta-research study

OBJECTIVES

A measurement tool to assess systematic reviews 2 (AMSTAR 2) was originally developed for systematic reviews (SRs) of health-care interventions. The aim of this study was to assess the applicability of AMSTAR 2 to SRs of non-intervention studies.

STUDY DESIGN AND SETTING

This was a meta-research study. We used 20 SRs for each of the following four types of SRs: Diagnostic Test Accuracy reviews, Etiology and/or Risk reviews, Prevalence and/or Incidence reviews, and Prognostic reviews (80 in total). Three authors applied AMSTAR 2 independently to each included SRs. Then, the authors assessed the applicability of each item to that SR type and any SR type.

RESULTS

Researchers unanimously indicated that 7 of 16 AMSTAR 2 items were applicable for all four specific SR types and any SR type (items 2, 5, 6, 7, 10, 14 and 16), but 8 of 16 items for any SR type. These items could cover generic SR methods that do not depend on a specific SR type.

CONCLUSION

AMSTAR 2 is only partially applicable for non-intervention SRs. There is a need to adapt/extend AMSTAR 2 for SRs of non-intervention studies. Our study can help to further define generic methodological aspects shared across SR types and methodological expectations for non-intervention SRs.

Predictors of transition in patients with clinical high risk for psychosis: an umbrella review

Diagnosis of a clinical high-risk (CHR) state enables timely treatment of individuals at risk for a psychotic disorder, thereby contributing to improving illness outcomes. However, only a minority of patients diagnosed with CHR will make the transition to overt psychosis. To identify patients most likely to benefit from early intervention, several studies have investigated characteristics that distinguish CHR patients who will later develop a psychotic disorder from those who will not. We aimed to summarize evidence from systematic reviews and meta-analyses on predictors of transition to psychosis in CHR patients, among characteristics and biomarkers assessed at baseline. A systematic search was conducted in Pubmed, Scopus, PsychInfo and Cochrane databases to identify reviews and meta-analyses of studies that investigated specific baseline predictors or biomarkers for transition to psychosis in CHR patients using a cross-sectional or longitudinal design. Non-peer-reviewed publications, gray literature, narrative reviews and publications not written in English were excluded from analyses. We provide a narrative synthesis of results from all included reviews and meta-analyses. For each included publication, we indicate the number of studies cited in each domain and its quality rating. A total of 40 publications (21 systematic reviews and 19 meta-analyses) that reviewed a total of 272 original studies qualified for inclusion. Baseline predictors most consistently associated with later transition included clinical characteristics such as attenuated psychotic and negative symptoms and functioning, verbal memory deficits and the electrophysiological marker of mismatch negativity. Few predictors reached a level of evidence sufficient to inform clinical practice, reflecting generalizability issues in a field characterized by studies with small, heterogeneous samples and relatively few transition events. Sample pooling and harmonization of methods across sites and projects are necessary to overcome these limitations.

The relationship between experience of knee pain and physical activity participation: A scoping review of quantitative studies

Objectives

Physical activity (PA) in people with knee pain can help to relieve pain and prevent health problems. However, engaging in regular PA is challenging for them due to chronic musculoskeletal pain. An understanding of how the experience of knee pain affects PA is important to minimize the negative effects of knee pain on participation in PA among this population. We examined the quantitative evidence describing the relationship between the experience of knee pain and participation in PA.

Methods

We conducted a scoping review based on the PRISMA-ScR process across 27 items and Peters' methodological framework. Searches using keywords were performed in Medline, PsycINFO, CINAHL and Scopus.

Results

Nine studies were included in this review. While one study clearly showed that increased knee pain increased PA, four studies indicated that knee pain decreased PA among individuals with knee pain. One study revealed that increased knee pain decreased moderate PA, but no significant difference with light intensity PA. Three studies showed that knee pain had no impact on PA. Moreover, psychological factors (catastrophizing and pain-related activity interference) and certain activities of daily living were risk factors for insufficient PA among people with knee pain.

Conclusions

The limited quantitative evidence on the relationship between the experience of knee pain and PA participation highlights a gap in the knowledge base. The review findings found mixed results, with a negative, positive or no relationship between knee pain and levels of PA among individuals with knee pain. The findings also suggest that the relationship can be affected by psychological factors and different levels and types of PA. Therefore, practitioners should consider multi-component interventions which aim to increase PA levels of individuals with knee pain, after considering the above mediating factors.

User experience of applying AMSTAR 2 to appraise systematic reviews of healthcare interventions: a commentary

BACKGROUND

'A Measurement Tool to Assess Systematic Reviews, version 2' (AMSTAR 2) is a validated 16-item scale designed to appraise systematic reviews (SRs) of healthcare interventions and to rate the overall confidence in their results. This commentary aims to describe the challenges with rating of the individual items and the application of AMSTAR 2 from the user perspective.

DISCUSSION

A group of six experienced users (methodologists working in different clinical fields for at least 10 years) identified and discussed the challenges in rating of each item and the general use of AMSTAR 2 to appraise SRs. A group discussion was used to develop recommendations on how users could deal with the identified challenges. We identified various challenges with the content of items 2-16 and with the derivation of the overall confidence ratings on AMSTAR 2. These challenges include the need (1) to provide additional definitions (e.g., what constitutes major deviations from SR protocol on item 2), (2) to choose a rating strategy for multiple conditions on single items (e.g., how to rate item 5 if studies were selected in duplicate, but consensus between two authors was not reported), and (3) to determine rules for deriving the confidence ratings (e.g., what items are critical for such ratings). Based on these challenges we formulated specific recommendations for items 2-16 that AMSTAR 2 users could consider before applying the tool. Our commentary adds to the existing literature by providing the first in-depth examination of the AMSTAR 2 tool from the user perspective. The identified challenges could be addressed by additional decision rules including definitions for ambiguous items and guidance for rating of complex items and derivation of confidence ratings. We recommend that a team consensus regarding such decision rules is required before appraisal procedure begins.

TRIAL REGISTRATION

Not applicable.

Obesity and main urologic cancers: Current systematic evidence, novel biological mechanisms, perspectives and challenges

Urologic cancers (UC) account for 13.1% of all new cancer cases and 7.9% of all cancer-related deaths. A growing body of evidence has indicated a potential causal link between obesity and UC. The aim of the present review is to appraise in a critical and integrative manner evidence from meta-analyses and mechanistic studies on the role of obesity in four prevalent UC (kidney-KC, prostate-PC, urinary bladder-UBC, and testicular cancer-TC). Special emphasis is given on Mendelian Randomization Studies (MRS) corroborating a genetic causal association between obesity and UC, as well as on the role of classical and novel adipocytokines. Furthermore, the molecular pathways that link obesity to the development and progression of these cancers are reviewed. Available evidence indicates that obesity confers increased risk for KC, UBC, and advanced PC (20-82%, 10-19%, and 6-14%, respectively), whereas for TC adult height (5-cm increase) may increase the risk by 13%. Obese females tend to be more susceptible to UBC and KC than obese males. MRS have shown that a higher genetic-predicted BMI may be causally linked to KC and UBC but not PC and TC. Biological mechanisms that are involved in the association between excess body weight and UC include the Insulin-like Growth Factor axis, altered availability of sex hormones, chronic inflammation and oxidative stress, abnormal secretion of adipocytokines, ectopic fat deposition, dysbiosis of the gastrointestinal and urinary tract microbiomes and circadian rhythm dysregulation. Anti-hyperglycemic and non-steroidal anti-inflammatory drugs, statins, and adipokine receptor agonists/antagonists show potential as adjuvant cancer therapies. Identifying obesity as a modifiable risk factor for UC may have significant public health implications, allowing clinicians to tailor individualized prevention strategies for patients with excess body weight.

Impact of industry sponsorship on the quality of systematic reviews of vaccines: a cross-sectional analysis of studies published from 2016 to 2019

BACKGROUND

Systematic reviews (SRs) provide the highest level of evidence and inform evidence-based decision making in health care. Earlier studies found association with industry to be negatively associated with methodological quality of SRs. However, this has not been investigated in SRs on vaccines.

METHODS

We performed a systematic literature search using MEDLINE and EMBASE in March 2020. The results were restricted to those published between 2016 and 2019 with no language restrictions. Study characteristics were extracted by one person and checked by an experienced reviewer. The methodological quality of the SRs was assessed with the AMSTAR 2 tool by multiple reviewers after a calibration exercise was performed. A summary score for each SR was calculated. The Mann-Whitney U test and Fisher's exact test were performed to compare both groups.

RESULTS

Out of 185 SRs that met all inclusion criteria, 27 SRs were industry funded. Those were matched with 30 non-industry funded SRs resulting in a total sample size of 57. The mean AMSTAR 2 summary score across all SRs was 0.49. Overall, the median AMSTAR 2 summary score was higher for the non-industry funded SRs than for the industry-funded SRs (0.62 vs. 0.36; p < .00001). Lower ratings for industry funded SRs were consistent across all but one AMSTAR 2 item, though significantly lower only for three specific items.

CONCLUSION

The methodological quality of SRs in vaccination is comparable to SRs in other fields, while it is still suboptimal. We are not able to provide a satisfactory explanation why industry funded SRs had a lower methodological quality than non-industry funded SRs over recent years. Industry funding is an important indicator of methodological quality for vaccine SRs and should be carefully considered when appraising SR quality.

Influence and effects of weight stigmatisation in media: A systematic

BACKGROUND

The lack of a comprehensive understanding of the role of mass media in perpetuating weight stigma hinders policy formulation. We reviewed the influence of mass media on weight stigmatisation and the effectiveness of media-based interventions designed to prevent or reduce stigma.

METHODS

We conducted a systematic review across seven databases from inception to December 2021. Included studies assessed exposure to or impact of weight stigma in mass media or examined interventions to reduce stigma through media in populations 12+ years. We synthesised data narratively, categorising studies based on similarity in focus to produce a set of integrated findings. The systematic review is registered in PROSPERO (No. CRD42020176306).

FINDINGS

One-hundred-and-thirteen records were eligible for inclusion from 2402 identified; 95 examined the prevalence of stigmatising content in mass media and its impact on stigma. Weight stigma was prevalent across media types, with the dominant discourse viewing overweight and obesity as an individual responsibility and overlooking systemic factors. Exposure to stigmatising content was found to negatively influence attitudes towards people with overweight or obesity. Few studies considered methods of reducing stigma in the media, with only two testing media-based interventions; their results were promising but limited.

INTERPRETATION

Weight stigma in media content is prevalent and harmful, but there is little guidance on reducing it. Future research focus needs to shift from assessing prevalence and impacts to weight stigma interventions.

FUNDING

None.

Adopting AMSTAR 2 critical appraisal tool for systematic reviews: speed of the tool uptake and barriers for its adoption

Background

In 2007, AMSTAR (A MeaSurement Tool to Assess systematic Reviews), a critical appraisal tool for systematic reviews (SRs), was published, and it has since become one of the most widely used instruments for SR appraisal. In September 2017, AMSTAR 2 was published as an updated version of the tool. This mixed-methods study aimed to analyze the extent of the AMSTAR 2 uptake and explore potential barriers to its uptake.

Methods

We analyzed the frequency of AMSTAR or AMSTAR 2 use in articles published in 2018, 2019 and 2020. We surveyed authors who have used AMSTAR but not AMSTAR 2 in the analyzed time frame to identify their reasons and barriers. The inclusion criterion for those authors was that the month of manuscript submission was after September 2017, i.e. after AMSTAR 2 was published.

Results

We included 871 studies. The majority (N = 451; 52%) used AMSTAR 2, while 44% (N = 382) used AMSTAR, 4% (N = 31) used R-AMSTAR and others used a combination of tools. In 2018, 81% of the analyzed studies used AMSTAR, while 16% used AMSTAR 2. In 2019, 52% used AMSTAR, while 44% used AMSTAR 2. Among articles published in 2020, 28% used AMSTAR, while AMSTAR 2 was used by 69%.

An author survey indicated that the authors did not use AMSTAR 2 mostly because they were not aware of it, their protocol was already established, or data collection completed at the time when the new tool was published. Barriers towards AMSTAR 2 use were lack of quantitative assessment, insufficient awareness, length, difficulties with a specific item.

Conclusion

In articles published in 2018-2020, that were submitted to a journal after AMSTAR 2 tool was published, almost half of the authors (44%) still used AMSTAR, the old version of the tool. However, the use of AMSTAR has been declining in each subsequent year. Our survey indicated that editors and peer-reviewers did not ask the authors to use the new version of the tool. Few barriers towards using AMSTAR 2 were identified, and thus it is anticipated that the use of the old version of AMSTAR will continue to decline.

Traduction franco-canadienne de l’Assessment of Systematic Reviews Revised (AMSTAR 2) : validation transculturelle et fidélité interjuges

Objective: Produce a French-Canadian translation of AMSTAR 2, affirm its content validity, and examine interrater reliability. Methods: Based on Vallerand's methodological approach, we conducted forward and parallel inverse-translations. Subsequently, an expert panel evaluated the translations to create a preliminary experimental French-Canadian version. A second expert panel examined this version and proposed additional modifications. Twenty future health professionals then rated the second experimental version for ambiguity on a scale (from 1 to 7). The principal co-investigators then reviewed the problematic elements and proposed a pre-official version. To ascertain content validity, a final back-translation was conducted resulting in the official version. Four judges evaluated 13 systematic reviews using the official French-Canadian version of AMSTAR 2. The Kappa coefficient was used to evaluate interrater reliability. Results: This rigorous adaptation enabled the development of a Franco-Canadian version of AMSTAR 2. Its application demonstrated low ambiguity (mean 1.15; SD 0.26) as well as good overall interrater reliability (total κ > 0.64) across all items. Conclusion: The French-Canadian version of AMSTAR 2 can now support francophone clinicians, educators, and managers in Canada as they undertake evidence-based practice.

High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta-epidemiological study

BACKGROUND

Redundant publication of systematic reviews and meta-analyses (SRs/MAs) on the same topic presents an increasing burden for clinicians. The aim of this study was to describe variabilities in effect size and methodological quality of overlapping surgery-related SRs/MAs and to investigate factors associated with their postpublication citations.

METHODS

PubMed/MEDLINE was searched to identify SRs/MAs of RCTs on thoracoabdominal surgeries published in 2015. Previous SRs/MAs on the same topics published within the preceding 5 years (2011-2015) were identified and 5-year citation counts (through to 2020) were evaluated. Discrepancies in pooled effect sizes and their methodological quality using A Measurement Tool to Assess Systematic Reviews (AMSTAR) among overlapping SRs/MAs were assessed. The SR/MA-level factors associated with 5-year citation counts were explored, using a mixed-effects regression model with a random intercept for surgical topics.

RESULTS

A total of 57 surgery-related SRs/MAs (48 topics) published in 2015 were identified, and 146 SRs/MAs had overlapping publications on 29 topics (60.4 per cent of all topics) in the preceding 5 years. There was considerable variability in methodological quality of SRs/MAs and coverage probability for relevant RCTs, resulting in discrepant effect size estimates for the same topic. High quality (AMSTAR score 8-11) was independently associated with higher 5-year citation counts (coefficient = 32.82; 95 per cent c.i. 15.63 to 50.02; P < 0.001).

CONCLUSION

Overlapping SRs/MAs with high variability in results and methodological quality were common in surgery. A high-quality SR/MA score was an independent predictor of more frequent citations. Researchers and journal editors should concentrate their efforts on limiting publications to higher-quality reviews.

Postoperative hiatal herniation after open vs. minimally invasive esophagectomy; a systematic review and meta-analysis

BACKGROUND

Post-esophagectomy hiatal hernia (PEHH) is a known, but relatively uncommon, complication after esophagectomies. The incidence of PEHH seems to be increasing since the introduction of minimally invasive esophagectomy. This systematic review and meta-analysis aimed to determine the pooled incidence of PEHH after esophagectomy, and to evaluate if minimally invasive technique is associated with increased risk for PEHH compared to open esophagectomy.

METHODS

A systematic search of PubMed, Medline via Ovid and Web of Science was performed. Retrospective and prospective studies in English language describing the incidence or risk factors for PEHH were included. Weighted incidence of PEHH after all types of esophagectomy, and after open or minimally invasive technique was calculated.

RESULTS

A total of 7943 esophagectomy patients were included in the analysis. In total, 310 patients (3.9%) were diagnosed with PEHH. The estimated weighted incidence rate for PEHH after open esophagectomy was 0.024 (95% confidence interval: 0.012-0.045) compared to 0.065 (95% confidence interval: 0.040-0.106) after minimally invasive esophagectomy. Odds ratio for PEHH after minimally invasive esophagectomy compared to open esophagectomy was 2.76 (95% confidence interval: 1.49-5.11).

CONCLUSION

The risk for post-esophagectomy hiatal hernia was significantly higher after minimally invasive esophagectomy compared to open technique. Heterogeneity and retrospective designs of the included studies were important limitations of the analysis. Future studies should investigate preventive measures to reduce PEHH after minimally invasive esophagectomy.

Technological progress in electronic health record system optimization: Systematic review of systematic literature reviews

BACKGROUND

The recent, rapid development of digital technologies offers new possibilities for more efficient implementation of electronic health record (EHR) and personal health record (PHR) systems. A growing volume of healthcare data has been the hallmark of this digital transformation. The large healthcare datasets' complexity and their dynamic nature pose various challenges related to processing, analysis, storage, security, privacy, data exchange, and usability.

MATERIALS AND METHODS

We performed a systematic review of systematic reviews to assess technological progress in EHR and PHR systems. We searched MEDLINE, Cochrane, Web of Science, and Scopus for systematic literature reviews on technological advancements that support EHR and PHR systems published between January 1, 2010, and October 06, 2020.

RESULTS

The searches resulted in a total of 2,448 hits. Of these, we finally selected 23 systematic reviews. Most of the included papers dealt with information extraction tools and natural language processing technology (n = 10), followed by studies that assessed the use of blockchain technology in healthcare (n = 8). Other areas of digital technology research included EHR and PHR systems in austere settings (n = 1), de-identification methods (n = 1), visualization techniques (n = 1), communication tools within EHR and PHR systems (n = 1), and methodologies for defining Clinical Information Models that promoted EHRs and PHRs interoperability (n = 1).

CONCLUSIONS

Technological advancements can improve the efficiency in the implementation of EHR and PHR systems in numerous ways. Natural language processing techniques, either rule-based, machine-learning, or deep learning-based, can extract information from clinical narratives and other unstructured data locked in EHRs and PHRs, allowing secondary research (i.e., phenotyping). Moreover, EHRs and PHRs are expected to be the primary beneficiaries of the blockchain technology implementation on Health Information Systems. Governance regulations, lack of trust, poor scalability, security, privacy, low performance, and high cost remain the most critical challenges for implementing these technologies.

Methodological quality was critically low in 9/10 systematic reviews in advanced cancer patients-A methodological study

OBJECTIVE

To assess the methodological quality and the consideration of heterogeneity in systematic reviews (SRs).

STUDY DESIGN AND SETTING

We conducted a methodological study (CRD42019134904) and searched three databases from January 2010 to July 2019. Interventional SRs with a statistically significant meta-analysis of at least four randomized controlled trials in advanced cancer patients were included. A MeaSurement Tool to Assess Systematic Reviews (AMSTAR) 2 was used to evaluate the SRs' methodological quality. The consideration of heterogeneity was categorized in clinical or/and methodological heterogeneity and not explored.

RESULTS

From 6234 identified references, 261 SRs were included. Most SRs had a critically low quality (230, 88.1%). The majority of them (209, 80.1%) was classified as critically low because of non-registration (222, 85.1%) combined with the non-reporting of excluded full-texts and missing justifications for exclusion (218, 83.5%). Heterogeneity in trial results was not explored at all in 51 (19.5%) SRs whereas clinical heterogeneity was considered in 117 (44.8%), methodological heterogeneity in 13 (5.0%), and both clinical and methodological heterogeneity in 80 (30.7%) SRs.

CONCLUSION

The consideration of these findings in trainings for review authors and peer reviewers could improve the awareness of quality criteria and the quality of future SRs.

TRIAL REGISTRATION

PROSPERO-ID: CRD42019134904.

Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement - a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools

Background

Systematic Reviews (SRs) can build the groundwork for evidence-based health care decision-making. A sound methodological quality of SRs is crucial. AMSTAR (A Measurement Tool to Assess Systematic Reviews) is a widely used tool developed to assess the methodological quality of SRs of randomized controlled trials (RCTs). Research shows that AMSTAR seems to be valid and reliable in terms of interrater reliability (IRR), but the test retest reliability (TRR) of AMSTAR has never been investigated. In our study we investigated the TRR of AMSTAR to evaluate the importance of its measurement and contribute to the discussion of the measurement properties of AMSTAR and other quality assessment tools.

Methods

Seven raters at three institutions independently assessed the methodological quality of SRs in the field of occupational health with AMSTAR. Between the first and second ratings was a timespan of approximately two years. Answers were dichotomized, and we calculated the TRR of all raters and AMSTAR items using Gwet’s AC1 coefficient. To investigate the impact of variation in the ratings over time, we obtained summary scores for each review.

Results

AMSTAR item 4 (Was the status of publication used as an inclusion criterion?) provided the lowest median TRR of 0.53 (moderate agreement). Perfect agreement of all reviewers was detected for AMSTAR-item 1 with a Gwet’s AC1 of 1, which represented perfect agreement. The median TRR of the single raters varied between 0.69 (substantial agreement) and 0.89 (almost perfect agreement). Variation of two or more points in yes-scored AMSTAR items was observed in 65% (73/112) of all assessments.

Conclusions

The high variation between the first and second AMSTAR ratings suggests that consideration of the TRR is important when evaluating the psychometric properties of AMSTAR.. However, more evidence is needed to investigate this neglected issue of measurement properties. Our results may initiate discussion of the importance of considering the TRR of assessment tools. A further examination of the TRR of AMSTAR, as well as other recently established rating tools such as AMSTAR 2 and ROBIS (Risk Of Bias In Systematic reviews), would be useful.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01231-y.

Exploring the effectiveness of technology-based learning on the educational outcomes of undergraduate healthcare students: an overview of systematic reviews protocol

INTRODUCTION

Rapid technology development due to the introduction of Industrial Revolution 4.0 and Internet of Things has created a demand and gradual transition from traditional teaching and learning to technology-based learning in higher education, including healthcare education. The COVID-19 pandemic has accelerated this process, with educators now required to quickly adapt to and adopt such changes. The abundance of available systematic reviews has made the effectiveness of such approaches ambiguous especially in healthcare education. Therefore, a protocol of the overview of systematic reviews (OoSR) is planned to extrapolate the effectiveness of technology-based learning in undergraduate healthcare education.

METHODS AND ANALYSIS

Scopus, CINAHL, Academic Search Complete, Cochrane Library, MEDLINE and Psychology and Behavioral Sciences Collection databases were selected. Screening was conducted independently by at least two authors and the decision for inclusion was done through discussion or involvement of an arbiter against a predetermined criteria. Included articles will be evaluated for quality using A MeaSurement Tool to Assess systematic Reviews and Risk of Bias in Systematic Review tools, while primary systematic review articles will be cross-checked and reported for any overlapping using the 'corrected covered area' method. Only narrative synthesis will be employed according to the predefined themes into two major dimensions-theory and knowledge generation (focusing on cognitive taxonomy due to its ability to be generalised across disciplines), and clinical-based competence (focusing on psychomotor and affective taxonomies due to discipline-specific influence). The type of technology used will be identified and extracted.

ETHICS AND DISSEMINATION

The OoSR involves analysis of secondary data from published literature, thus ethical approval is not required. The findings will provide a valuable insight for policymakers, stakeholders, and researchers in terms of technology-based learning implementation and gaps identification. The findings will be published in several reports due to the extensiveness of the topic and will be disseminated through peer-reviewed publications and conferences.

PROSPERO REGISTRATION NUMBER

CRD4202017974.

Does Nonsurgical Periodontal Treatment Improve Systemic Health?

Clinicians frequently stress the importance of maintaining good oral health for multiple reasons, including its link to systemic health. Because periodontal treatment reduces inflammation in oral tissues, some hypothesize it may positively affect systemic outcomes by reducing inflammation in the body. A significant number of systematic reviews (SRs) and meta-analyses (MAs) have evaluated the effect of periodontal treatment on systemic outcomes. However, inconsistent findings and questionable methodological rigor make drawing conclusions difficult. We conducted a systematic review of reviews that studied the effect of nonsurgical periodontal treatment on systemic disease outcomes. We report on outcomes evaluated, categorizing them as biomarkers, and surrogate or clinical endpoints. In addition, we used A MeaSurement Tool to Access systematic Reviews 2 (AMSTAR 2) to evaluate the methodological quality of the reviews. Of the 52 studies included in our review, 21 focused on diabetes, 15 on adverse birth outcomes, 8 on cardiovascular disease, 3 each on obesity and rheumatoid arthritis, and 2 on chronic kidney disease. Across all studies, surrogate endpoints predominated as outcomes, followed by biomarkers and, rarely, actual disease endpoints. Ninety-two percent of studies had "low" or "critically low" AMSTAR 2 confidence ratings. Criteria not met most frequently included advance registration of the protocol, justification for excluding individual studies, risk of bias from individual studies being included in the review, and appropriateness of meta-analytical methods. There is a dearth of robust evidence on whether nonsurgical periodontal treatment improves systemic disease outcomes. Future reviews should adhere more closely to methodological guidelines for conducting and reporting SRs/MAs than has been the case to date. Beyond improved reviews, additional rigorous research on whether periodontal treatment affects systemic health is needed. We highlight the potential of large-scale databases containing matched medical and dental record data to inform and complement future clinical research studying the effect of periodontal treatment on systemic outcomes.

Authors should clearly report how they derived the overall rating when applying AMSTAR 2-a cross-sectional study

OBJECTIVES

A measurement tool to assess systematic reviews (SRs) 2 (AMSTAR 2) allows for deriving the overall confidence in an SR. We investigated how authors derived the overall confidence rating and whether different schemes lead to different results.

STUDY DESIGN AND SETTING

We compared three different schemes (original 7-item scheme, a self-developed 5-item scheme, and the AMSTAR Web site) to derive the overall confidence in AMSTAR 2 using two distinct samples of SRs. Multiple bibliographic databases were searched for articles to analyze how AMSTAR 2 was applied by others.

RESULTS

In both samples (n = 60 and n = 58), the Friedman test revealed a significant difference between the schemes (P < 0.001). The Web site scheme was the least strict one, whereas between the 5-item and 7-item scheme, no differences were found in post hoc analyses. We included 53 publications applying AMSTAR 2 identified in our literature search. Only 37 of them (70%) used the original 7-item scheme. Less than half of them (18 of 37) reported how they derived the overall rating.

CONCLUSION

Authors should clearly report how they have derived the overall rating when applying AMSTAR 2. Reporting should allow for reproducing the overall ratings for editors, peer reviewers, and readers.

Assessing the methodological quality of systematic reviews using the AMSTAR tool

Introduction: Making decisions based on evidence has been a challenge for health professionals, given the need to have the tools and skills to carry out a critical appraisal of the evidence and assess the validity of the results. Systematic reviews of the literature (SRL) have been used widely to answer questions in the clinical field. Tools have been developed that support the appraisal of the quality of the studies. AMSTAR is one of these, validated and supported by reproducible evidence, which guides the methodological quality of the SRL. Objectives: To show a historical, theoretical and practical guide for critical assessment of systematic reviews using AMSTAR to guide the argumental bases for their use according to the components of this methodological structure in health research, and to provide practical examples of how to apply this checklist. Methods: We conducted a non-exhaustive review of literature in Pubmed and Cochrane Library using “AMSTAR” and “Systematic Reviews” as free terms without language or publication date limit; we also collected information from experts in the evaluation of the quality of the evidence. Conclusions: AMSTAR is an instrument used, validated and supported by reproducible evidence for the evaluation of the internal validity of systematic reviews of the literature. It consists of 16 items that assess the overall methodological quality of an SRL. It is currently used indiscriminately and favorably, but it is not exempt from limitations and future updates based on new reproducibility and validation studies.

Clinical management of tendinopathy: A systematic review of systematic reviews evaluating the effectiveness of tendinopathy treatments

While the pathoetiology is disputed, a wide array of treatments is available to treat tendinopathy. The most common treatments found in the literature include therapeutic modalities, exercise protocols, and surgical interventions; however, their effectiveness remains ambiguous. The purpose of this study was to perform a systematic review of systematic reviews to determine the ability of therapeutic interventions to improve pain and dysfunction in patients with tendinopathy regardless of type or location. Five databases were searched for systematic reviews containing only randomized control trials to determine the effectiveness of treatments for tendinopathies based on pain and patient-reported outcomes. Systematic reviews were assessed via the Assessment of Multiple Systematic Reviews (AMSTAR) for methodological quality. From the database search, 3,295 articles were found, 107 passed the initial inclusion criteria. After further review, 25 systematic reviews were included in the final qualitative analysis. The AMSTAR scores were relatively high (8.8 ± 1.0) across the 25 systematic reviews. Eccentric exercises were the most common and consistently effective treatment for tendinopathy across systematic reviews. Low-level laser therapy and extracorporeal shockwave therapy demonstrated moderate effectiveness, while platelet-rich plasma injections demonstrated inconclusive evidence on their ability to decrease tendinopathy-related pain and improve function. Corticosteroids also showed some effectiveness for short-term pain, but for the long-term use deemed ineffective and at times contraindicated. Regarding surgical options, minimally invasive procedures were more effective compared to open surgical interventions. When treating tendinopathy regardless of location, eccentric exercises were the best treatment option to improve tendinopathy-related pain and improve self-reported function.

Evidence from systematic reviews on photobiomodulation of human bone and stromal cells: Where do we stand?

This study summarizes the available evidence from systematic reviews on the in vitro effects of photobiomodulation on the proliferation and differentiation of human bone and stromal cells by appraising their methodological quality. Improvements for future studies are also highlighted, with particular emphasis on in vitro protocols and cell-related characteristics. Six reviews using explicit eligibility criteria and methods selected in order to minimize bias were included. There was no compelling evidence on the cellular mechanisms of action or treatment parameters of photobiomodulation; compliance with quality assessment was poor. A rigorous description of laser parameters (wavelength, power, beam spot size, power density, energy density, repetition rate, pulse duration or duty cycle, exposure duration, frequency of treatments, and total radiant energy), exposure conditions (methods to ensure a uniform irradiation and to avoid cross-irradiation, laser-cell culture surface distance, lid presence during irradiation) and cell-related characteristics (cell type or line, isolation and culture conditions, donor-related factors where applicable, tissue source, cell phenotype, cell density, number of cell passages in culture) should be included among eligibility criteria for study inclusion. These methodological improvements will maximize the contribution of in vitro studies on the effects of photobiomodulation on human bone and stromal cells to evidence-based translational research.

Current methods for development of rapid reviews about diagnostic tests: an international survey

BACKGROUND

Rapid reviews (RRs) have emerged as an efficient alternative to time-consuming systematic reviews-they can help meet the demand for accelerated evidence synthesis to inform decision-making in healthcare. The synthesis of diagnostic evidence has important methodological challenges. Here, we performed an international survey to identify the current practice of producing RRs for diagnostic tests.

METHODS

We developed and administered an online survey inviting institutions that perform RRs of diagnostic tests from all over the world.

RESULTS

All participants (N = 25) reported the implementation of one or more methods to define the scope of the RR; however, only one strategy (defining a structured question) was used by ≥90% of participants. All participants used at least one methodological shortcut including the use of a previous review as a starting point (92%) and the use of limits on the search (96%). Parallelization and automation of review tasks were not extensively used (48 and 20%, respectively).

CONCLUSION

Our survey indicates a greater use of shortcuts and limits for conducting diagnostic test RRs versus the results of a recent scoping review analyzing published RRs. Several shortcuts are used without knowing how their implementation affects the results of the evidence synthesis in the setting of diagnostic test reviews. Thus, a structured evaluation of the challenges and implications of the adoption of these RR methods is warranted.

Effects of CPOE-based medication ordering on outcomes: an overview of systematic reviews

BACKGROUND

Computerised provider order entry (CPOE) systems are widely used in clinical settings for the electronic ordering of medications, laboratory tests and radiological therapies. However, evidence regarding effects of CPOE-based medication ordering on clinical and safety outcomes is mixed. We conducted an overview of systematic reviews (SRs) to characterise the cumulative effects of CPOE use for medication ordering in clinical settings.

METHODS

MEDLINE, EMBASE, CINAHL and the Cochrane Library were searched to identify published SRs from inception to 12 February 2018. SRs investigating the effects of the use of CPOE for medication ordering were included. Two reviewers independently extracted data and assessed the methodological quality of included SRs.

RESULTS

Seven SRs covering 118 primary studies were included for review. Pooled studies from the SRs in inpatient settings showed that CPOE use resulted in statistically significant decreases in medication errors and adverse drug events (ADEs); however, there was considerable variation in the magnitude of their relative risk reduction (54%-92% for errors, 35%-53% for ADEs). There was no significant relative risk reduction on hospital mortality or length of stay. Bibliographic analysis showed limited overlap (24%) among studies included across all SRs.

CONCLUSION

SRs on CPOEs included predominantly non-randomised controlled trials and observational studies with varying foci. SRs predominantly focused on inpatient settings and often lacked comparison groups; SRs used inconsistent definitions of outcomes, lacked descriptions regarding the effects on patient harm and did not differentiate among the levels of available decision support. With five of the seven SRs having low to moderate quality, findings from the SRs must be interpreted with caution. We discuss potential directions for future primary studies and SRs of CPOE.

Use of AMSTAR-2 in the methodological assessment of systematic reviews: protocol for a methodological study

BACKGROUND

Systematic reviews (SRs) with or without meta-analyses (MAs) are widely used in resolving questions in various healthcare areas (such as, traditional Chinese medicine, public health and surgery), and they are the cornerstone of evidence-based healthcare. However, the reliability of SRs is typically influenced by their methodological quality. AMSTAR (A Measurement Tool to Assess Systematic Reviews) and AMSTAR-2 tools can assess the methodological quality of SRs, and the use of AMSTAR has been investigated. However, AMSTAR-2 is now widely used to evaluate the methodological quality of SRs, but the use of AMSTAR-2 for determining the methodological quality of SRs has not yet been investigated and assessed thoroughly. Thus, we designed the present study to investigate the use of AMSTAR-2 in studies that assessed the methodological quality of a sample of SRs with the AMSTAR-2 and provide references to potential users of AMSTAR-2 tool.

METHODS

Four commonly used electronic databases including PubMed, EmBase, the Cochrane Library, and Web of Science will be searched following a comprehensive search strategy to identify and retrieve studies that have used AMSTAR-2 tool for evaluating the methodological quality of SRs. Two independent authors will retrieve bibliometric information and methodological data, including all author names, time of publication, and journal names, whether a specific score value was given for each item, and whether overall quality assessment was performed. Descriptive statistical analyses will be used to present the study results, e.g., frequencies and percentages, mean and standard deviation (SD) or median and interquartile range (IQR). In addition, subgroup analyses will be performed to identify the methodological differences (e.g., the reporting of study designs included in SRs) between overviews and methodological studies. The risk ratio (RR) with 95% confidence interval (95% CI) will be calculated to measure the methodological differences. Cytoscape 3.7.1 software tool will be used to construct collaboration network maps. Further, Microsoft Office Excel 2016 and Stata 12.0 will be used to manage and analyze data.

DISCUSSION

The results of this study will identify any gaps in the use of AMSTAR-2 and important bibliometric features, such as active researchers and journals, provide guidance to researchers in various healthcare areas (such as, traditional Chinese medicine and public health) for using AMSTAR-2 tool and help them in developing cooperation and submitting their manuscripts.

Non-communicable diseases research output in the Eastern Mediterranean region: an overview of systematic reviews

BACKGROUND

Rates of non-communicable diseases (NCDs) are rapidly rising in the Eastern Mediterranean Region (EMR). Systematic reviews satisfy the demand from practitioners and policy makers for prompt comprehensive evidence. The aim of this study is to review trends in NCD systematic reviews research output and quality by time and place, describe design and focus, and examine gaps in knowledge produced.

METHODS

Using the Montori et al. systematic reviews filter, MeSH and keywords were applied to search Medline Ovid, Cochrane Central and Epistemonikos for publications from 1996 until 2015 in the 22 countries of the EMR. The 'Measurement Tool to Assess Systematic Reviews', AMSTAR, was used to assess the methodological quality of the papers.

RESULTS

Our search yielded 2439 papers for abstract and title screening, and 89 papers for full text screening. A total of 39 (43.8%) studies included meta-analysis. Most of the papers were judged as being of low AMSTAR quality (83.2%), and only one paper was judged as being of high AMSTAR quality. Whilst annual number of papers increased over the years, the growth was mainly attributed to an increase in low-quality publications approaching in 2015 over four times the number of medium-quality publications. Reviews were significantly more likely to be characterized by higher AMSTAR scores (±SD) when meta-analysis was performed compared to when meta-analysis was not performed (3.4 ± 1.5 vs 2.6 ± 2.0; p-value = 0.034); and when critical appraisal of the included studies was conducted (4.3 ± 2.3 vs 2.5 ± 1.5; p-value = 0.004). Most of the reviews focused on cancer and diabetes as an outcome (25.8% and 24.7%, respectively), and on smoking, dietary habits and physical activity as exposures (15.7%, 12.4%, 9.0%, respectively). There was a blatant deficit in reviews examining associations between behaviors and physiologic factors, notably metabolic conditions.

CONCLUSIONS

Systematic reviews research in the EMR region are overwhelmingly of low quality, with gaps in the literature for studies on cardiovascular disease and on associations between behavioral factors and intermediary physiologic parameters. This study raises awareness of the need for high-quality evidence guided by locally driven research agenda responsive to emerging needs in countries of the EMR.

Assessing the Quality of Systematic Reviews in Healthcare Using AMSTAR and AMSTAR2

Abstract. The current study assessed the consistency between A Measurement Tool to Assess Systematic Reviews (AMSTAR) and its updated version (AMSTAR2) applied to the same systematic reviews in healthcare. Data from k = 10 systematic reviews were coded by two raters using AMSTAR and AMSTAR2. AMSTAR and AMSTAR2 perfectly agreed on a subset of nine individual items and strongly correlated based on the total scores (percentage scores: ρ = .84, p = .002, k = 10; absolute scores: Spearman ρ = .83, p = .003, k = 10). The overall review quality was medium to high on AMSTAR, while the overall confidence in the results was low to critically low on AMSTAR2 for the same systematic reviews. AMSTAR2 can identify the sources of strengths, weaknesses, and biases in systematic reviews. However, the interpretation of the overall confidence in results of systematic reviews requires additional guidelines for users.

Factors associated with adolescent HPV vaccination in the U.S.: A systematic review of reviews and multilevel framework to inform intervention development

Multilevel factors impact HPV vaccine series initiation and completion among adolescents in the U.S. Synthesis of these factors is needed to inform intervention development and to direct future research. Current frameworks synthesizing factors focus on females only and do not include both series initiation and completion outcomes. We conducted a systematic review of reviews to identify modifiable individual-, provider-, and clinic-level factors associated with HPV vaccination outcomes among U.S. adolescents and developed a multilevel framework illustrating relations between factors to inform intervention development. We searched Medline, PsychInfo, Pubmed, CINAHL, and ERIC databases and included reviews published 2006 to July 2, 2018 describing individual-, provider-, or clinic-level factors quantitatively associated with HPV vaccination among U.S. adolescents. Two coders independently screened reviews, extracted data, and determined quality ratings. Sixteen reviews containing 481 unique primary studies met criteria. Factors synthesized into the multilevel framework included parent psychosocial factors (knowledge, beliefs, outcome expectations, intentions) and behaviors, provider recommendation, and patient-targeted and provider-targeted clinic systems. The scope of our framework and review advances research in two key ways. First, the framework illustrates salient modifiable factors at multiple levels on which to intervene to increase HPV vaccination. Second, the review identified critical gaps in the literature at each level. Future research should link the body of literature on parental intentions to vaccination outcomes, identify provider psychosocial factors associated with recommendation behaviors and subsequent vaccine uptake in their patient population, and understand clinic factors associated with successful implementation of patient- and provider-targeted system-level interventions.

Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks

BACKGROUND

A standard or consensus definition of a systematic review does not exist. Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to analyse the definition of a systematic review (SR) in health care literature, elements of the definitions that are used and to propose a starting point for an explicit and non-ambiguous SR definition.

METHODS

We included overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks. We extracted the definitions of SRs, as well as the inclusion and exclusion criteria that could indicate which definition of a SR the authors used. We extracted individual elements of SR definitions, categorised and quantified them.

RESULTS

Among the 535 analysed sources of information, 188 (35%) provided a definition of a SR. The most commonly used reference points for the definitions of SRs were Cochrane and the PRISMA statement. We found 188 different elements of SR definitions and divided them into 14 categories. The highest number of SR definition elements was found in categories related to searching (N = 51), analysis/synthesis (N = 23), overall methods (N = 22), quality/bias/appraisal/validity (N = 22) and aim/question (N = 13). The same five categories were also the most commonly used combination of categories in the SR definitions.

CONCLUSION

Currently used definitions of SRs are vague and ambiguous, often using terms such as clear, explicit and systematic, without further elaboration. In this manuscript we propose a more specific definition of a systematic review, with the ultimate aim of motivating the research community to establish a clear and unambiguous definition of this type of research.

Best Practice Guidelines and Essential Methodological Steps to Conduct Rigorous and Systematic Meta-Reviews

BACKGROUND

A growing body of primary study and systematic review literature evaluates interventions and phenomena in applied and health psychology. Reviews of reviews (i.e., meta-reviews) systematically synthesise and utilise this vast and often overwhelming literature; yet, currently there are few practical guidelines for meta-review authors to follow.

OBJECTIVE

The aim of this article is to provide an overview of the best practice guidelines for all research synthesis and to detail additional specific considerations and methodological details for the best practice of conducting a rigorous meta-review.

METHODS

This article provides readers with six systematic and practical steps along with accompanying examples to address with rigor the unique challenges that arise when authors familiar with systematic review methods begin a meta-review: (a) detailing a clear scope, (b) identifying synthesis literature through strategic searches, (c) considering datedness of the literature, (d) addressing overlap among included reviews, (e) choosing and applying review quality tools, and (f) appropriate options for handling the synthesis and reporting of the vast amount of data collected in a meta-review.

CONCLUSIONS

We have curated best practice recommendations and practical tips for conducting a meta-review. We anticipate that assessments of meta-review quality will ultimately formalise best-method guidelines.

Processed meat intake and chronic disease morbidity and mortality: An overview of systematic reviews and meta-analyses

Despite the nutritional value of meat, a large volume of reviews and meta-analyses suggests that processed meat intake is associated with an increased risk of chronic diseases. However, assessments of the quality of these published reviews internal validity are generally lacking. We systematically reviewed and assessed the quality alongside summarizing the results of previously published systematic reviews and meta-analyses that examined the association between processed meat intake and cancers, type II diabetes (T2D), and cardiovascular diseases (CVD). Reviews and meta-analyses published until May 2018 were identified through a systematic literature search in the databases MEDLINE and EMBASE, and reference lists of included reviews. The quality of the systematic reviews and meta-analyses was assessed using A Measurement Tool to Assess Systematic Reviews (AMSTAR). All eligible reviews had to comply with two quality requirements: providing sufficient information on quality assessment of the primary studies and a comprehensive search. The results were summarized for T2D, CVD, and each of the different cancer types. The certainty in the estimates of the individual outcomes was rated using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) method. In total, 22 systematic reviews were eligible and thus included in this review. More than 100 reviews were excluded because quality assessment of the primary studies had not been performed. The AMSTAR score of the included reviews ranged from 5 to 8 indicating moderate quality. Overall, the quality assessments of primary studies of the reviews are generally lacking; the scientific quality of the systematic reviews reporting positive associations between processed meat intake and risk of various cancers, T2D and CVD is moderate, and the results from case-control studies suggest more often a positive association than the results from cohort studies. The overall certainty in the evidence was very low across all individual outcomes, due to serious risk of bias and imprecision.

Changing prescribing behaviours with educational outreach: an overview of evidence and practice

BACKGROUND

General practitioners (GPs), or family practitioners, are tasked with prescribing medications that can be harmful to the community if they are inappropriately prescribed or used (e.g. opioids). Educational programs, such as educational outreach (EO), are designed to change the behaviour of health professionals. The purpose of this study was to identify the efficacy of EO programs at changing the prescribing behaviour of GPs.

METHODS

This study included an evidence and practice review, comprising a rapid review supplemented by interviews with people who are familiar with EO implementation for regulation purposes. Seven databases were searched using terms related to health professionals and prescribing. Systematic and narrative reviews published in English after 2007 were included. Non-statistical analysis was used to report intervention efficacy. Three government representatives participated in semi-structured interviews to aid in understanding the relevance of review findings to the Victorian context. Interviews were transcribed verbatim and thematically analysed for emerging themes.

RESULTS

Fourteen reviews were identified for the evidence review. Isolated (e.g. EO program delivered by itself) and multifaceted (e.g. EO program supplemented by other interventions) programs were found to change prescribing behaviours. However, limited evidence suggests that EO can successfully change prescribing behaviours specific to GPs. Isolated EO can successfully change health professional prescribing behaviours, although cheaper alternatives such as letters might be just as effective. Multifaceted EO can also successfully change health professional prescribing behaviours, especially in older adults, but it remains unclear as to what combination of interventions works best. Success factors for EO reported by government representatives included programs having practical rather than didactic foci; making EO compulsory; focussing EO on preventing adverse events; using monetary or professional development incentives; and in-person delivery.

CONCLUSIONS

Educational outreach can successfully change prescribing behaviours but evidence specific to GPs is lacking. Key characteristics of EO that could optimise success include ensuring the EO program is tailored, involves practical learning and uses incentives that are meaningful to clinicians.

A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool

OBJECTIVES

The objectives of this study were to determine the interrater reliability (IRR) of assessment of multiple systematic reviews (AMSTAR) 2 for reviews of pharmacological or psychological interventions for the treatment of major depression, to compare it to that of AMSTAR and risk of bias in systematic reviews (ROBIS), and to assess the convergent validity between the appraisal tools.

STUDY DESIGN AND SETTING

Two groups of four raters were each assigned one of two samples of 30 systematic reviews. All eight raters applied AMSTAR 2 to their sample. Each group also applied either AMSTAR or ROBIS. Fleiss' kappa and Gwet's AC₁ were calculated, and agreement between the tools was assessed.

RESULTS

The median kappa values as a measure of IRR indicated a moderate agreement for AMSTAR 2 (median = 0.51), a substantial agreement for AMSTAR (median = 0.62), and a fair agreement for ROBIS (median = 0.27). Validity results showed a positive association for AMSTAR and AMSTAR 2 (r = 0.91) as well as ROBIS and AMSTAR 2 (r = 0.84). For the overall rating, AMSTAR 2 showed a high concordance with ROBIS and a lower concordance with AMSTAR.

CONCLUSION

The IRR of AMSTAR 2 was found to be slightly lower than the IRR of AMSTAR and higher than the IRR of ROBIS. Validity measurements indicate that AMSTAR 2 is closely related to both ROBIS and AMSTAR.