The landscape of systematic reviews in urology (1998 to 2015): an assessment of methodological quality

Enucleation of the prostate as retreatment for recurrent or residual benign prostatic obstruction: a systematic review and a meta-analysis

BACKGROUND

Anatomical endoscopic enucleation of the prostate (AEEP) is recommended for first line surgical treatment of benign prostatic obstruction (BPO) caused by moderate and large prostatic adenoma. However, its role in the retreatment setting after failed previous surgical treatment for BPO remains uncaptured. In this scope, we performed a systematic review and meta-analysis aiming to assess the safety and efficacy of AEEP in the retreatment setting.

METHODS

We searched PubMed, Cochrane Library and Embase databases from inception to March 2022 for prospective or retrospective studies involving patients undergoing prostatic enucleation for recurrent or residual BPO after previous standard or minimally invasive surgical treatments for BPO. Based on data availability, we performed a meta-analysis comparing AEEP in patients with recurrent or residual BPO versus AEEP for primary BPO.

PROSPERO

CRD42022308941).

RESULTS

We included 15 studies in the systematic review and 10 in the meta-analysis (6553 patients, 841 with recurrent or residual BPO and 5712 with primary BPO). All included studies involved patients undergoing HoLEP or ThuLEP. In terms of Qmax, post-void residual, International Prostate Symptom Score, removed adenoma, operative time, duration of catheterization and hospital stay, as well as complications, HoLEP for recurrent or residual BPO was equally effective compared to HoLEP for primary BPO up to 1 year postoperatively. Importantly, the beneficial effect of HoLEP on the retreatment setting was observed after previous standard or minimally invasive surgical treatments for BPO. The overall strength of evidence for all outcomes was deemed very low.

CONCLUSIONS

HoLEP may be safely and effectively used in experienced hands for the surgical treatment of recurrent or residual BPO in patients with large or moderate prostates following previous open, endoscopic or minimally invasive surgical treatment for BPO.

Synchronous Surgical Management of Erectile Dysfunction and Stress Urinary Incontinence: A Systematic Review and Meta-Analysis of Reoperation Rates

INTRODUCTION

The efficacy, safety, and postoperative outcomes of synchronous surgery for concomitant erectile dysfunction (ED) and stress urinary incontinence (SUI) remain unclear.

OBJECTIVES

We performed a systematic review and meta-analysis to evaluate the available synchronous surgical approaches for concomitant ED and SUI and to assess the reoperation rates compared to asynchronous surgery and surgery only for ED or SUI.

METHODS

We searched PubMed, Cochrane Library, and Embase databases until June 2022 for relevant studies. Based on data availability, we performed a meta-analysis of odds ratios (ORs) comparing reoperation rates after synchronous surgery in patients with concomitant ED and SUI versus asynchronous surgery, as well as surgery solely for ED or SUI (PROSPERO: CRD42022326941).

RESULTS

We included 18 studies in the systematic review (16,517 patients) and 5 in the meta-analysis. Comparing synchronous implantation of penile prosthesis and artificial urinary sphincter (AUS) versus asynchronous surgery, no statistically significant differences were observed in the reoperation rates [OR:0.98, 95% confidence interval (CI): 0.52-1.84, I²:0%). Comparing synchronous implantation of both penile prosthesis and AUS versus implantation of only a penile prosthesis or an AUS, combined surgery was associated with higher reoperation rates (OR:2.02, 95%CI: 1.29-3.16, I²:36% and OR:1.7, 95%CI: 1.25-2.32, I²:0%, respectively). Synchronous surgery led to high satisfaction rates and significant improvement in ED and SUI. Evidence for the combination of penile prosthesis with a male sling or the ProACT device is low, but data suggests it may be safe and effective. The synchronous placement of a Mini-Jupette sling and penile prosthesis represents a promising treatment modality for the correction of ED and mild SUI and/or climacturia.

CONCLUSIONS

Synchronous penile prosthesis and AUS implantation appears safe and effective in patients with severe ED and SUI. Further high-quality studies are mandatory to strengthen the current scarce evidence for synchronous surgery in patients with ED and SUI. Pyrgidis N, Barham DW, Hammad M, et al. Synchronous Surgical Management of Erectile Dysfunction and Stress Urinary Incontinence: A Systematic Review and Meta-Analysis of Reoperation Rates. Sex Med Rev 2022;XX:XXX-XXX.

Appraising the quality of meta-analysis for breast cancer treatment in the adjuvant setting: A systematic review

INTRODUCTION

Breast cancer is the tumor with highest incidence in women worldwide and adjuvant treatment is extremely important to achieve disease control. Given the relevance of systematic reviews, their rigor should be warranted to avoid biased conclusions. Our objective was to investigate the methodological quality of meta-analysis of early breast cancer adjuvant treatment.

MATERIAL AND METHODS

Comprehensive searches were performed using electronic databases from 1/1/2007 to 11/12/2018. All studies identified as a systematic review with meta-analysis investigating the efficacy of breast cancer adjuvant treatments were included. Two reviewers independently assessed titles and abstracts, then full-texts for eligibility. Quality was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) version 2 tool.

RESULTS

Of 950 citations retrieved, 66 studies (7.0%) were deemed eligible. Methodological quality was highly variable, median AMSTAR score 8.5 (IQR 7-9.5) and range 0-16. There was a weak positive correlation between journal impact factor and AMSTAR score (r = 0.17) and citation rate and AMSTAR score (r = 0.16). Cochrane Systematic Reviews were of higher quality than reviews from other journals. Overall confidence was critically low for 61 (92.4%) studies, and the least well-reported domains were the statement of conflict of interest and funding source for the included studies (4.6%), the report of a pre-defined study protocol (15.2%), and the description of details of excluded studies (6.1%).

CONCLUSIONS

Our findings reinforce concerns about the design, conduction and interpretation of meta-analysis in current literature. Methodological quality should be carefully considered and journal editors, decision makers and readers in general, must follow a critical approach to this studies.

The quality of systematic reviews and meta-analyses assessing the treatment and management of male infertility

BACKGROUND

Male factor infertility (MFI) is a common medical condition which requires high-quality research to guide clinical practice; however, systematic reviews (SRs) and meta-analyses (MAs) often vary in quality, raising concerns regarding the validity of their results. We sought to perform an objective analysis of SRs and MAs in MFI treatment and management and to report on the quality of published literature.

METHODS

A comprehensive search in PubMed/MEDLINE and Embase was used to identify relevant publications. Primary search terms were male infertility, male sterility, and male subfertility. Two authors independently performed searches, screened citations for eligibility, extracted data for analysis, and graded methodological quality using the validated AMSTAR (A Measurement Tool to Assess Systematic Reviews) instrument, a validated tool used in the critical appraisal of SRs/MAs.

RESULTS

Of 27 publications met inclusion criteria and were included in the analysis. Mean AMSTAR score (± SD) among all publications was 7.4 (1.9) out of 11, reflecting "fair to good" quality. Non-pharmacological medical treatment for MFI was the most commonly assessed intervention (n = 13, 48.1%). No publications met all AMSTAR criteria. While the number of SRs/MAs has increased over time (P = 0.037), the quality of publications has not significantly changed (P = 0.72). SRs/MAs of the Cochrane Library had higher AMSTAR score than non-Cochrane SRs/MAs (8.5 vs 6.3, P = 0.002).

CONCLUSIONS

The methodological quality of SRs/MAs should be assessed to ensure high-quality evidence for clinical practice guidelines in MFI treatment and management. This review highlights a need for increased effort to publish high-quality studies in MFI treatment and management.

How Can Alternative Exercise Traditions Help Against the Background of the COVID-19 in Cancer Care? An Overview of Systematic Reviews

Purpose

To evaluate the quality of systematic reviews/meta-analyses (SR/MAs) on alternative exercise traditions in cancer care.

Methods

PubMed, Cochrane Library, Web of Science, Ovid, EBSCOhost, WanFang Database, China National Knowledge Infrastructure, and VIP Database were searched from their inception to June 30, 2020. The search used a combination of subject words and free words. The search terms included “Tai Chi,” “Qigong,” “Baduanjin,” “Yoga,” “Pilates,” “Cancer,” “Meta-analysis,” and “Systematic review.” Two researchers independently performed literature screening and data extraction and used AMSTAR scale and PRISMA statement to evaluate the methodology and the quality of the evidence.

Results

A total of 26 SR/MAs were included. The AMSTAR scale score was 5–10 points, with an average of 7.46±1.33 points. Overall, the methodological quality was considered to be moderate. The PRISMA list score was 13–24 points, with an average of 19.19±2.86 points. Among the papers evaluated, eight reports were relatively complete. A total of 15 papers indicated certain defects. A total of three papers showed relatively serious information defects.

Conclusion

Relative to the control group, alternative exercise traditions may be effective in improving the quality of life, anxiety, depression, distress, and fatigue of cancer patients. However, they may not necessarily improve patients’ sleep outcome, body mass index, and pain. During convalescence, cancer patients can be encouraged to start engaging in physical exercise, and professionals can develop appropriate exercise alternatives to ensure the expected effect of exercise while ensuring the safety of patients. The methodological quality of the systematic evaluations of the intervention effects of alternative exercise traditions on cancer patients is not satisfactory. Hence, focus should be directed to the improvement of the preliminary design scheme, publication status, literature retrieval, conflict of interest, and other aspects.

Knowledge and confidence level of Canadian urology residents toward biostatistics: A national survey

INTRODUCTION

Adequate knowledge of biostatistics is essential for healthcare providers to stay up to date with medical advances and maintain an evidence-based practice. However, training in clinical research in Canadian residency programs varies considerably. Our study aimed to determine Canadian urology trainees' knowledge of biostatistics and interpretation of the scientific literature.

METHODS

We conducted a national survey of all Canadian urology residents and fellows, which assessed experiences with biostatistics, self-reported confidence with statistical questions, and knowledge of biostatistical concepts.

RESULTS

Out of 201 urology trainees, 74 (36.8%) responded to the survey. The majority of respondents disagreed or strongly disagreed with the fact that they had sufficient training in biostatistics in medical school (67.6%) or that their current knowledge was sufficient for the rest of their career (66.1%). If given the chance, 82.3% of respondents would like to learn more about biostatistics. About half of respondents were able to correctly identify ordinal variables (51.5%), discrete variables (52.9%), or interpret adjusted odds ratios (50.0%). Despite senior residents reporting more confidence on biostatistical questions, our results did not demonstrate significant differences in overall knowledge according to level of training or experience with biostatistics.

CONCLUSIONS

Our results identified important knowledge gaps among current Canadian urology trainees. Most trainees do not believe they have sufficient training in biostatistics. Knowledge of basic statistical concepts was lower than expected and did not significantly differ according to level of training. Our results highlight the need for structured, dedicated training in biostatistics for urology trainees within the new Competence by Design teaching framework.

A Longitudinal Assessment of the Reporting Quality of Randomized Controlled Trials for Surgical Interventions to Treat Nephrolithiasis Over 16 Years (2002 to 2017)

Purpose: Transparently reported, high-quality randomized controlled trials (RCTs) play a critical role in guiding evidence based clinical practice and informing evidence-based guidelines in patients with nephrolithiasis. Prior studies have found reporting quality to be low. We performed this study to assess whether the reporting of RCTs has improved over time. Materials and Methods: This study was governed by an a priori protocol. We performed a systematic literature search for RCTs analyzing nephrolithiasis treatment. Selection of eligible studies and data abstraction were performed by two of three reviewers independently and in duplicate. We developed and pilot tested a data extraction checklist based on the Consolidated Standards of Reporting Trials (CONSORT) criteria on a scale of 0 to 25. Our primary outcome measure was the mean CONSORT score. We performed statistical hypothesis testing to compare scores between 2002-2006, 2007-2011, and 2012-2017. Results: A total of 203 studies (2002-06: 38; 2007-11: 64; 2012-17: 101) met inclusion criteria. The most common procedure types studied were percutaneous nephrolithotomy (35.1%), shockwave lithotripsy (25.4%), and ureteroscopy (22.9%). Asia contributed an increasing proportion of studies (25.6%, 44.6%, and 74.3%, respectively) in these three time periods. The main journals of publication were the Journal of Endourology (23.9%), the Journal of Urology (19.5%), and Urology (8.3%). The mean ± standard error of the CONSORT summary scores was 11.4 ± 0.4, (2002 to 2006), 12.1 ± 0.3, (2007 to 2011), and 13.3 ± 0.4 (p = 0.003) reflecting an increase by 1.92 (95% confidence interval: 0.86 - 2.98). Conclusions: The number of RCTs investigating the use of urologic devices to treat stone disease has substantially increased overtime. There has been a small improvement in reporting quality; however, this remains suboptimal overall. Increased efforts to promote the transparent reporting of RCTs in endourology are warranted.

The fate of urological systematic reviews registered in PROSPERO

PURPOSE

To identify urologic systematic reviews (SRs) registered to PROSPERO that resulted in a publication, and to evaluate their methodological quality and concordance with their stated a priori protocols.

METHODS

We searched PubMed to identify urologic SR protocols registered in PROSPERO that resulted in a publication and assessed their methodological quality and protocols in relation to their stated a priori protocols in PROSPERO.

RESULTS

Of the 576 urologic SR protocols registered in PROSPERO up to December 2017, 201 (34.9%) resulted in a full SR publication, but only 40 (17.7%) updated their registration record accordingly. Publications were spread over 100 different journals, with a median time-to-publication of 29 months (95% CI 25.0-33.0). The most common topic by far was prostate cancer (59.7%), followed by voiding issues (15.3%), and renal transplantation (15.3%). Only little over half the reviews (52.74%) explicitly stated primary outcome(s) that matched the primary outcome of their corresponding PROSPERO protocol. Notable methodologic deviations from registered protocols included planned restriction on study design (33%), heterogeneity analysis (42%) and planned risk of bias analysis (65.2%).

CONCLUSION

SR authors in urology are increasingly using PROSPERO to register their titles, but our findings indicate that registration alone is not a guarantor of a high-quality SR product. There appears to be a critical need to raise the bar for review authors registering protocols in PROSPERO, with an emphasis on transparency in their publication status updates as well as deviations from their a priori protocols.

Infrequent use of clinical trials registries in published systematic reviews in urology

OBJECTIVE

Validity of systematic reviews may be affected by non-publication of statistically non-significant or unfavorable clinical trial results. One function of clinical trial registries is to make these non-published studies available and thereby reduce potential publication bias. We aim to assess the use of clinical trial registries in published systematic reviews in urology.

METHODS

We identified all systematic reviews published in the five highest-impact general urology journals that publish original research between January 1 and December 31, 2017. Full-text analysis was performed to determine if protocols included searching clinical trial registries meeting WHO Registry Network criteria.

RESULTS

Of a total of 204 search results, 92 were included in the analysis as systematic reviews. The majority searched the MEDLINE (91, 98%), EMBASE (69, 75%), and Cochrane (49, 53%) databases. Based on published methods, only 16 (17%) systematic reviews searched any clinical trial registries: 14 (15%) ClinicalTrials.gov, 6 (6%) WHO International Clinical Trials Registry Platform, and 2 (2%) ISRCTN registry. Rates of searching clinical trial registries were low regardless of the journal: 8 of 34 (24%) in European Urology; 2 of 10 (20%) in BJU International; 3 of 17 (18%) in Urology; 2 of 18 (11%) in The Journal of Urology; and 1 of 13 (8%) in World Journal of Urology.

CONCLUSION

The majority of recently published systematic reviews in urology do not routinely search clinical trial registries. Inclusion of these registries can help identify unpublished trial data, which may improve the quality of systematic reviews by reducing publication bias.

Characteristics, trend, and methodological quality of systematic reviews and meta-analyses in nuclear medicine: A bibliometric analysis of studies published between 2005 and 2016

To evaluate the characteristics, trend, and quality of systematic reviews and meta-analyses in nuclear medicine.We performed a PubMed search to identify systematic reviews and meta-analyses published between 2005 and 2016 in the field of nuclear medicine. The following data were extracted: journal name, impact factor, type of study, topics with cancer type, imaging modalities, authors (number, country, affiliation, presence of nuclear medicine specialists and statisticians, discordance between the first and corresponding authors), funding, methodological quality, methods used for quality assessment, and statistical methods.We included 185 nuclear medicine articles. Meta-analyses (n = 164; 88.6%) were published about 7 times more frequently than systematic reviews. Oncology was the most commonly studied topic (n = 125, 67.6%). The first authors were most frequently located in China (n = 73; 39.5%). PET was the most commonly used modality (n = 150; 81.1%). Both the number of authors and the ratio of discordance between the first and corresponding authors tended to progressively increase over time.The mean AMSTAR score increased over time (5.77 in 2005-2008, 6.71 in 2009-2012, and 7.44 in 2013-2016). The proportion of articles with quality assessment increased significantly (20/26 in 2005-2008, 54/65 in 2009-2012, and 79/94 in 2013-2016). The most commonly used assessment tool was quality assessment of diagnostic accuracy studies (n = 85; 54.9%).The number and quality of systematic reviews and meta-analyses in nuclear medicine have significantly increased over the review period; however, the quality of these articles varies. Efforts to overcome specific weaknesses of the methodologies can provide opportunities for quality improvement.

A critical appraisal of biomarkers in prostate cancer

PURPOSE

A number of urine and blood-based biomarker tests have been described for prostate cancer, although to date there has only been a limited exploration of the methodology behind the validation studies that underpin these tests.

METHODS

In this review, a selection of commercially available urine and blood-based biomarker tests for prostate cancer are described, and the underlying key validation studies for each test are critically appraised using the Standards for Reporting Diagnostic Accuracy (STARD) 2015 statement.

RESULTS

The ExoDx Prostate Intelliscore, SelectMDx, Progensa PCA3, Mi-Prostate Score, 4K Score, and Prostate Health Index (PHI) tests were reviewed. Most of the validation studies supporting these tests perform exploratory analyses to determine cut-off values in a post hoc manner, comprise cohorts that are primarily Caucasian, report receiver operating characteristic curves that combine the biomarker's result with established clinical nomograms and are based on a reference standard (prostate biopsy) that lacks central pathology review. Deficiencies in STARD reporting guidelines include frequent failure to provide a published study protocol, prospective study registration in a registry, a flow diagram, justification for sample size determination, a discussion of adverse events with testing, and information on how missing or indeterminate test results should be managed.

CONCLUSIONS

Key validation studies that support many commercially available urine and blood-based biomarkers for prostate cancers have deficiencies in transparency based on STARD reporting guidelines, and limitations in methodology must be considered when deciding when these tests should be applied in clinical practice.

Use of the IDEAL framework in the urological literature: where are we in 2018?

OBJECTIVES

To assess uptake and application of the IDEAL principles in original surgical procedure- or device-related clinical research studies, as well as its reported relevance as characterized by secondary publications, editorials and reviews.

MATERIALS AND METHODS

IDEAL (Idea, Development, Exploration, Assessment, Long-term study) is a framework that provides stage-specific guidance for surgical innovation and represented a major advance towards raising evidential standards. We performed a comprehensive literature search of all urology-related publications citing one or more of seven key publications on IDEAL in The Lancet and BMJ using multiple databases up to 31 December 2017.

RESULTS

We identified a total of 150 urology-related manuscripts citing IDEAL, of which 83 (55.3%) were original research and 67 (44.7%) were secondary publications. Among the original research articles, 40 (48.2%) did not explicitly apply IDEAL principles or were not surgical innovation studies. The IDEAL phases of the 43 (51.8%) remaining original research studies were IDEAL, in nine (20.9%), 27 (62.8%), four (9.3%), 0 (0%), and three publications (7.0%), respectively. Across IDEAL stages, 30 (75.0%) studies were prospective, 29 (85.3%) reported ethical oversight, and 39 (90.7%) captured treatment-related harms. None of the studies collected information on physician experience.

CONCLUSIONS

The IDEAL framework has found widespread adoption in the urology literature as witnessed by a large number of original manuscripts and secondary publications citing IDEAL; however, its application is largely limited to the early stages of surgical innovation, frequently with inappropriate and incomplete implementation. Further efforts are needed to guide investigators in the optimal use of the IDEAL framework as it relates to surgical innovation in urology.

Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR

Background

Systematic reviews (SRs) in the field of neuropathic pain (NeuP) are increasingly important for decision-making. However, methodological flaws in SRs can reduce the validity of conclusions. Hence, it is important to assess the methodological quality of NeuP SRs critically. Additionally, it remains unclear which assessment tool should be used. We studied the methodological quality of SRs published in the field of NeuP and compared two assessment tools.

Methods

We systematically searched 5 electronic databases to identify SRs of randomized controlled trials of interventions for NeuP available up to March 2015. Two independent reviewers assessed the methodological quality of the studies using the Assessment of Multiple Systematic Reviews (AMSTAR) and the revised AMSTAR (R-AMSTAR) tools. The scores were converted to percentiles and ranked into 4 grades to allow comparison between the two checklists. Gwet’s AC1 coefficient was used for interrater reliability assessment.

Results

The 97 included SRs had a wide range of methodological quality scores (AMSTAR median (IQR): 6 (5–8) vs. R-AMSTAR median (IQR): 30 (26–35)). The overall agreement score between the 2 raters was 0.62 (95% CI 0.39–0.86) for AMSTAR and 0.62 (95% CI 0.53–0.70) for R-AMSTAR. The 31 Cochrane systematic reviews (CSRs) were consistently ranked higher than the 66 non-Cochrane systematic reviews (NCSRs). The analysis of individual domains showed the best compliance in a comprehensive literature search (item 3) on both checklists. The results for the domain that was the least compliant differed: conflict of interest (item 11) was the item most poorly reported on AMSTAR vs. publication bias assessment (item 10) on R-AMSTAR. A high positive correlation between the total AMSTAR and R-AMSTAR scores for all SRs, as well as for CSRs and NCSRs, was observed.

Conclusions

The methodological quality of analyzed SRs in the field of NeuP was not optimal, and CSRs had a higher quality than NCSRs. Both AMSTAR and R-AMSTAR tools produced comparable quality ratings. Our results point out to weaknesses in the methodology of existing SRs on interventions for the management NeuP and call for future improvement by better adherence to analyzed quality checklists, either AMSTAR or R-AMSTAR.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0493-y) contains supplementary material, which is available to authorized users.

A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer

OBJECTIVE

To compare cancer-related systematic reviews (SRs) published in the Cochrane Database of SRs (CDSR) and high-impact journals, with respect to type, content, quality and citation rates.

DESIGN

Methodological SR with assessment and comparison of SRs and meta-analyses. Two authors independently assessed methodological quality using an Assessment of Multiple Systematic Reviews (AMSTAR)-based extraction form. Both authors independently screened search results, extracted content-relevant characteristics and retrieved citation numbers of the included reviews using the Clarivate Analytics Web of Science database.

DATA SOURCES

Cancer-related SRs were retrieved from the CDSR, as well as from the 10 journals which publish oncological SRs and had the highest impact factors, using a comprehensive search in both the CDSR and MEDLINE.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included all cancer-related SRs and meta-analyses published from January 2011 to May 2016. Methodological SRs were excluded.

RESULTS

We included 346 applicable Cochrane reviews and 215 SRs from high-impact journals. Cochrane reviews consistently met more individual AMSTAR criteria, notably with regard to an a priori design (risk ratio (RR) 3.89; 95% CI 3.10 to 4.88), inclusion of the grey literature and trial registries (RR 3.52; 95% CI 2.84 to 4.37) in their searches, and the reporting of excluded studies (RR 8.80; 95% CI 6.06 to 12.78). Cochrane reviews were less likely to address questions of prognosis (RR 0.04; 95% CI 0.02 to 0.09), use individual patient data (RR 0.03; 95% CI 0.01 to 0.09) or be based on non-randomised controlled trials (RR 0.04; 95% CI 0.02 to 0.09). Citation rates of Cochrane reviews were notably lower than those for high-impact journals (Cochrane reviews: mean number of citations 6.52 (range 0-143); high-impact journal SRs: 74.45 (0-652)).

CONCLUSIONS

When comparing cancer-related SRs published in the CDSR versus those published in high-impact medical journals, Cochrane reviews were consistently of higher methodological quality, but cited less frequently.

Quality assessment of systematic reviews on total hip or knee arthroplasty using mod-AMSTAR

Background

Increasing numbers of systematic reviews (SRs) on total knee arthroplasty (TKA) and total hip arthroplasty (THA) have been published in recent years, but their quality has been unclear. The purpose of this study is to evaluate the methodological quality of SRs on TKA and THA.

Methods

We searched Ovid-Medline, Ovid-Embase, Cochrane Databases (including HTA, DARE, and CDSR), CBM, CNKI, Wang Fang, and VIP, from January 2014 to December 2015 for THA and TKA. The quality of SRs was assessed using the modified 25-item “Assessment of Multiple Systematic Reviews” (mod-AMSTAR) tool, which was based on the AMSTAR scale. A T-test, nonparametric test, and linear regression were conducted to assess the relationship between bibliographical characteristics and methodological quality.

Results

Sixty-three SRs were included, from which the majority of SRs (50, 79.4%) were conducted in Asia. Only 4 reviews were rated as high quality, and most were weak in providing a priori design (6, 9.5%), not limiting the publication type (8, 13%), providing an excluded primary studies list (4, 6.3%) and reporting support for the included primary studies (1, 1.6%). Reviews published in English journals performed better than did Chinese journals in duplicate data extraction (81.3% vs 46.7%, p = 0.017; 70.8% vs 33.3%, p = 0.009) and providing source of support for the SR (87.5% vs 33.3%, P < 0.001). Reviews published in journals with a higher impact factor were associated with a higher mod-AMSTAR score (regression coefficient: 0.38, 95%CI: 0.11–0.65; P = 0.006).

Conclusion

The methodological quality of the included SRs is far from satisfactory. Authors of SRs should conform to the recommendations outlined in the mod-AMSTAR items. Areas needing improvement were providing a priori design, not limiting the publication type, providing an excluded primary studies list, and reporting conflicts of interest.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0488-8) contains supplementary material, which is available to authorized users.

"Assessing the methodological quality of systematic reviews in radiation oncology: A systematic review"

OBJECTIVE

The objective of our study was to evaluate the methodological quality of systematic reviews and meta-analyses in Radiation Oncology.

METHODS

A systematic literature search was conducted for all eligible systematic reviews and meta-analyses in Radiation Oncology from 1966 to 2015. Methodological characteristics were abstracted from all works that satisfied the inclusion criteria and quality was assessed using the critical appraisal tool, AMSTAR. Regression analyses were performed to determine factors associated with a higher score of quality.

RESULTS

Following exclusion based on a priori criteria, 410 studies (157 systematic reviews and 253 meta-analyses) satisfied the inclusion criteria. Meta-analyses were found to be of fair to good quality while systematic reviews were found to be of less than fair quality. Factors associated with higher scores of quality in the multivariable analysis were including primary studies consisting of randomized control trials, performing a meta-analysis, and applying a recommended guideline related to establishing a systematic review protocol and/or reporting.

CONCLUSIONS

Systematic reviews and meta-analyses may introduce a high risk of bias if applied to inform decision-making based on AMSTAR. We recommend that decision-makers in Radiation Oncology scrutinize the methodological quality of systematic reviews and meta-analyses prior to assessing their utility to inform evidence-based medicine and researchers adhere to methodological standards outlined in validated guidelines when embarking on a systematic review.

Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool

OBJECTIVES

No gold standard exists to assess methodological quality of systematic reviews (SRs). Although Assessing the Methodological Quality of Systematic Reviews (AMSTAR) is widely accepted for analyzing quality, the ROBIS instrument has recently been developed. This study aimed to compare the capacity of both instruments to capture the quality of SRs concerning psoriasis interventions.

STUDY DESIGN AND SETTING

Systematic literature searches were undertaken on relevant databases. For each review, methodological quality and bias risk were evaluated using the AMSTAR and ROBIS tools. Descriptive and principal component analyses were conducted to describe similarities and discrepancies between both assessment tools.

RESULTS

We classified 139 intervention SRs as displaying high/moderate/low methodological quality and as high/low risk of bias. A high risk of bias was detected for most SRs classified as displaying high or moderate methodological quality by AMSTAR. When comparing ROBIS result profiles, responses to domain 4 signaling questions showed the greatest differences between bias risk assessments, whereas domain 2 items showed the least.

CONCLUSION

When considering SRs published about psoriasis, methodological quality remains suboptimal, and the risk of bias is elevated, even for SRs exhibiting high methodological quality. Furthermore, the AMSTAR and ROBIS tools may be considered as complementary when conducting quality assessment of SRs.