Replication of systematic reviews: is it to the benefit or detriment of methodological quality?

OBJECTIVES

To perform an overview of the overlap of systematic reviews (SRs) assessing direct oral anticoagulants and characterize these reviews in terms of bias and methodological quality (PROSPERO: CRD42022316273).

STUDY DESIGN AND SETTING

A PubMed-indexed search was performed from inception to January 31, 2022 to identify SRs evaluating direct oral anticoagulants in patients treated for an acute venous thromboembolism. The risk of bias of these SRs was assessed according to the Risk Of Bias In Systematic reviews tool. Redundancy was defined as overlap in terms of the type of population considered, the interventions compared, and the studies included.

RESULTS

A total of 144 SRs were evaluated, of which 26 (18.1%) were classified as original, 87 (60.4%) as conceptual replications, and 31 (21.5%) as excessive replications. The risk of bias was high in 19 (73.1%) of the original SRs, 65 (74.7%) of the conceptual replications, and 21 (67.7%) of the excessive replications. Compared to the original SRs, the overall methodological quality was not improved in either conceptual or excessive replications.

CONCLUSION

A large number of SRs was classified as replications; a fifth constituted excessive replications. The replications showed no improvement in overall methodological quality compared to the original SRs.

Methodological and reporting quality assessment of network meta-analyses in anesthesiology: a systematic review and meta-epidemiological study

PURPOSE

The scientific rigour of the conduct and reporting of anesthesiology network meta-analyses (NMAs) is unknown. This systematic review and meta-epidemiological study assessed the methodological and reporting quality of NMAs in anesthesiology.

METHODS

We searched four databases, including MEDLINE, PubMed, Embase, and the Cochrane Systematic Reviews Database, for anesthesiology NMAs published from inception to October 2020. We assessed the compliance of NMAs against A Measurement Tool to Assess Systematic Reviews (AMSTAR-2), Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement for Network Meta-Analyses (PRISMA-NMA), and PRISMA checklists. We measured the compliance across various items in AMSTAR-2 and PRISMA checklists and provided recommendations to improve quality.

RESULTS

Using the AMSTAR-2 rating method, 84% (52/62) of NMAs were rated "critically low." Quantitatively, the median [interquartile range] AMSTAR-2 score was 55 [44-69]%, while the PRISMA score was 70 [61-81]%. Methodological and reporting scores showed a strong correlation (R = 0.78). Anesthesiology NMAs had a higher AMSTAR-2 score and PRISMA score if they were published in higher impact factor journals (P = 0.006 and P = 0.01, respectively) or followed PRISMA-NMA reporting guidelines (P = 0.001 and P = 0.002, respectively). Network meta-analyses from China had lower scores (P < 0.001 and P < 0.001, respectively). Neither score improved over time (P = 0.69 and P = 0.67, respectively).

CONCLUSION

The current study highlights numerous methodological and reporting deficiencies in anesthesiology NMAs. Although the AMSTAR tool has been used to assess the methodological quality of NMAs, dedicated tools for conducting and assessing the methodological quality of NMAs are urgently required.

STUDY REGISTRATION

PROSPERO (CRD42021227997); first submitted 23 January 2021.

Development and validation of a Medication Adherence Universal Questionnaire: the MAUQ

BACKGROUND

Different questionnaires assess self-reported medication adherence and others quantify aspects of patients attitudes towards medication, but not together in a single instrument. Gathering these two aspects in a single instrument could reduce patients survey burden.

AIM

The aim of this study was to develop the Medication Adherence Universal Questionnaire (MAUQ) using the Maastricht Utrecht Adherence in Hypertension short version (MUAH-16) factorial structure as the hypothesized model.

METHOD

A multistep process started with the modification of the MUAH-16 to obtain the MAUQ. Patients using at least one antihypertensive medicine were recruited. The two questionnaires, the MUAH-16 and MAUQ, were applied. A confirmatory factor analysis (CFA) was performed using the initial MUAH-16 s-order 4-factor model. An additional bifactor model with four uncorrelated factors and an overall score was tested. The comparative fit index (CFI), root mean square error of approximation (RMSEA) with confidence intervals (CIs), and standardized root mean squared residual (SRMR) were used to assess both models.

RESULTS

A sample of 300 hypertensive patients completed the instruments. The CFA with the second-order 4-factor solution resulted in similar results for the MUAH-16 and MAUQ: CFIs of 0.934 and 0.930, RMSEAs of 0.043 [CI 0.030-0.056] and 0.045 [CI 0.031-0.057] and SRMRs of 0.060 and 0.061, respectively. The CFA with the bifactor model showed slightly better results for both the MUAH-16 and MAUQ: CFIs of 0.974 and 0.976, RMSEAs of 0.030 [CI 0.005-0.046] and 0.028 [CI 0.001-0.044], and SRMRs of 0.043 and 0.044, respectively.

CONCLUSION

CFA demonstrated that the MAUQ presented a better fit to both models than the MUAH-16, obtaining a robust universal free instrument to assess medicine-taking behaviour and four medicine beliefs components.

Mapping the characteristics, methodological quality and standards of reporting of network meta-analyses on antithrombotic therapies: An overview

BACKGROUND

Although a large number of network meta-analyses (NMAs) in the field of cardiology are available, little is known about their methodological quality. We aimed to map the characteristics and critically appraised the standards of conduct and evidence reporting of NMAs assessing antithrombotic therapies for the treatment or prophylaxis of heart diseases and cardiac surgical procedures.

METHODS

We systematically searched PubMed and Scopus to identify NMAs comparing the clinical effects of antithrombotic therapies. Overall characteristics of the NMAs were extracted and their reporting quality and methodological quality were evaluated using the PRISMA-NMA checklist and AMSTAR-2, respectively.

RESULTS

We found 86 NMAs published between 2007 and 2022. Comparisons among direct-acting oral anticoagulants were available in 61 (71%) NMAs. Although around 75% of NMAs stated that they followed international guidelines for conduct and reporting, only one third provided a protocol/register. Complete search strategies and publication bias assessment were lacking in around 53% and 59% of studies, respectively. Most NMAs (n = 77, 90%) provided supplemental material; however, only 5 (6%) made the complete raw data available. Network diagrams were depicted in most studies (n = 67, 78%), yet network geometry was described in only 11 (12.8%) of them. Mean adherence to the PRISMA-NMA checklist was 65.1 ± 16.5%. AMSTAR-2 assessment showed 88% of the NMAs had critically low methodological quality.

CONCLUSION

Although there is a wide diffusion of NMA-type studies on antithrombotics for heart diseases, their methodological and reporting quality remains suboptimal. This may reflect fragile clinical practices due to misleading conclusions from critically low-quality NMAs.

Efficacy and safety of pharmacological interventions for managing sickle cell disease complications in children and adolescents: Systematic review with network meta-analysis

This study aimed to synthesize the evidence on the effects of disease-modifying agents for managing sickle cell disease (SCD) in children and adolescents by means of a systematic review with network meta-analyses, surface under the cumulative ranking curve (SUCRA) and stochastic multicriteria acceptability analyses (SMAA) (CRD42022328471). Eightteen randomized controlled trials (hydroxyurea [n = 7], l-arginine [n = 3], antiplatelets [n = 2], immunotherapy/monoclonal antibodies [n = 2], sulfates [n = 2], docosahexaenoic acid [n = 1], niprisan [n = 1]) were analyzed. SUCRA and SMAA demonstrated that hydroxyurea at higher doses (30 mg/kg/day) or at fixed doses (20 mg/kg/day) and immunotherapy/monoclonal antibodies are more effective for preventing vaso-occlusive crisis (i.e., lower probabilities of incidence of this event; 14, 25, and 30%, respectively), acute chest syndrome (probabilities ranging from 8 to 30%), and needing of transfusions (11-31%), while l-arginine (100-200 mg/kg) and placebo were more prone to these events. Therapies were overall considered safe; however, antiplatelets and sulfates may lead to more severe adverse events. Although the evidence was graded as insufficient and weak, hydroxyurea remains the standard of care for this population, especially if a maximum tolerated dose schedule is considered.

Epidemiological, methodological, and statistical characteristics of network meta-analysis in anaesthesia: a systematic review

BACKGROUND

Network meta-analyses (NMAs) combine direct and indirect estimates to provide mixed (or network) estimates of effect sizes. The scientific rigour of the conduct and reporting of anaesthesia NMAs is unknown. This review assessed the epidemiological, methodological, and statistical characteristics of anaesthesia NMAs.

METHODS

We searched four databases for anaesthesia NMAs and developed a 64-item checklist to evaluate NMAs. For 29 binary items, we defined compliance as 'the ratio of NMAs that was awarded a 'yes' for that item, divided by the total number of NMAs. The compliance of such binary items was reclassified as very low (≤25%), low (26-50%), fair (51-75%), and high (>75%). We amalgamated findings from 29 key items to provide specific recommendations (post hoc). We compared the compliance of NMAs in anaesthesia across 26 items, with that of cancer NMAs and Cochrane NMAs, and analysed improvement over time (post hoc).

RESULTS

Among 62 included NMAs, compliance was low (26-50%) for protocol registration, use of PRISMA-NMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for NMA), publication bias assessment, evidence appraisal, reporting of Bayesian methodology and consistency evaluation. Compliance was very low (≤25%) for bias assessment, biostatistician involvement, search specialist, and use of predefined important differences.

CONCLUSIONS

Anaesthesia NMAs need improvement in their conduct and reporting. Anaesthesia journals should mandate the registration of protocols and reporting of NMAs using PRISMA-NMA. Authors should carefully assess publication bias, and use updated bias assessment tools, and evidence appraisal methods designed for NMAs.

SYSTEMATIC REVIEW PROTOCOL

PROSPERO CRD42021227608.

Efficacy and safety of pharmacological interventions for managing sickle cell disease in children and adolescents: protocol for a systematic review with network meta-analysis

INTRODUCTION

Sickle cell disease (SCD), an inherited haemoglobinopathy, has important impact on morbidity and mortality, especially in paediatrics. Previous systematic reviews are limited to adult patients or focused only on few therapies. We aim to synthesise the evidence on efficacy and safety of pharmacological interventions for managing SCD in children and adolescents.

METHODS AND ANALYSIS

This systematic review protocol is available at Open Science Framework (doi:10.17605/OSF.IO/CWAE9). We will follow international recommendations on conduction and report of systematic reviews and meta-analyses. Searches will be conducted in PubMed, Scopus and Web of Science (no language nor time restrictions) (first pilot searches performed in May 2022). We will include randomised controlled trials comparing the effects of disease-modifying agents in patients with SCD under 18 years old. Outcomes of interest will include: vaso-occlusive crisis, haemoglobin levels, chest syndrome, stroke, overall survival and adverse events. We will provide a narrative synthesis of the findings, and whenever possible, results will be pooled by means of pairwise or Bayesian network meta-analyses with surface under the cumulative ranking curve analyses. Different statistical methods and models will be tested. Dichotomous outcomes will be reported as OR, risk ratio or HR, while continuous data will be reported as standard mean differences, both with 95% CI/credibility interval. The methodological quality of the trials will be evaluated using the Risk of Bias 2.0 tool, and the certainty of the evidence will be assessed with the Grading of Recommendations Assessment, Development and Evaluation approach.

ETHICS AND DISSEMINATION

This study refers to a systematic review, so no ethics approval is necessary. We intent to publish our findings in international, peer-reviewed journal. Data will also be presented to peers in scientific events. Additionally, the results obtained in this study may contribute towards the update of therapeutic guidelines and for the development of health policies for SCD.

PROSPERO REGISTRATION NUMBER

CRD42022328471.

Evaluation of the Reporting Standard Guidelines of Network Meta-Analyses in Physical Therapy: A Systematic Review

The concept of network meta-analyses (NMA) has been introduced to the field of physical therapy. However, the reporting standard guidelines of these studies have not been evaluated. In this systematic review, we included all published NMA physical therapy studies that compared the clinical efficacy of three or more interventions to evaluate whether NMAs in physical therapy exhibit adequate reporting recommendations. PubMed, EMBASE, Web of Science, and the Cochrane Library were searched up to 30 June 2022. Among the 252 identified articles, 19 NMAs including 805 randomized controlled trials were included. We applied both preferred reporting items for systematic reviews and meta-analysis (PRISMA) and PRISMA-NMA checklists, which are 27- and 32-item reporting standard guidelines assessment tools, respectively. Protocol registrations (68.4%), risk of bias across studies (63.2%), additional analysis (57.9%), and funding (31.6%) were problematic items considering the PRISMA guidelines. Four studies reported all five new NMA-reporting items, and 15 (78.9%) did not address items S1-5 from the PRISMA-NMA guidelines. The median score (interquartile range) of the reporting standard guidelines was 27.0 (25.8-28.0). The identified shortcomings of published NMAs should be addressed while training researchers, and they should be encouraged to apply PRISMA-NMA, as a recognized tool for assessing NMA reporting guidelines is required.

Methodological assessment and overall confidence in the results of systematic reviews with network meta-analyses in Endodontics

AIM

The aims of the study were to assess the methodological quality of systematic reviews with network meta-analyses (NMAs) in Endodontics using the "A MeaSurement Tool to Assess systematic Reviews" (AMSTAR 2) tool, and to evaluate the overall confidence in the results of the individual reviews included in the analysis.

METHODOLOGY

Systematic reviews with NMAs within the specialty of Endodontics published in English were identified from the PubMed, EbBSCOhost and SCOPUS databases from inception to July 2021. Two reviewers were involved independently in the selection of the reviews, data extraction, methodological quality assessment and overall confidence rating. Disagreements were resolved by discussion between the reviewers to achieve consensus; if disagreements persisted, a third reviewer made the final decision. The methodological quality of the included NMAs was appraised using the AMSTAR 2 checklist, which contains 16 items. The reviewers scored each item - 'Yes' - when the item was fully addressed, 'Partial Yes' - when the item was not fully addressed, or 'No' - when the item was not addressed. The overall confidence in the results of each review was classified as 'High', 'Moderate', 'Low' or 'Critically low' based on the criteria reported by the AMSTAR 2 developers.

RESULTS

Twelve systematic reviews with NMAs were included. All the NMAs adequately reported Item 1 ("Did the research questions and inclusion criteria for the review include the components of PICO?"), Item 8 ("Did the review authors describe the included studies in adequate detail?"), Item 9 ("Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?") and Item 16 ("Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?") , whereas only one NMA reported Item 10 adequately ("Did the review authors report on the sources of funding for the studies included in the review?"). The overall confidence in the results of eight reviews was categorised as "Critically low", one review was "Low", two reviews were "Moderate" and one review was "High".

CONCLUSION

The overall confidence in the results for the majority of systematic reviews with NMAs in Endodontics was judged to be 'Critically low' as their methodological quality was below the necessary standard. AMSTAR 2 and PRISMA for NMA guidelines are available to guide authors to produce high quality systematic reviews with NMAs and for Editors and peer-reviewers when assessing submissions to journals.

The methodological quality on systematic reviews of surgical randomised controlled trials: A cross-sectional survey

BACKGROUND

Systematic reviews of RCTs have been developed to address end-users' needs and are regarded as the highest level of evidence. Flaws in the design, conduct and analyses of a systematic review can lead to erroneous conclusions and increase the research waste.

OBJECTIVE

We undertook a cross-sectional survey to identify the critical areas of weakness in systematic reviews for surgical interventions by AMSTAR 2.

METHODS

We searched PubMed, EMbase and Cochrane Library to summarize systematic reviews of surgical RCTs published in 2017. The information regarding general characteristics and methodological characteristics were gathered. We conducted descriptive analyses of study characteristics of included systematic reviews and explored the difference among varied methodological quality.

RESULTS

Totally 141 systematic reviews were identified. We found only four reviews (2.8%) were high quality, 3 (2.1%) were moderate quality, 8 (5.7%) were low quality, and the remaining 126 (89.4%) were of critical low quality. The critical weaknesses were lack of pre-registration or published protocols (29.1%), comprehensive literature search (17.7%), lists of excluded studies and reasons for exclusion (19.1%), description of detailed interventions (8.5%), extraction of funding source from included trials (10.6%), and consideration of the risk of bias of included trials when synthesized (16.3%) and interpret (20.6%) the results. Higher methodological quality was only positively associated with Cochrane systematic review.

CONCLUSION

Although two-thirds of included systematic reviews in the field of surgery were published in journals ranking Q1, the methodological quality is suboptimal and needs to be substantially improved. More efforts of multi-disciplinary teams' collaboration, continual education and training, integrally connection between primary studies and systematic review and contributing surgical research to practice should be imperative.

Critical evaluation of reporting quality of network meta-analyses assessing the effectiveness of acupuncture

BACKGROUND

and purpose: Network meta-analyses (NMAs) comparing the effectiveness of multiple acupuncture have been published but the key concepts underlying NMAs have not been properly reported. This critical evaluation aims to assess the completeness of reporting for NMAs of acupuncture to enhance the validity of findings.

MATERIALS AND METHODS

Five databases were searched. The characteristics and reporting quality based on the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension Statement for reporting of NMA (PRISMA-NMA) were evaluated. NMAs were categorized into two groups according to the publication date, reporting guideline, and intervention. The group differences and correlation coefficients were calculated.

RESULTS

Forty-two NMAs of acupuncture were evaluated. The reporting quality for new items in the key concepts of NMAs was low (median 70.0 % (interquartile range 60.0-80.0)). While the issue of consistency was well reported, the assumption of transitivity and network geometry showed poor reporting. Seventeen studies that followed the PRISMA-NMA guideline showed a higher reporting rate for essential concepts of NMA. The recency of publication did not guarantee clear reporting.

CONCLUSION

The reporting quality of NMAs of acupuncture was low. The researchers should follow the guidelines on the reporting of NMAs.

Methodological standards for conducting and reporting meta-analyses: Ensuring the replicability of meta-analyses of pharmacist-led medication review

BACKGROUND

Meta-analyses of clinical pharmacy services are frequently criticized for restricted data transparency and reproducibility.

OBJECTIVES

To describe the methodological characteristics of meta-analyses of pharmacist-led medication reviews, to identify the elements that limit their replicability and robustness, and to propose recommendations for an appropriate conduction and reporting.

METHODS

A meta-research study was conducted. Systematic searches of the PubMed, Scopus, and Web of Science databases were performed to identify meta-analyses of pharmacist services. Meta-analyses assessing the effect of pharmacist-led medication reviews were selected for data extraction, analysis and replication. Two replication exercises were performed for the two most common outcomes: (i) considering the data provided by authors to construct the meta-analysis and (ii) considering the raw data available in the primary studies included. Prediction intervals (PI), fragility index (FI), and number needed to treat (NNT) were also calculated for each replicated meta-analysis.

RESULTS

Nine studies reporting meta-analyses about pharmacist-led medication review were found comprising 30 different outcomes. Eleven meta-analyses, including six for hospital admission and five for mortality, were replicated. In five meta-analyses, the pooled effect sizes of the replicated meta-analyses differed from the original ones. Only four meta-analyses mentioned the statistical method used. Other meta-analytic parameters (e.g., q-value, tau2) were omitted in all studies. In nine meta-analyses, the data from primary studies had been incorrectly extracted for at least one variable. The PI demonstrated that the uncertainty intervals of the effect sizes were always underestimated by the authors. NNTs showed wide intervals, ranging from benefit to harm, in almost all meta-analyses. Nine recommendations to facilitate the replication of a meta-analysis were proposed: providing all original data needed to build the analysis; informing about the imputed data or data obtained from different sources; performing sensitivity analyses for imputed or unpublished data; inform about all the statistical methods used; providing all statistical results; and reporting the PI, FI and NNT.

CONCLUSION

Errors in data extraction and poor reporting of meta-analytic parameters are common in the pharmacy literature. We proposed nine recommendations to enhance data reproducibility and interpretability. Journal editors and peer reviewers should ensure that authors strictly comply with minimum standards for conduction and reporting of meta-analyses.

Prior Choices of Between-Study Heterogeneity in Contemporary Bayesian Network Meta-analyses: an Empirical Study

BACKGROUND

Network meta-analysis (NMA) is a popular tool to compare multiple treatments in medical research. It is frequently implemented via Bayesian methods. The prior choice of between-study heterogeneity is critical in Bayesian NMAs. This study evaluates the impact of different priors for heterogeneity on NMA results.

METHODS

We identified all NMAs with binary outcomes published in The BMJ, JAMA, and The Lancet during 2010-2018, and extracted information about their prior choices for heterogeneity. Our primary analyses focused on those with publicly available full data. We re-analyzed the NMAs using 3 commonly-used non-informative priors and empirical informative log-normal priors. We obtained the posterior median odds ratios and 95% credible intervals of all comparisons, assessed the correlation among different priors, and used Bland-Altman plots to evaluate their agreement. The kappa statistic was also used to evaluate the agreement among these priors regarding statistical significance.

RESULTS

Among the selected Bayesian NMAs, 52.3% did not specify the prior choice for heterogeneity, and 84.1% did not provide rationales. We re-analyzed 19 NMAs with full data available, involving 894 studies, 173 treatments, and 395,429 patients. The correlation among posterior median (log) odds ratios using different priors were generally very strong for NMAs with over 20 studies. The informative priors produced substantially narrower credible intervals than non-informative priors, especially for NMAs with few studies. Bland-Altman plots and kappa statistics indicated strong overall agreement, but this was not always the case for a specific NMA.

CONCLUSIONS

Priors should be routinely reported in Bayesian NMAs. Sensitivity analyses are recommended to examine the impact of priors, especially for NMAs with relatively small sample sizes. Informative priors may produce substantially narrower credible intervals for such NMAs.

Methodological quality and risk of bias of meta-analyses of pharmacy services: A systematic review

BACKGROUND

A suboptimal meta-analysis with misleading conclusions, frequently published in the healthcare journals, can compromise decision making in clinical practice.

OBJECTIVE

To evaluate the reporting quality, methodological quality, and risk of bias of meta-analyses of pharmacy services.

METHODS

Systematic searches to identify all the meta-analyses reporting the effect of pharmacy services were performed in PubMed, Scopus, and Web of Science. The reporting quality, the methodological quality, and the risk of bias of the included meta-analyses were evaluated using PRISMA checklist, R-AMSTAR, and ROBIS, respectively.

RESULTS

A total of 109 meta-analyses were eligible for the study. The heterogeneity, the quality of evidence, and the quality analyses were poorly reported on authors' conclusions (14.3%, 14.7%, and 17.4%, respectively). The median scores of PRISMA and R-AMSTAR tolls were 24 (IQR 21.75-25), and 30 (IQR 27-32.5), respectively. Additionally, most of the studies were considered as high risk of bias (n = 83, 76.1%). No association between the date of publication and guideline compliance exists. PRISMA score was higher in studies published in high impact factor journals (rho = 0.313; p = 0.002), in articles that reported the quality of evidence obtained (p = 0.018), and in those that stated the need for future studies in their conclusions (p = 0.011). R-AMSTAR score was higher in studies published in high impact factor journals (rho = 0.338; p = 0.001), in those which reported the quality of evidence (p = 0.002), and in articles that described the quality analyses in their conclusions (p = 0.046). An association between the risk of bias and the recognition of the need for further studies in their conclusions (p = 0.041) was also found.

CONCLUSION

The rapid increase of the meta-analyses of pharmacy services was not associated with higher quality. Mechanistic meta-analyses with poor conclusions are commonly published. Quality of the analyses, strength of evidence, heterogeneity, and absence of confrontation with current guidelines are rarely considered when synthetizing evidence and making recommendations.

Dressing interventions to heal pressure ulcers: A protocol for an overview of systematic reviews and meta-analysis

BACKGROUND

Pressure ulcer (PU) is defined as a lesion or trauma to the skin and underlying tissue resulting from unrelieved pressure, shear, friction, moisture, or a combination of all these, usually appearing over a bony prominence. We aim to evaluate the credibility of systematic reviews and meta-analyses that assess the effectiveness, safety, and economy of the dressing treatments for PU through an overview.

METHODS

We searched the following electronic bibliographic databases: PubMed, Embase, Cochrane Library, CINAHL Complete, PsycARTICLES, PsycINFO, DynaMed Plus, as well as the Chinese databases without any language restriction. We will include meta-analyses that dressings treatments in the management of PUs. For each meta-analysis, we will estimate the effect size of a treatment through the random-effect model and the fixed-effect model, and we will evaluate between-study heterogeneity (Cochrane's Q and I statistics) and small-study effect (Egger's test); we will also estimate the evidence of excess significance bias. Methodological quality of each meta-analysis will be evaluated by using Assessment of Multiple Systematic Reviews 2.

RESULTS

This study is ongoing and the results will be submitted to a peer-reviewed journal for publication.

ETHICS AND DISSEMINATION

Ethical approval is not applicable, since this is an overview based on published articles.

PROTOCOL REGISTRATION NUMBER

The protocol has been registered on PROSPERO under the number CRD42020161232.

Statistical analyses and quality of individual participant data network meta-analyses were suboptimal: a cross-sectional study

BACKGROUND

Network meta-analyses using individual participant data (IPD-NMAs) have been increasingly used to compare the effects of multiple interventions. Although there have been many studies on statistical methods for IPD-NMAs, it is unclear whether there are statistical defects in published IPD-NMAs and whether the reporting of statistical analyses has improved. This study aimed to investigate statistical methods used and assess the reporting and methodological quality of IPD-NMAs.

METHODS

We searched four bibliographic databases to identify published IPD-NMAs. The methodological quality was assessed using AMSTAR-2 and reporting quality assessed based on PRISMA-IPD and PRISMA-NMA. We performed stratified analyses and correlation analyses to explore the factors that might affect quality.

RESULTS

We identified 21 IPD-NMAs. Only 23.8% of the included IPD-NMAs reported statistical techniques used for missing participant data, 42.9% assessed the consistency, and none assessed the transitivity. None of the included IPD-NMAs reported sources of funding for trials included, only 9.5% stated pre-registration of protocols, and 28.6% assessed the risk of bias in individual studies. For reporting quality, compliance rates were lower than 50.0% for more than half of the items. Less than 15.0% of the IPD-NMAs reported data integrity, presented the network geometry, or clarified risk of bias across studies. IPD-NMAs with statistical or epidemiological authors often better assessed the inconsistency (P = 0.017). IPD-NMAs with a priori protocol were associated with higher reporting quality in terms of search (P = 0.046), data collection process (P = 0.031), and syntheses of results (P = 0.006).

CONCLUSIONS

The reporting of statistical methods and compliance rates of methodological and reporting items of IPD-NMAs were suboptimal. Authors of future IPD-NMAs should address the identified flaws and strictly adhere to methodological and reporting guidelines.

Methodological and reporting quality in non-Cochrane systematic review updates could be improved: a comparative study

OBJECTIVES

The aim of the study was to compare the methodological and reporting quality of updated systematic reviews (SRs) and original SRs.

STUDY DESIGN AND SETTING

We included 30 pairs of non-Cochrane updated and original SRs, identified from a search of PubMed and Embase.com. We used Assessment of Multiple Systematic Reviews-2 (AMSTAR-2) to assess methodological quality and Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) for reporting quality. Stratified analyses were conducted to compare the differences between updated SRs and original SRs and explore factors that might affect the degree of quality change.

RESULTS

Of the 60 non-Cochrane SRs, only two (3.3%) were of low quality, the remaining 58 (96.7%) were of critical low quality. There were no statistically significant differences in methodological quality between the updated SRs and original SRs, although the compliance rates of eight items of updated SRs were higher than that of original SRs. Updated SRs showed an improvement on 15 PRISMA items, but no items with statistically significant differences. The differences in fully reported AMSTAR-2 and PRISMA items between original SRs and updated SRs were also not statistically significant after adjusting for multiple review characteristics.

CONCLUSION

The methodological and reporting quality of updated SRs were not improved compared with original SRs, although the quality could be further improved for both updated SRs and original SRs.

Use of Prediction Intervals in Network Meta-analysis

This cross-sectional study examines the prevalence of reporting prediction intervals in network meta-analysis articles and provides a worked example.

Description of network meta-analysis geometry: A metrics design study

Background

The conduction and report of network meta-analysis (NMA), including the presentation of the network-plot, should be transparent. We aimed to propose metrics adapted from graph theory and social network-analysis literature to numerically describe NMA geometry.

Methods

A previous systematic review of NMAs of pharmacological interventions was performed. Data on the graph’s presentation were collected. Network-plots were reproduced using Gephi 0.9.1. Eleven geometric metrics were tested. The Spearman test for non-parametric correlation analyses and the Bland-Altman and Lin’s Concordance tests were performed (IBM SPSS Statistics 24.0).

Results

From the 477 identified NMAs only 167 graphs could be reproduced because they provided enough information on the plot characteristics. The median nodes and edges were 8 (IQR 6–11) and 10 (IQR 6–16), respectively, with 22 included studies (IQR 13–35). Metrics such as density (median 0.39, ranged 0.07–1.00), median thickness (2.0, IQR 1.0–3.0), percentages of common comparators (median 68%), and strong edges (median 53%) were found to contribute to the description of NMA geometry. Mean thickness, average weighted degree and average path length produced similar results than other metrics, but they can lead to misleading conclusions.

Conclusions

We suggest the incorporation of seven simple metrics to report NMA geometry. Editors and peer-reviews should ensure that guidelines for NMA report are strictly followed before publication.