Methodological and reporting quality of systematic reviews referenced in the clinical practice guideline for pediatric high-blood pressure

Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

BACKGROUND

The application of artificial intelligence (AI) in the delivery of health care is a promising area, and guidelines, consensus statements, and standards on AI regarding various topics have been developed.

OBJECTIVE

We performed this study to assess the quality of guidelines, consensus statements, and standards in the field of AI for medicine and to provide a foundation for recommendations about the future development of AI guidelines.

METHODS

We searched 7 electronic databases from database establishment to April 6, 2022, and screened articles involving AI guidelines, consensus statements, and standards for eligibility. The AGREE II (Appraisal of Guidelines for Research & Evaluation II) and RIGHT (Reporting Items for Practice Guidelines in Healthcare) tools were used to assess the methodological and reporting quality of the included articles.

RESULTS

This systematic review included 19 guideline articles, 14 consensus statement articles, and 3 standard articles published between 2019 and 2022. Their content involved disease screening, diagnosis, and treatment; AI intervention trial reporting; AI imaging development and collaboration; AI data application; and AI ethics governance and applications. Our quality assessment revealed that the average overall AGREE II score was 4.0 (range 2.2-5.5; 7-point Likert scale) and the mean overall reporting rate of the RIGHT tool was 49.4% (range 25.7%-77.1%).

CONCLUSIONS

The results indicated important differences in the quality of different AI guidelines, consensus statements, and standards. We made recommendations for improving their methodological and reporting quality.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews (CRD42022321360); https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=321360.

An analysis of the evidence underpinning the national comprehensive cancer network practice guidelines

OBJECTIVE

This study assesses the quality and completeness of systematic reviews (SRs) included by the National Comprehensive Cancer Network (NCCN) cancer screening clinical practice guidelines (CPGs).

METHODS

We evaluated SRs according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess systematic Reviews).

RESULTS

Seven NCCN CPGs were included with 109 SRs. The mean PRISMA percent completeness of included SRs was 71 % (range 0.1-1.0). The mean AMSTAR-2 percent completeness was 56 % (range 0.05-0.99). Of the 70 SRs assessed via AMSTAR-2, 42 (60 %) received a "critically low" rating, 11 (15.7 %) received "low" ratings, and 17 (24.3 %) received "moderate". None of the SRs received a "high" rating.

CONCLUSION

Lack of adherence to AMSTAR-2 and PRISMA reporting standards among the SRs included is prevalent. We suggest improved reporting of SR inclusion criteria and evaluation to bolster the reporting quality of SRs underpinning CPG recommendations.

Evaluation of Reproducible Research Practices in Oncology Systematic Reviews With Meta-analyses Referenced by National Comprehensive Cancer Network Guidelines

IMPORTANCE

Reproducible research practices are essential to biomedical research because these practices promote trustworthy evidence. In systematic reviews and meta-analyses, reproducible research practices ensure that summary effects used to guide patient care are stable and trustworthy.

OBJECTIVE

To evaluate the reproducibility in theory of meta-analyses in oncology systematic reviews cited by the 49 National Comprehensive Cancer Network (NCCN) guidelines for the treatment of cancer by site and evaluate whether Cochrane reviews or systematic reviews that report adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines use more reproducible research practices.

DESIGN, SETTING, AND PARTICIPANTS

A cross-sectional investigation of all systematic reviews with at least 1 meta-analysis and at least 1 included randomized clinical trial (RCT) that are cited by NCCN guidelines for treatment of cancer by site. We scanned the reference list of all NCCN guidelines (n = 49) for potential systematic reviews and meta-analyses. All retrieved studies were screened, and data were extracted, independently and in duplicate. The analysis was carried out between May 6, 2018, and January 28, 2019.

MAIN OUTCOMES AND MEASURES

The frequency of reproducible research practices, defined as (1) effect estimate and measure of precision (eg, hazard ratio with 95% confidence interval); (2) clear list of studies included for each analysis; and (3) for subgroup and sensitivity analyses, it must be clear which studies were included in each group or level.

RESULTS

We identified 1124 potential systematic reviews, and 154 meta-analyses comprising 3696 meta-analytic effect size estimates were included. Only 2375 of the 3696 meta-analytic estimates (64.3%), including subgroup and sensitivity analyses, were reproducible in theory. Forest plots appear to improve the reproducibility of meta-analyses. All meta-analytic estimates were reproducible in theory in 100 systematic reviews (64.9%), and in 15 systematic reviews (9.7%), no meta-analytic estimates could potentially be reproduced. Data were said to be imputed in 29 meta-analyses, but none specified which data. Only 1 meta-analysis included a link to an online data set.

CONCLUSIONS AND RELEVANCE

More reproducible research practices are needed in oncology meta-analyses, as suggested by those that are cited by the NCCN. Reporting meta-analyses in forest plots and requirements for full data sharing are recommended.