Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review

To evaluate the performance accuracy and workload savings of artificial intelligence (AI)-based automation tools in comparison with human reviewers in medical literature screening for systematic reviews (SR) of primary studies in cancer research in order to gain insights on improving the efficiency of producing SRs. Medline, Embase, the Cochrane Library, and PROSPERO databases were searched from inception to November 30, 2022. Then, forward and backward literature searches were completed, and the experts in this field including the authors of the articles included were contacted for a thorough grey literature search. This SR was registered on PROSPERO (CRD 42023384772). Among the 3947 studies obtained from search, five studies met the preplanned study selection criteria. These five studies evaluated four AI tools: Abstrackr (four studies), RobotAnalyst (one), EPPI-Reviewer (one), and DistillerSR (one). Without missing final included citations, Abstrackr eliminated 20%-88% of titles and abstracts (time saving of 7-86 hours) and 59% of the full-texts (62 h) from human review across four different cancer-related SRs. In comparison, RobotAnalyst (1% of titles and abstracts, 1 h), EPPI Review (38% of titles and abstracts, 58 h; 59% of full-texts, 62 h), DistillerSR (42% of titles and abstracts, 22 h) also provided similar or lower work savings for single cancer-related SRs. AI-based automation tools exhibited promising but varying levels of accuracy and efficiency during the screening process of medical literature for conducting SRs in the cancer field. Until further progress is made and thorough evaluations are conducted, AI tools should be utilized as supplementary aids rather than complete substitutes for human reviewers.

Compelling evidence from meta-epidemiological studies demonstrates overestimation of effects in randomized trials that fail to optimize randomization and blind patients and outcome assessors

OBJECTIVES

To investigate the impact of potential risk of bias elements on effect estimates in randomized trials.

STUDY DESIGN AND SETTING

We conducted a systematic survey of meta-epidemiological studies examining the influence of potential risk of bias elements on effect estimates in randomized trials. We included only meta-epidemiological studies that either preserved the clustering of trials within meta-analyses (compared effect estimates between trials with and without the potential risk of bias element within each meta-analysis, then combined across meta-analyses; between-trial comparisons), or preserved the clustering of substudies within trials (compared effect estimates between substudies with and without the element, then combined across trials; within-trial comparisons). Separately for studies based on between- and within-trial comparisons, we extracted ratios of odds ratios (RORs) from each study and combined them using a random-effects model. We made overall inferences and assessed certainty of evidence based on Grading of Recommendations, Assessment, development, and Evaluation and Instrument to assess the Credibility of Effect Modification Analyses.

RESULTS

Forty-one meta-epidemiological studies (34 of between-, 7 of within-trial comparisons) proved eligible. Inadequate random sequence generation (ROR 0.94, 95% confidence interval [CI] 0.90-0.97) and allocation concealment (ROR 0.92, 95% CI 0.88-0.97) probably lead to effect overestimation (moderate certainty). Lack of patients blinding probably overestimates effects for patient-reported outcomes (ROR 0.36, 95% CI 0.28-0.48; moderate certainty). Lack of blinding of outcome assessors results in effect overestimation for subjective outcomes (ROR 0.69, 95% CI 0.51-0.93; high certainty). The impact of patients or outcome assessors blinding on other outcomes, and the impact of blinding of health-care providers, data collectors, or data analysts, remain uncertain. Trials stopped early for benefit probably overestimate effects (moderate certainty). Trials with imbalanced cointerventions may overestimate effects, while trials with missing outcome data may underestimate effects (low certainty). Influence of baseline imbalance, compliance, selective reporting, and intention-to-treat analysis remain uncertain.

CONCLUSION

Failure to ensure random sequence generation or adequate allocation concealment probably results in modest overestimates of effects. Lack of patients blinding probably leads to substantial overestimates of effects for patient-reported outcomes. Lack of blinding of outcome assessors results in substantial effect overestimation for subjective outcomes. For other elements, though evidence for consistent systematic overestimate of effect remains limited, failure to implement these safeguards may still introduce important bias.

Presentation approaches for enhancing interpretability of patient-reported outcomes in meta-analyses: a systematic survey of Cochrane reviews

OBJECTIVES

To systematically survey Cochrane reviews' approaches to calculating, presenting, and interpreting pooled estimates of patient-reported outcome measures (PROMs).

STUDY DESIGN AND SETTING

We retrospectively selected 200 Cochrane reviews that met the eligibility criteria. Two researchers independently extracted the pooled effect measures and approaches for pooling and interpreting the effect measures, reaching consensus through discussions.

RESULTS

When primary studies used the same PROM, Cochrane review authors most often used mean differences (MDs) (81.9%) for calculating the pooled effect measures; when primary studies used different PROMs, the review authors often applied standardized mean differences (SMDs) (54.3%). Although in most cases (80.1%) the review authors interpreted the importance of effect, they failed, in 48.5% of the pooled effect measures, to report criteria for categorizing the magnitude of effect. When authors interpreted the importance of the effect, for those with primary studies using the same PROM, they most often referred to the minimally important differences (MIDs) (75.0%); for those with primary studies using different PROMs, the approaches used varied.

CONCLUSION

Cochrane review authors most often used MDs or SMDs for calculating and presenting the pooled effect measures of PROs but often failed to make explicit their criteria for categorizing the magnitude of effect.

Instruments assessing risk of bias of randomized trials frequently included items that are not addressing risk of bias issues

OBJECTIVES

To establish whether items included in instruments published in the last decade assessing risk of bias of randomized controlled trials (RCTs) are indeed addressing risk of bias.

STUDY DESIGN AND SETTING

We searched Medline, Embase, Web of Science, and Scopus from 2010 to October 2021 for instruments assessing risk of bias of RCTs. By extracting items and summarizing their essential content, we generated an item list. Items that two reviewers agreed clearly did not address risk of bias were excluded. We included the remaining items in a survey in which 13 experts judged the issue each item is addressing: risk of bias, applicability, random error, reporting quality, or none of the above.

RESULTS

Seventeen eligible instruments included 127 unique items. After excluding 61 items deemed as clearly not addressing risk of bias, the item classification survey included 66 items, of which the majority of respondents deemed 20 items (30.3%) as addressing risk of bias; the majority deemed 11 (16.7%) as not addressing risk of bias; and there proved substantial disagreement for 35 (53.0%) items.

CONCLUSION

Existing risk of bias instruments frequently include items that do not address risk of bias. For many items, experts disagree on whether or not they are addressing risk of bias.

Methodological approaches for developing, reporting, and assessing evidence-based clinical practice guidelines: A systematic survey

OBJECTIVE

To produce a mapping and feature summary of approaches and tools available for the CPG community to develop, report, or assess four types of CPGs: (1) Standard original (or de novo) CPG, (2) Rapid original CPG, (3) Adapted/adopted CPGs, and (4) Updated CPGs.

STUDY DESIGN

The systematic literature search was conducted using Embase and PubMed, covering the period from January 2010 to October 13 2020. Two websites that collect and recommend approaches/tools to develop, report, or assess CPGs were also searched: Guidelines International Network and Equator Network. We screened the search results to include methodological papers that aimed to develop specific approaches/tools to develop, report, or assess any of the aforementioned four CPG types.

RESULTS

Among 10,581 citations, 46 papers reporting 46 approaches/tools were included. Of these 46 approaches/tools, 33 were about CPG development, seven were for CPG reporting, and six for CPG assessment. Among the 33 development approaches/tools, 26 did not state usability or validity information; but nine from 13 reporting or assessment approaches/tools did.

CONCLUSIONS

This study provides a mapping and feature summary of the current available approaches/tools, which serves to improve users' understanding to pave the way for informed choice and application.

Diversity of bioactive compound content across 71 genera of marine, freshwater, and terrestrial cyanobacteria

Cyanobacterial blooms have increased in frequency, distribution, and intensity due to climate change and anthropogenic nutrient input. The release of bioactive compounds accumulated in these blooms can affect the health of humans and the environment. The co-occurrence of bioactive metabolites is well-documented in bloom samples from marine and freshwater ecosystems, with fewer reports from unialgal isolates. Cyanobacteria also are important terrestrial ecosystem components, especially in drylands, but reports on bioactive molecules from terrestrial cyanobacteria are sparse. This study determined bioactive metabolite profiles for 71 genera of cyanobacteria from seven orders isolated from freshwater (12 genera), marine (15 genera), and terrestrial (44 genera) habitats originally. Cultures were harvested for bioactive metabolites when entering the late-exponential phase for all 157 strains, and 33 were sampled at both early and late exponential phases. Bioactive metabolites were analyzed using an ultra high performance/pressure liquid chromatography in-line with a time-of-flight mass spectrometer. Overall, 12 bioactive classes of the 28 identified were ubiquitous in all samples. On average, each freshwater genus produced ∼12 bioactive classes, whereas each marine genus contained > 4 bioactive classes, and each terrestrial genus contained ∼6 bioactive classes. While 10 of 12 freshwater genera produced at least 10 bioactive classes, only a single genus each from marine and terrestrial habitats had the same number of bioactive classed accumulated. Aeruginosin was found in 58 of 71 total genera, carmabin in 51 of 71 genera, and anabaenopeptin in 48 of 71 genera. Chemotaxonomic use of these secondary metabolites may help resolve higher-level genetic classification(s). An additional growth curve experiment showed that bioactive metabolites were produced at both early and late exponential growth phases. The bioactive metabolite accumulation pattern between early and late exponential phases differed by bioactive classes, genera, and habitats. This survey of 55 bioactive classes in cyanobacteria isolated from freshwater, marine, and terrestrial habitats (71 genera) provides as one of the first systematic bioactive metabolite profiles for cyanobacteria, which should be useful in environmental and drinking water management. Further, it offers novel insights about the toxin potential of selected terrestrial cyanobacteria.

Analysis and reporting of stratified cluster randomized trials-a systematic survey

Background

In order to correctly assess the effect of intervention from stratified cluster randomized trials (CRTs), it is necessary to adjust for both clustering and stratification, as failure to do so leads to misleading conclusions about the intervention effect. We have conducted a systematic survey to examine the current practices about analysis and reporting of stratified CRTs.

Method

We used the search terms to identify the stratified CRTs from MEDLINE since the inception to July 2019. In phase 1, we screened the title and abstract for English-only studies and selected, including the main results paper of the identified protocols, for the next phase. In phase 2, we screened the full text and selected studies for data abstraction. The data abstraction form was piloted and developed using the REDCap. We abstracted data on multiple design and methodological aspects of the study including whether the primary method adjusted for both clustering and stratification, reporting of sample size, randomization, and results.

Results

We screened 2686 studies in the phase 1 and selected 286 studies for phase 2—among them 185 studies were selected for data abstraction. Most of the selected studies were two-arm 140/185 (76%) and parallel-group 165/185 (89%) trials. Among these 185 studies, 27 (15%) of them did not provide any sample size or power calculation, while 105 (57%) studies did not mention any method used for randomization within each stratum. Further, 43 (23%) and 150 (81%) of 185 studies did not provide the definition of all the strata, while more than 60% of the studies did not include all the stratification variable(s) in the flow chart or baseline characteristics table. More than half 114/185 (62%) of the studies did not adjust the primary method for both clustering and stratification.

Conclusion

Stratification helps to achieve the balance among intervention groups. However, to correctly assess the intervention effect from stratified CRTs, it is important to adjust the primary analysis for both stratification and clustering. There are significant deficiencies in the reporting of methodological aspects of stratified CRTs, which require substantial improvements in several areas including definition of strata, inclusion of stratification variable(s) in the flow chart or baseline characteristics table, and reporting the stratum-specific number of clusters and individuals in the intervention groups.

Supplementary information

Supplementary information accompanies this paper at 10.1186/s13063-020-04850-w.

Investigating the impact of design characteristics on statistical efficiency within discrete choice experiments: A systematic survey

Objectives

This study reviews simulation studies of discrete choice experiments to determine (i) how survey design features affect statistical efficiency, (ii) and to appraise their reporting quality.

Outcomes

Statistical efficiency was measured using relative design (D-) efficiency, D-optimality, or D-error.

Methods

For this systematic survey, we searched Journal Storage (JSTOR), Since Direct, PubMed, and OVID which included a search within EMBASE. Searches were conducted up to year 2016 for simulation studies investigating the impact of DCE design features on statistical efficiency. Studies were screened and data were extracted independently and in duplicate. Results for each included study were summarized by design characteristic. Previously developed criteria for reporting quality of simulation studies were also adapted and applied to each included study.

Results

Of 371 potentially relevant studies, 9 were found to be eligible, with several varying in study objectives. Statistical efficiency improved when increasing the number of choice tasks or alternatives; decreasing the number of attributes, attribute levels; using an unrestricted continuous “manipulator” attribute; using model-based approaches with covariates incorporating response behaviour; using sampling approaches that incorporate previous knowledge of response behaviour; incorporating heterogeneity in a model-based design; correctly specifying Bayesian priors; minimizing parameter prior variances; and using an appropriate method to create the DCE design for the research question. The simulation studies performed well in terms of reporting quality. Improvement is needed in regards to clearly specifying study objectives, number of failures, random number generators, starting seeds, and the software used.

Conclusion

These results identify the best approaches to structure a DCE. An investigator can manipulate design characteristics to help reduce response burden and increase statistical efficiency. Since studies varied in their objectives, conclusions were made on several design characteristics, however, the validity of each conclusion was limited. Further research should be conducted to explore all conclusions in various design settings and scenarios. Additional reviews to explore other statistical efficiency outcomes and databases can also be performed to enhance the conclusions identified from this review.