Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry

Closing the gap between open source and commercial large language models for medical evidence summarization

Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to the proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance. Utilizing a benchmark dataset, MedReview, consisting of 8161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the performance of open-source models was all improved after fine-tuning. The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were manifested in both a human evaluation and a larger-scale GPT4-simulated evaluation.

Artificial intelligence applied in human health technology assessment: a scoping review protocol

OBJECTIVE

This scoping review aims to map studies that applied artificial intelligence (AI) tools to perform health technology assessment tasks in human health care. The review also aims to understand specific processes in which the AI tools were applied and to comprehend the technical characteristics of these tools.

INTRODUCTION

Health technology assessment is a complex, time-consuming, and labor-intensive endeavor. The development of automation techniques using AI has opened up new avenues for accelerating such assessments in human health settings. This could potentially aid health technology assessment researchers and decision-makers to deliver higher quality evidence.

INCLUSION CRITERIA

This review will consider studies that assesses the use of AI tools in any process of health technology assessment in human health. However, publications in which AI is a means of clinical aid, such as diagnostics or surgery will be excluded.

METHODS

A search for relevant articles will be conducted in databases such as CINAHL (EBSCOhost), Embase (Ovid), MEDLINE (PubMed), Science Direct, Computer and Applied Sciences Complete (EBSCOhost), LILACS, Scopus, and Web of Science Core Collection. A search for gray literature will be conducted in GreyLit.Org, ProQuest Dissertations and Theses, Google Scholar, and the Google search engine. No language filters will be applied. Screening, selection, and data extraction will be performed by 2 independent reviewers. The results will be presented in graphic and tabular format, accompanied by a narrative summary.

DETAILS OF THIS REVIEW CAN BE FOUND IN OPEN SCIENCE FRAMEWORK

osf.io/3rm8g.

Simplifying synthesis of the expanding glioblastoma literature: a topic modeling approach

PURPOSE

Our study aims to discover the leading topics within glioblastoma (GB) research, and to examine if these topics have "hot" or "cold" trends. Additionally, we aim to showcase the potential of natural language processing (NLP) in facilitating research syntheses, offering an efficient strategy to dissect the landscape of academic literature in the realm of GB research.

METHODS

The Scopus database was queried using "glioblastoma" as the search term, in the "TITLE" and "KEY" fields. BERTopic, an NLP-based topic modeling (TM) method, was used for probabilistic TM. We specified a minimum topic size of 300 documents and 5% probability cutoff for outlier detection. We labeled topics based on keywords and representative documents and visualized them with word clouds. Linear regression models were utilized to identify "hot" and "cold" topic trends per decade.

RESULTS

Our TM analysis categorized 43,329 articles into 15 distinct topics. The most common topics were Genomics, Survival, Drug Delivery, and Imaging, while the least common topics were Surgical Resection, MGMT Methylation, and Exosomes. The hottest topics over the 2020s were Viruses and Oncolytic Therapy, Anticancer Compounds, and Exosomes, while the cold topics were Surgical Resection, Angiogenesis, and Tumor Metabolism.

CONCLUSION

Our NLP methodology provided an extensive analysis of GB literature, revealing valuable insights about historical and contemporary patterns difficult to discern with traditional techniques. The outcomes offer guidance for research directions, policy, and identifying emerging trends. Our approach could be applied across research disciplines to summarize and examine scholarly literature, guiding future exploration.

Methodological components, structure and quality assessment tools for evidence summaries: a scoping review

OBJECTIVE

The objective of this review was to identify and map the available information related to the definition, structure, and core methodological components of evidence summaries, as well as to identify any indicators of quality.

INTRODUCTION

Evidence summaries offer a practical solution to overcoming some of the barriers present in evidence-based health care, such as lack of access to evidence at the point of care, and the knowledge and expertise to evaluate the quality and translate the evidence into clinical decision-making. However, lack of transparency in reporting and inconsistencies in the methodology of evidence summary development have previously been cited and pose problems for end-users (eg, clinicians, policymakers).

INCLUSION CRITERIA

Any English-language resource that described the methodological development or appraisal of an evidence summary was included.

METHODS

PubMed, Embase, and CINAHL (EBSCOhost) were systematically searched in November 2019, with no limits on the search. The search was updated in June 2021 and January 2023. Gray literature searches and pearling of references of included sources were also conducted at the same time as the database searches. All resources (ie, articles, papers, books, dissertations, reports, and websites) were eligible for inclusion in the review if they evaluated or described the development or appraisal of an evidence summary methodology within a point-of-care context and were published in English. Literature reviews (eg, systematic reviews, rapid reviews), including summaries of evidence on interventions or health care activities that either measure effects, a phenomena of interest, or where the objective was the development, description or evaluation of methods without a clear point-of-care target, were excluded from the review.

RESULTS

A total of 76 resources (n=56 articles from databases and n=20 reports from gray literature sources) were included in the review. The most common type/name included critically appraised topic (n=18) and evidence summary (n=17). A total of 25 resources provided a definition of an evidence summary: commonalities included a clinical question; a structured, systematic literature search; a description of literature selection; and appraisal of evidence. Of these 25, 16 included descriptors such as brief, concise, rapid, short, succinct and snapshot. The reported methodological components closely reflected the definition results, with the most reported methodological components being a systematic, multi-database search, and critical appraisal. Evidence summary examples were mostly presented as narrative summaries and usually included a reference list, background or clinical context, and recommendations or implications for practice or policy. Four quality assessment tools and a systematic review of tools were included.

CONCLUSIONS

The findings of this study highlight the wide variability in the definition, language, methodological components and structure used for point-of-care resources that met our definition of an evidence summary. This scoping review is one of the first steps aimed at improving the credibility and transparency of evidence summaries in evidence-based health care, with further research required to standardize the definitions and methodologies associated with point-of-care resources and accepted tools for quality assessment.

SUPPLEMENTAL DIGITAL CONTENT

A Chinese-language version of the abstract of this review is available at http://links.lww.com/SRX/A59, studies ineligible following full-text review http://links.lww.com/SRX/A60.

Efficiency and Workload Reduction of Semi-automated Citation Screening Software for Creating Clinical Practice Guidelines: A Prospective Observational Study

BACKGROUND

We evaluated the applicability of automated citation screening in developing clinical practice guidelines.

METHODS

We prospectively compared the efficiency of citation screening between the conventional (Rayyan) and semi-automated (ASReview software) methods. We searched the literature for five clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock. Objective measurements of the time required to complete citation screening were recorded. Following the first screening round, in the primary analysis, the sensitivity, specificity, positive predictive value, and overall screening time were calculated for both procedures using the semi-automated tool as index and the results of the conventional method as standard reference. In the secondary analysis, the same parameters were compared between the two procedures using the final list of included studies after the second screening session as standard reference.

RESULTS

Among the five CQs after the first screening session, the highest and lowest sensitivity, specificity, and positive predictive values were 0.241 and 0.795; 0.991 and 1.000; and 0.482 and 0.929, respectively. In the secondary analysis, the highest sensitivity and specificity in the semi-automated citation screening were 1.000 and 0.997, respectively. The overall screening time per 100 studies was significantly shorter with semi-automated than with conventional citation screening.

CONCLUSION

The potential advantages of the semi-automated method (shorter screening time and higher discriminatory rate for the final list of studies) warrant further validation.

Automation of duplicate record detection for systematic reviews: Deduplicator

BACKGROUND

To describe the algorithm and investigate the efficacy of a novel systematic review automation tool "the Deduplicator" to remove duplicate records from a multi-database systematic review search.

METHODS

We constructed and tested the efficacy of the Deduplicator tool by using 10 previous Cochrane systematic review search results to compare the Deduplicator's 'balanced' algorithm to a semi-manual EndNote method. Two researchers each performed deduplication on the 10 libraries of search results. For five of those libraries, one researcher used the Deduplicator, while the other performed semi-manual deduplication with EndNote. They then switched methods for the remaining five libraries. In addition to this analysis, comparison between the three different Deduplicator algorithms ('balanced', 'focused' and 'relaxed') was performed on two datasets of previously deduplicated search results.

RESULTS

Before deduplication, the mean library size for the 10 systematic reviews was 1962 records. When using the Deduplicator, the mean time to deduplicate was 5 min per 1000 records compared to 15 min with EndNote. The mean error rate with Deduplicator was 1.8 errors per 1000 records in comparison to 3.1 with EndNote. Evaluation of the different Deduplicator algorithms found that the 'balanced' algorithm had the highest mean F1 score of 0.9647. The 'focused' algorithm had the highest mean accuracy of 0.9798 and the highest recall of 0.9757. The 'relaxed' algorithm had the highest mean precision of 0.9896.

CONCLUSIONS

This demonstrates that using the Deduplicator for duplicate record detection reduces the time taken to deduplicate, while maintaining or improving accuracy compared to using a semi-manual EndNote method. However, further research should be performed comparing more deduplication methods to establish relative performance of the Deduplicator against other deduplication methods.

ChatGPT for digital pathology research

The rapid evolution of generative artificial intelligence (AI) models including OpenAI's ChatGPT signals a promising era for medical research. In this Viewpoint, we explore the integration and challenges of large language models (LLMs) in digital pathology, a rapidly evolving domain demanding intricate contextual understanding. The restricted domain-specific efficiency of LLMs necessitates the advent of tailored AI tools, as illustrated by advancements seen in the last few years including FrugalGPT and BioBERT. Our initiative in digital pathology emphasises the potential of domain-specific AI tools, where a curated literature database coupled with a user-interactive web application facilitates precise, referenced information retrieval. Motivated by the success of this initiative, we discuss how domain-specific approaches substantially minimise the risk of inaccurate responses, enhancing the reliability and accuracy of information extraction. We also highlight the broader implications of such tools, particularly in streamlining access to scientific research and democratising access to computational pathology techniques for scientists with little coding experience. This Viewpoint calls for an enhanced integration of domain-specific text-generation AI tools in academic settings to facilitate continuous learning and adaptation to the dynamically evolving landscape of medical research.

Machine learning enables automated screening for systematic reviews and meta-analysis in urology

PURPOSE

To investigate and implement semiautomated screening for meta-analyses (MA) in urology under consideration of class imbalance.

METHODS

Machine learning algorithms were trained on data from three MA with detailed information of the screening process. Different methods to account for class imbalance (Sampling (up- and downsampling, weighting and cost-sensitive learning), thresholding) were implemented in different machine learning (ML) algorithms (Random Forest, Logistic Regression with Elastic Net Regularization, Support Vector Machines). Models were optimized for sensitivity. Besides metrics such as specificity, receiver operating curves, total missed studies, and work saved over sampling were calculated.

RESULTS

During training, models trained after downsampling achieved the best results consistently among all algorithms. Computing time ranged between 251 and 5834 s. However, when evaluated on the final test data set, the weighting approach performed best. In addition, thresholding helped to improve results as compared to the standard of 0.5. However, due to heterogeneity of results no clear recommendation can be made for a universal sample size. Misses of relevant studies were 0 for the optimized models except for one review.

CONCLUSION

It will be necessary to design a holistic methodology that implements the presented methods in a practical manner, but also takes into account other algorithms and the most sophisticated methods for text preprocessing. In addition, the different methods of a cost-sensitive learning approach can be the subject of further investigations.

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Software that employs screening prioritization through active learning (AL) has accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. However, failing to find a relevant paper might alter the findings of a systematic review, highlighting the importance of identifying elusive papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a helpful tool for detecting such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted, mimicking the screening process on a dataset containing titles, abstracts, and labels used for an already published systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the elusive relevant papers. Future research should examine the characteristics of elusive relevant papers to discover why they might take a long time to be found.

Performance of a Large Language Model in Screening Citations

Importance

Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined.

Objective

To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening.

Design, Setting, and Participants

This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024.

Exposures

LLM (GPT-4 Turbo)-assisted citation screening or the conventional method.

Main Outcomes and Measures

The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared.

Results

In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, -15.25 minutes [95% CI, -17.70 to -12.79 minutes]).

Conclusions and Relevance

In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews.

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.

Towards evidence-based practice 2.0: leveraging artificial intelligence in healthcare

Background

Evidence-based practice (EBP) involves making clinical decisions based on three sources of information: evidence, clinical experience and patient preferences. Despite popularization of EBP, research has shown that there are many barriers to achieving the goals of the EBP model. The use of artificial intelligence (AI) in healthcare has been proposed as a means to improve clinical decision-making. The aim of this paper was to pinpoint key challenges pertaining to the three pillars of EBP and to investigate the potential of AI in surmounting these challenges and contributing to a more evidence-based healthcare practice. We conducted a selective review of the literature on EBP and the integration of AI in healthcare to achieve this.

Challenges with the three components of EBP

Clinical decision-making in line with the EBP model presents several challenges. The availability and existence of robust evidence sometimes pose limitations due to slow generation and dissemination processes, as well as the scarcity of high-quality evidence. Direct application of evidence is not always viable because studies often involve patient groups distinct from those encountered in routine healthcare. Clinicians need to rely on their clinical experience to interpret the relevance of evidence and contextualize it within the unique needs of their patients. Moreover, clinical decision-making might be influenced by cognitive and implicit biases. Achieving patient involvement and shared decision-making between clinicians and patients remains challenging in routine healthcare practice due to factors such as low levels of health literacy among patients and their reluctance to actively participate, barriers rooted in clinicians' attitudes, scepticism towards patient knowledge and ineffective communication strategies, busy healthcare environments and limited resources.

AI assistance for the three components of EBP

AI presents a promising solution to address several challenges inherent in the research process, from conducting studies, generating evidence, synthesizing findings, and disseminating crucial information to clinicians to implementing these findings into routine practice. AI systems have a distinct advantage over human clinicians in processing specific types of data and information. The use of AI has shown great promise in areas such as image analysis. AI presents promising avenues to enhance patient engagement by saving time for clinicians and has the potential to increase patient autonomy although there is a lack of research on this issue.

Conclusion

This review underscores AI's potential to augment evidence-based healthcare practices, potentially marking the emergence of EBP 2.0. However, there are also uncertainties regarding how AI will contribute to a more evidence-based healthcare. Hence, empirical research is essential to validate and substantiate various aspects of AI use in healthcare.

Trends in stroke-related journals: Examination of publication patterns using topic modeling

OBJECTIVES

This study aims to demonstrate the capacity of natural language processing and topic modeling to manage and interpret the vast quantities of scholarly publications in the landscape of stroke research. These tools can expedite the literature review process, reveal hidden themes, and track rising research areas.

MATERIALS AND METHODS

Our study involved reviewing and analyzing articles published in five prestigious stroke journals, namely Stroke, International Journal of Stroke, European Stroke Journal, Translational Stroke Research, and Journal of Stroke and Cerebrovascular Diseases. The team extracted document titles, abstracts, publication years, and citation counts from the Scopus database. BERTopic was chosen as the topic modeling technique. Using linear regression models, current stroke research trends were identified. Python 3.1 was used to analyze and visualize data.

RESULTS

Out of the 35,779 documents collected, 26,732 were classified into 30 categories and used for analysis. "Animal Models," "Rehabilitation," and "Reperfusion Therapy" were identified as the three most prevalent topics. Linear regression models identified "Emboli," "Medullary and Cerebellar Infarcts," and "Glucose Metabolism" as trending topics, whereas "Cerebral Venous Thrombosis," "Statins," and "Intracerebral Hemorrhage" demonstrated a weaker trend.

CONCLUSIONS

The methodology can assist researchers, funders, and publishers by documenting the evolution and specialization of topics. The findings illustrate the significance of animal models, the expansion of rehabilitation research, and the centrality of reperfusion therapy. Limitations include a five-journal cap and a reliance on high-quality metadata.

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

The Power of Rapid Reviews for Bridging the Knowledge-to-Action Gap in Evidence-Based Virtual Health Care

Despite the surge in popularity of virtual health care services as a means of delivering health care through technology, the integration of research evidence into practice remains a challenge. Rapid reviews, a type of time-efficient evidence synthesis, offer a potential solution to bridge the gap between knowledge and action. This paper aims to highlight the experiences of the Fraser Health Authority's Virtual Health team in conducting rapid reviews. This paper discusses the experiences of the Virtual Health team in conducting 15 rapid reviews over the course of 1.5 years and the benefit of involving diverse stakeholders including researchers, project and clinical leads, and students for the creation of user-friendly knowledge products to summarize results. The Virtual Health team found rapid reviews to be a valuable tool for evidence-informed decision-making in virtual health care. Involving stakeholders and focusing on implementation considerations are crucial for maximizing the impact of rapid reviews. Health care decision makers are encouraged to consider implementing rapid review processes to improve the translation of research evidence into practice, ultimately enhancing patient outcomes and promoting a culture of evidence-informed care.

Mapping the Degenerative Cervical Myelopathy Research Landscape: Topic Modeling of the Literature

STUDY DESIGN

Topic modeling of literature.

OBJECTIVES

Our study has 2 goals: (i) to clarify key themes in degenerative cervical myelopathy (DCM) research, and (ii) to evaluate the current trends in the popularity or decline of these topics. Additionally, we aim to highlight the potential of natural language processing (NLP) in facilitating research syntheses.

METHODS

Documents were retrieved from Scopus, preprocessed, and modeled using BERTopic, an NLP-based topic modeling method. We specified a minimum topic size of 25 documents and 50 words per topic. After the models were trained, they generated a list of topics and corresponding representative documents. We utilized linear regression models to examine trends within the identified topics. In this context, topics exhibiting increasing linear slopes were categorized as "hot topics," while those with decreasing slopes were categorized as "cold topics".

RESULTS

Our analysis retrieved 3510 documents that were classified into 21 different topics. The 3 most frequently occurring topics were "OPLL" (ossification of the posterior longitudinal ligament), "Anterior Fusion," and "Surgical Outcomes." Trend analysis revealed the hottest topics of the decade to be "Animal Models," "DCM in the Elderly," and "Posterior Decompression" while "Morphometric Analyses," "Questionnaires," and "MEP and SSEP" were identified as being the coldest topics.

CONCLUSIONS

Our NLP methodology conducted a thorough and detailed analysis of DCM research, uncovering valuable insights into research trends that were otherwise difficult to discern using traditional techniques. The results provide valuable guidance for future research directions, policy considerations, and identification of emerging trends.

Building a sharable literature collection to advance the science and practice of implementation facilitation

Background

Implementation science seeks to produce generalizable knowledge on strategies that promote the adoption and sustained use of evidence-based innovations. Literature reviews on specific implementation strategies can help us understand how they are conceptualized and applied, synthesize findings, and identify knowledge gaps. Although rigorous literature reviews can advance scientific knowledge and facilitate theory development, they are time-consuming and costly to produce. Improving the efficiency of literature review processes and reducing redundancy of effort is especially important for this rapidly developing field. We sought to amass relevant literature on one increasingly used evidence-based strategy, implementation facilitation (IF), as a publicly available resource.

Methods

We conducted a rigorous systematic search of PubMed, CINAHL, and Web of Science citation databases for peer-reviewed, English-language articles with "facilitation" and a combination of other terms published from January 1996 to December 2021. We searched bibliographies of articles published from 1996 to 2015 and identified articles during the full text review that reported on the same study. Two authors screened 3,168 abstracts. After establishing inter-rater reliability, they individually conducted full-text review of 786 relevant articles. A multidisciplinary team of investigators provided recommendations for preparing and disseminating the literature collection.

Findings

The literature collection is comprised of 510 articles. It includes 277 empirical studies of IF and 77 other articles, including conceptual/theoretical articles, literature reviews, debate papers and descriptions of large-scale clinical initiatives. Over half of the articles were published between 2017 and 2021. The collection is publicly available as an Excel file and as an xml file that can be imported into reference management software.

Conclusion

We created a publicly accessible collection of literature about the application of IF to implement evidence-based innovations in healthcare. The comprehensiveness of this collection has the potential to maximize efficiency and minimize redundancy in scientific inquiry about this strategy. Scientists and practitioners can use the collection to more rapidly identify developments in the application of IF and to investigate a wide range of compelling questions on its use within and across different healthcare disciplines/settings, countries, and payer systems. We offer several examples of how this collection has already been used.

Evidence syntheses to support decision-making related to the Covid-19 pandemic

The COVID-19 pandemic generated a large volume of scientific productions with different quality levels. The speed with which knowledge was produced and shared worldwide imposed on health management the challenge of seeking ways to identify the best available evidence to support its decisions. In response to this challenge, the Department of Science and Technology of the Brazilian Ministry of Health started offering a service to produce and provide scientific knowledge addressing priority public health issues in the pandemic scenario. Drug treatments, non-pharmacological measures, testing, reinfection and immunological response, immunization, pathophysiology, post-COVID syndrome and adverse events are among the topics covered. In this article, we discuss the strengths and lessons learned, as well as the challenges and perspectives that present a real example of how to offer the best scientific evidence in a timely manner in order to assist the decision-making process during a public health emergency.

Comparative analysis of Cochrane and non-Cochrane reviews over three decades

BACKGROUND

Systematic reviews are viewed as the best study design to guide clinical decision-making as they are the least biased publications assuming they are well-conducted and include well-designed studies. Cochrane was initiated in 1993 with an aim of conducting high-quality systematic reviews. We aimed to examine the publication rates of non-Cochrane systematic reviews (henceforth referred to simply as "systematic reviews") and Cochrane reviews produced throughout Cochrane's existence and characterize changes throughout the period.

METHODS

This observational study collected data on systematic reviews published between 1993 and 2022 in PubMed. Identified Cochrane reviews were linked to data from the Cochrane Database of Systematic Reviews via their Digital Object Identifier. Systematic reviews and Cochrane reviews were analyzed separately. Two authors screened a random sample of records to validate the overall sample, providing a precision of 98%.

RESULTS

We identified 231,602 (94%) systematic reviews and 15,038 (6%) Cochrane reviews. Publication of systematic reviews has continuously increased with a median yearly increase rate of 26%, while publication of Cochrane reviews has decreased since 2015. From 1993 to 2002, Cochrane reviews constituted 35% of all systematic reviews in PubMed compared with 3.5% in 2013-2022. Systematic reviews consistently had fewer authors than Cochrane reviews, but the number of authors increased over time for both. Chinese first authors conducted 15% and 4% of systematic reviews published from 2013-2022 and 2003-2012, respectively. Most Cochrane reviews had first authors from the UK (36%). The native English-speaking countries the USA, the UK, Canada, and Australia produced a large share of systematic reviews (42%) and Cochrane reviews (62%). The largest publishers of systematic reviews in the last 10 years were gold open access journals.

CONCLUSIONS

Publication of systematic reviews is increasing rapidly, while fewer Cochrane reviews have been published through the last decade. Native English-speaking countries produced a large proportion of both types of systematic reviews. Gold open access journals and Chinese first authors dominated the publication of systematic reviews for the past 10 years. More research is warranted examining why fewer Cochrane reviews are being published. Additionally, examining these systematic reviews for research waste metrics may provide a clearer picture of their utility.

A systematic review of animal and human data comparing the nasal potential difference test between cystic fibrosis and control

The nasal potential difference test (nPD) is an electrophysiological measurement which is altered in patients and animal models with cystic fibrosis (CF). Because protocols and outcomes vary substantially between laboratories, there are concerns over its validity and precision. We performed a systematic literature review (SR) of the nPD to answer the following review questions: A. Is the nasal potential difference similarly affected in CF patients and animal models?", and B. "Is the nPD in human patients and animal models of CF similarly affected by various changes in the experimental set-up?". The review protocol was preregistered on PROSPERO (CRD42021236047). We searched PubMed and Embase with comprehensive search strings. Two independent reviewers screened all references for inclusion and extracted all data. Included were studies about CF which described in vivo nPD measurements in separate CF and control groups. Risk of bias was assessed, and three meta-analyses were performed. We included 130 references describing nPD values for CF and control subjects, which confirmed substantial variation in the experimental design and nPD outcome between groups. The meta-analyses showed a clear difference in baseline nPD values between CF and control subjects, both in animals and in humans. However, baseline nPD values were, on average, lower in animal than in human studies. Reporting of experimental details was poor for both animal and human studies, and urgently needs to improve to ensure reproducibility of experiments within and between species.

A span-based model for extracting overlapping PICO entities from randomized controlled trial publications

OBJECTIVES

Extracting PICO (Populations, Interventions, Comparison, and Outcomes) entities is fundamental to evidence retrieval. We present a novel method, PICOX, to extract overlapping PICO entities.

MATERIALS AND METHODS

PICOX first identifies entities by assessing whether a word marks the beginning or conclusion of an entity. Then, it uses a multi-label classifier to assign one or more PICO labels to a span candidate. PICOX was evaluated using 1 of the best-performing baselines, EBM-NLP, and 3 more datasets, ie, PICO-Corpus and randomized controlled trial publications on Alzheimer's Disease (AD) or COVID-19, using entity-level precision, recall, and F1 scores.

RESULTS

PICOX achieved superior precision, recall, and F1 scores across the board, with the micro F1 score improving from 45.05 to 50.87 (P ≪.01). On the PICO-Corpus, PICOX obtained higher recall and F1 scores than the baseline and improved the micro recall score from 56.66 to 67.33. On the COVID-19 dataset, PICOX also outperformed the baseline and improved the micro F1 score from 77.10 to 80.32. On the AD dataset, PICOX demonstrated comparable F1 scores with higher precision when compared to the baseline.

CONCLUSION

PICOX excels in identifying overlapping entities and consistently surpasses a leading baseline across multiple datasets. Ablation studies reveal that its data augmentation strategy effectively minimizes false positives and improves precision.

From Text to Insight: A Natural Language Processing-Based Analysis of Topics and Trends in Neurosurgery

BACKGROUND AND OBJECTIVES

Neurosurgical research is a rapidly evolving field, with new research topics emerging continually. To provide a clearer understanding of the evolving research landscape, our study aimed to identify and analyze the prevalent research topics and trends in Neurosurgery.

METHODS

We used BERTopic, an advanced natural language processing-based topic modeling approach, to analyze papers published in the journal Neurosurgery . Using this method, topics were identified based on unique sets of keywords that encapsulated the core themes of each article. Linear regression models were then trained on the topic probabilities to identify trends over time, allowing us to identify "hot" (growing in prominence) and "cold" (decreasing in prominence) topics. We also performed a focused analysis of the trends in the current decade.

RESULTS

Our analysis led to the categorization of 12 438 documents into 49 distinct topics. The topics covered a wide range of themes, with the most commonly identified topics being "Spinal Neurosurgery" and "Treatment of Cerebral Ischemia." The hottest topics of the current decade were "Peripheral Nerve Surgery," "Unruptured Aneurysms," and "Endovascular Treatments" while the cold topics were "Chiari Malformations," "Thromboembolism Prophylaxis," and "Infections."

CONCLUSION

Our study underscores the dynamic nature of neurosurgical research and the evolving focus of the field. The insights derived from the analysis can guide future research directions, inform policy decisions, and identify emerging areas of interest. The use of natural language processing in synthesizing and analyzing large volumes of academic literature demonstrates the potential of advanced analytical techniques in understanding the research landscape, paving the way for similar analyses across other medical disciplines.

Exploiting Natural Language Processing to Unveil Topics and Trends of Traumatic Brain Injury Research

Traumatic brain injury (TBI) has evolved from a topic of relative obscurity to one of widespread scientific and lay interest. The scope and focus of TBI research have shifted, and research trends have changed in response to public and scientific interest. This study has two primary goals: first, to identify the predominant themes in TBI research; and second, to delineate "hot" and "cold" areas of interest by evaluating the current popularity or decline of these topics. Hot topics may be dwarfed in absolute numbers by other, larger TBI research areas but are rapidly gaining interest. Likewise, cold topics may present opportunities for researchers to revisit unanswered questions. We utilized BERTopic, an advanced natural language processing (NLP)-based technique, to analyze TBI research articles published since 1990. This approach facilitated the identification of key topics by extracting sets of distinctive keywords representative of each article's core themes. Using these topics' probabilities, we trained linear regression models to detect trends over time, recognizing topics that were gaining (hot) or losing (cold) relevance. Additionally, we conducted a specific analysis focusing on the trends observed in TBI research in the current decade (the 2020s). Our topic modeling analysis categorized 42,422 articles into 27 distinct topics. The 10 most frequently occurring topics were: "Rehabilitation," "Molecular Mechanisms of TBI," "Concussion," "Repetitive Head Impacts," "Surgical Interventions," "Biomarkers," "Intracranial Pressure," "Posttraumatic Neurodegeneration," "Chronic Traumatic Encephalopathy," and "Blast Induced TBI," while our trend analysis indicated that the hottest topics of the current decade were "Genomics," "Sex Hormones," and "Diffusion Tensor Imaging," while the cooling topics were "Posttraumatic Sleep," "Sensory Functions," and "Hyperosmolar Therapies." This study highlights the dynamic nature of TBI research and underscores the shifting emphasis within the field. The findings from our analysis can aid in the identification of emerging topics of interest and areas where there is little new research reported. By utilizing NLP to effectively synthesize and analyze an extensive collection of TBI-related scholarly literature, we demonstrate the potential of machine learning techniques in understanding and guiding future research prospects. This approach sets the stage for similar analyses in other medical disciplines, offering profound insights and opportunities for further exploration.

Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models

BACKGROUND

The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results.

METHODS

We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis.

RESULTS

The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results.

CONCLUSIONS

This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.

Cocreating an Automated mHealth Apps Systematic Review Process With Generative AI: Design Science Research Approach

BACKGROUND

The use of mobile devices for delivering health-related services (mobile health [mHealth]) has rapidly increased, leading to a demand for summarizing the state of the art and practice through systematic reviews. However, the systematic review process is a resource-intensive and time-consuming process. Generative artificial intelligence (AI) has emerged as a potential solution to automate tedious tasks.

OBJECTIVE

This study aimed to explore the feasibility of using generative AI tools to automate time-consuming and resource-intensive tasks in a systematic review process and assess the scope and limitations of using such tools.

METHODS

We used the design science research methodology. The solution proposed is to use cocreation with a generative AI, such as ChatGPT, to produce software code that automates the process of conducting systematic reviews.

RESULTS

A triggering prompt was generated, and assistance from the generative AI was used to guide the steps toward developing, executing, and debugging a Python script. Errors in code were solved through conversational exchange with ChatGPT, and a tentative script was created. The code pulled the mHealth solutions from the Google Play Store and searched their descriptions for keywords that hinted toward evidence base. The results were exported to a CSV file, which was compared to the initial outputs of other similar systematic review processes.

CONCLUSIONS

This study demonstrates the potential of using generative AI to automate the time-consuming process of conducting systematic reviews of mHealth apps. This approach could be particularly useful for researchers with limited coding skills. However, the study has limitations related to the design science research methodology, subjectivity bias, and the quality of the search results used to train the language model.

Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review

To evaluate the performance accuracy and workload savings of artificial intelligence (AI)-based automation tools in comparison with human reviewers in medical literature screening for systematic reviews (SR) of primary studies in cancer research in order to gain insights on improving the efficiency of producing SRs. Medline, Embase, the Cochrane Library, and PROSPERO databases were searched from inception to November 30, 2022. Then, forward and backward literature searches were completed, and the experts in this field including the authors of the articles included were contacted for a thorough grey literature search. This SR was registered on PROSPERO (CRD 42023384772). Among the 3947 studies obtained from search, five studies met the preplanned study selection criteria. These five studies evaluated four AI tools: Abstrackr (four studies), RobotAnalyst (one), EPPI-Reviewer (one), and DistillerSR (one). Without missing final included citations, Abstrackr eliminated 20%-88% of titles and abstracts (time saving of 7-86 hours) and 59% of the full-texts (62 h) from human review across four different cancer-related SRs. In comparison, RobotAnalyst (1% of titles and abstracts, 1 h), EPPI Review (38% of titles and abstracts, 58 h; 59% of full-texts, 62 h), DistillerSR (42% of titles and abstracts, 22 h) also provided similar or lower work savings for single cancer-related SRs. AI-based automation tools exhibited promising but varying levels of accuracy and efficiency during the screening process of medical literature for conducting SRs in the cancer field. Until further progress is made and thorough evaluations are conducted, AI tools should be utilized as supplementary aids rather than complete substitutes for human reviewers.

Navigating PROSPERO4animals: 10 top tips for efficient pre-registration of your animal systematic review protocol

Systematic reviews are an essential tool in identifying knowledge gaps and synthesizing evidence from in vivo animal research to improve human health. The review process follows an explicit and systematic methodology to minimize bias, but is not immune to biases or methodological flaws. Pre-registering a systematic review protocol has several benefits, including avoiding unplanned duplication of reviews, reducing reporting biases, and providing structure throughout the review process. It also helps to align the opinions of review team members and can shield researchers from post-hoc critique. PROSPERO4animals is the international prospective register of systematic reviews (PROSPERO) for the preregistration of systematic review of animal studies. As administrators, here we provide 10 tips to facilitate pre-registration in PROSPERO4animals. These tips address common difficulties that both beginners and experienced researchers may face when pre-registering their systematic review protocols. This article aims to help authors write and register a detailed systematic review protocol on PROSPERO4animals.

Systematic review using a spiral approach with machine learning

With the accelerating growth of the academic corpus, doubling every 9 years, machine learning is a promising avenue to make systematic review manageable. Though several notable advancements have already been made, the incorporation of machine learning is less than optimal, still relying on a sequential, staged process designed to accommodate a purely human approach, exemplified by PRISMA. Here, we test a spiral, alternating or oscillating approach, where full-text screening is done intermittently with title/abstract screening, which we examine in three datasets by simulation under 360 conditions comprised of different algorithmic classifiers, feature extractions, prioritization rules, data types, and information provided (e.g., title/abstract, full-text included). Overwhelmingly, the results favored a spiral processing approach with logistic regression, TF-IDF for vectorization, and maximum probability for prioritization. Results demonstrate up to a 90% improvement over traditional machine learning methodologies, especially for databases with fewer eligible articles. With these advancements, the screening component of most systematic reviews should remain functionally achievable for another one to two decades.

Learning health systems and evidence ecosystems: a perspective on the future of evidence-based medicine and evidence-based guideline development

Despite forming the cornerstone of modern clinical practice for decades, implementation of evidence-based medicine at scale remains a crucial challenge for health systems. As a result, there has been a growing need for conceptual models to better contextualise and pragmatize the use of evidence-based medicine, particularly in tandem with patient-centred care. In this commentary, we highlight the emergence of the learning health system as one such model and analyse its potential role in pragmatizing both evidence-based medicine and patient-centred care. We apply the learning health system lens to contextualise the key activity of evidence-based guideline development and implementation, and highlight how current inefficiencies and bottlenecks in the evidence synthesis phase of evidence-based guideline development threaten downstream adherence. Lastly, we introduce the evidence ecosystem as a complementary model to learning health systems, and propose how innovative developments from the evidence ecosystem may be integrated with learning health systems to better enable health impact at speed and scale.

Catchii: Empowering literature review screening in healthcare

A systematic review is a type of literature review that aims to collect and analyse all available evidence from the literature on a particular topic. The process of screening and identifying eligible articles from the vast amounts of literature is a time-consuming task. Specialised software has been developed to aid in the screening process and save significant time and labour. However, the most suitable software tools that are available often come with a cost or only offer either a limited or a trial version for free. In this paper, we report the release of a new software application, Catchii, which contains all the important features of a systematic review screening application while being completely free. It supports a user at different stages of screening, from detecting duplicates to creating the final flowchart for a publication. Catchii is designed to provide a good user experience and streamline the screening process through its clean and user-friendly interface on both computers and mobile devices. All in all, Catchii is a valuable addition to the current selection of systematic review screening applications. It enables researchers without financial resources to access features found in the best paid tools, while also diminishing costs for those who have previously relied on paid applications. Catchii is available at https://catchii.org.

The 2-week systematic review (2weekSR) method was successfully blind-replicated by another team: a case study

OBJECTIVE

To assess the replicability of a 2-week systematic review (index 2weekSR) created with the assistance of automation tools using the fidelity method.

METHODS

A Preferred Reporting Items for Systematic reviews and Meta-Analyses compliant SR protocol was developed based on the published information of the index 2weekSR study. The replication team consisted of three reviewers. Two reviewers blocked off time during the replication. The total time to complete tasks and the meta-analysis results were compared with the index 2weekSR study. Review process fidelity scores (FSs) were calculated for review methods and outcomes. Barriers to completing the replication were identified.

RESULTS

The review was completed over 63 person-hours (11 workdays/15 calendar days). A FS of 0.95 was achieved for the methods, with 3 (of 8) tasks only partially replicated, and an FS of 0.63 for the outcomes, with 6 (of 7) only partially replicated and one task was not replicated. Nonreplication was mainly caused by missing information in the index 2weekSR study that was not required in standard reporting guidelines. The replication arrived at the same conclusions as the original study.

CONCLUSION

A 2weekSR study was replicated by a small team of three reviewers supported by automation tools. Including additional information when reporting SRs should improve their replicability.

Evidence pertaining to modifiable risk factors for oral diseases: an umbrella review to Inform oral health messages for Australia

The aim of this umbrella review was to collate and appraise the evidence base regarding modifiable risk factors for the prevention of oral diseases to inform the update of the Oral Health Messages for Australia. Eleven questions related to modifiable risk factors and dental disease were investigated. Electronic databases (Medline, Embase and PubMed) were searched from January 2010 to October 2022. Systematic reviews evaluating interventions/exposures in healthy subjects from high-income countries, where Westernized practices, oral health promotion and healthcare systems are similar to Australia, were included. Quality appraisal of included systematic reviews was guided by the AMSTAR tool. Of the 3637 articles identified, 29 articles met eligibility criteria. High-quality systematic reviews were identified for questions relating to diet, infant feeding, dental check-ups and oral hygiene. Free sugars consumption above 5% of energy intake, infrequent toothbrushing, smoking/vaping and alcohol intake were consistently associated with poorer oral health outcomes. Breastfeeding up to the age of 24 months was not associated with an increased risk of early childhood caries. The use of interdental cleaning devices and mouthguards during contact sports are likely to be effective in preventing dental disease.

Title-plus-abstract versus title-only first-level screening approach: a case study using a systematic review of dietary patterns and sarcopenia risk to compare screening performance

BACKGROUND

Conducting a systematic review is a time- and resource-intensive multi-step process. Enhancing efficiency without sacrificing accuracy and rigor during the screening phase of a systematic review is of interest among the scientific community.

METHODS

This case study compares the screening performance of a title-only (Ti/O) screening approach to the more conventional title-plus-abstract (Ti + Ab) screening approach. Both Ti/O and Ti + Ab screening approaches were performed simultaneously during first-level screening of a systematic review investigating the relationship between dietary patterns and risk factors and incidence of sarcopenia. The qualitative and quantitative performance of each screening approach was compared against the final results of studies included in the systematic review, published elsewhere, which used the standard Ti + Ab approach. A statistical analysis was conducted, and contingency tables were used to compare each screening approach in terms of false inclusions and false exclusions and subsequent sensitivity, specificity, accuracy, and positive predictive power.

RESULTS

Thirty-eight citations were included in the final analysis, published elsewhere. The current case study found that the Ti/O first-level screening approach correctly identified 22 citations and falsely excluded 16 citations, most often due to titles lacking a clear indicator of study design or outcomes relevant to the systematic review eligibility criteria. The Ti + Ab approach correctly identified 36 citations and falsely excluded 2 citations due to limited population and intervention descriptions in the abstract. Our analysis revealed that the performance of the Ti + Ab first-level screening was statistically different compared to the average performance of both approaches (Chi-squared: 5.21, p value 0.0225) while the Ti/O approach was not (chi-squared: 2.92, p value 0.0874). The predictive power of the first-level screening was 14.3% and 25.5% for the Ti/O and Ti + Ab approaches, respectively. In terms of sensitivity, 57.9% of studies were correctly identified at the first-level screening stage using the Ti/O approach versus 94.7% by the Ti + Ab approach.

CONCLUSIONS

In the current case study comparing two screening approaches, the Ti + Ab screening approach captured more relevant studies compared to the Ti/O approach by including a higher number of accurately eligible citations. Ti/O screening may increase the likelihood of missing evidence leading to evidence selection bias.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO Protocol Number: CRD42020172655.

A mapping exercise using automated techniques to develop a search strategy to identify systematic review tools

The Systematic Review Toolbox aims provide a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. Identifying publications surrounding specific systematic review tools is currently challenging, leading to a high screening burden for few eligible records. We aimed to develop a search strategy that could be regularly and automatically run to identify eligible records for the SR Toolbox, thus reducing time on task and burden for those involved. We undertook a mapping exercise to identify the PubMed IDs of papers indexed within the SR Toolbox. We then used the Yale MeSH Analyser and Visualisation of Similarities (VOS) Viewer text-mining software to identify the most commonly used MeSH terms and text words within the eligible records. These MeSH terms and text words were combined using Boolean Operators into a search strategy for Ovid MEDLINE. Prior to the mapping exercise and search strategy development, 81 software tools and 55 'Other' tools were included within the SR Toolbox. Since implementation of the search strategy, 146 tools have been added. There has been an increase in tools added to the toolbox since the search was developed and its corresponding auto-alert in MEDLINE was originally set up. Developing a search strategy based on a mapping exercise is an effective way of identifying new tools to support the systematic review process. Further research could be conducted to help prioritise records for screening to reduce reviewer burden further and to adapt the strategy for disciplines beyond healthcare.

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project

Background  Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods  Researchers manually annotated 70 items of information ('entities') in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the 'FLAIR' framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results  The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator's score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms ( e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions  While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper 3).

Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews

BACKGROUND

Evidence-based medicine requires synthesis of research through rigorous and time-intensive systematic literature reviews (SLRs), with significant resource expenditure for data extraction from scientific publications. Machine learning may enable the timely completion of SLRs and reduce errors by automating data identification and extraction.

METHODS

We evaluated the use of machine learning to extract data from publications related to SLRs in oncology (SLR 1) and Fabry disease (SLR 2). SLR 1 predominantly contained interventional studies and SLR 2 observational studies. Predefined key terms and data were manually annotated to train and test bidirectional encoder representations from transformers (BERT) and bidirectional long-short-term memory machine learning models. Using human annotation as a reference, we assessed the ability of the models to identify biomedical terms of interest (entities) and their relations. We also pretrained BERT on a corpus of 100,000 open access clinical publications and/or enhanced context-dependent entity classification with a conditional random field (CRF) model. Performance was measured using the F1 score, a metric that combines precision and recall. We defined successful matches as partial overlap of entities of the same type.

RESULTS

For entity recognition, the pretrained BERT+CRF model had the best performance, with an F1 score of 73% in SLR 1 and 70% in SLR 2. Entity types identified with the highest accuracy were metrics for progression-free survival (SLR 1, F1 score 88%) or for patient age (SLR 2, F1 score 82%). Treatment arm dosage was identified less successfully (F1 scores 60% [SLR 1] and 49% [SLR 2]). The best-performing model for relation extraction, pretrained BERT relation classification, exhibited F1 scores higher than 90% in cases with at least 80 relation examples for a pair of related entity types.

CONCLUSIONS

The performance of BERT is enhanced by pretraining with biomedical literature and by combining with a CRF model. With refinement, machine learning may assist with manual data extraction for SLRs.

Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations

Evidence-based medicine, the practice in which healthcare professionals refer to the best available evidence when making decisions, forms the foundation of modern healthcare. However, it relies on labour-intensive systematic reviews, where domain specialists must aggregate and extract information from thousands of publications, primarily of randomised controlled trial (RCT) results, into evidence tables. This paper investigates automating evidence table generation by decomposing the problem across two language processing tasks: named entity recognition, which identifies key entities within text, such as drug names, and relation extraction, which maps their relationships for separating them into ordered tuples. We focus on the automatic tabulation of sentences from published RCT abstracts that report the results of the study outcomes. Two deep neural net models were developed as part of a joint extraction pipeline, using the principles of transfer learning and transformer-based language representations. To train and test these models, a new gold-standard corpus was developed, comprising over 550 result sentences from six disease areas. This approach demonstrated significant advantages, with our system performing well across multiple natural language processing tasks and disease areas, as well as in generalising to disease domains unseen during training. Furthermore, we show these results were achievable through training our models on as few as 170 example sentences. The final system is a proof of concept that the generation of evidence tables can be semi-automated, representing a step towards fully automating systematic reviews.

Framework for a living systematic review and meta-analysis for the surgical treatment of bladder cancer: introducing EVIglance to urology

Background

Knowledge of current and ongoing studies is critical for identifying research gaps and enabling evidence-based decisions for individualized treatment. However, the increasing number of scientific publications poses challenges for healthcare providers and patients in all medical fields to stay updated with the latest evidence. To overcome these barriers, we aim to develop a living systematic review and open-access online evidence map of surgical therapy for bladder cancer (BC), including meta-analyses.

Methods

Following the guidelines provided in the Cochrane Handbook for Systematic Reviews of Interventions and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement, a systematic literature search on uro-oncological therapy in BC will be performed across various literature databases. Within the scope of a meta-analysis and living systematic review, relevant randomized controlled trials will be identified. Data extraction and quantitative analysis will be conducted, along with a critical appraisal of the quality and risk of bias of each study. The available research evidence will be entered into an open-access framework (www.evidencemap.surgery) and will also be accessible via the EVIglance app. Regular semi-automatic updates will enable the implementation of a real-living review concept and facilitate resource-efficient screening.

Discussion

A regularly updated evidence map provides professionals and patients with an open-access knowledge base on the current state of research, allowing for decision-making based on recent evidence. It will help identify an oversupply of evidence, thus avoiding redundant work. Furthermore, by identifying research gaps, new hypotheses can be formulated more precisely, enabling planning, determination of sample size, and definition of endpoints for future trials.

A Systematic Review of the Effect of Cystic Fibrosis Treatments on the Nasal Potential Difference Test in Animals and Humans

To address unmet treatment needs in cystic fibrosis (CF), preclinical and clinical studies are warranted. Because it directly reflects the function of the Cystic Fibrosis Transmembrane conductance Regulator (CFTR), the nasal potential difference test (nPD) can not only be used as a reliable diagnostic test for CF but also to assess efficacy of experimental treatments. We performed a full comprehensive systematic review of the effect of CF treatments on the nPD compared to control conditions tested in separate groups of animal and human subjects. Our review followed a preregistered protocol. We included 34 references: 20 describing mouse studies, 12 describing human studies, and 2 describing both. We provide a comprehensive list of these studies, which assessed the effects of antibiotics, bone marrow transplant, CFTR protein, CFTR RNA, directly and indirectly CFTR-targeting drugs, non-viral and viral gene transfer, and other treatments. Our results support the nPD representing a reliable method for testing treatment effects in both animal models and human patients, as well as for diagnosing CF. However, we also observed the need for improved reporting to ensure reproducibility of the experiments and quantitative comparability of the results within and between species (e.g., with meta-analyses). Currently, data gaps warrant further primary studies.

How to Conduct a Systematic Review and Meta-Analysis: A Guide for Clinicians

Evidence-based practice relies on using research evidence to guide clinical decision-making. However, staying current with all published research can be challenging. Many clinicians use review articles that apply predefined methods to locate, identify, and summarize all available evidence on a topic to guide clinical decision-making. This paper discusses the role of review articles, including narrative, scoping, and systematic reviews, to synthesize existing evidence and generate new knowledge. It provides a step-by-step guide to conducting a systematic review and meta-analysis, covering key steps such as formulating a research question, selecting studies, evaluating evidence quality, and reporting results. This paper is intended as a resource for clinicians looking to learn how to conduct systematic reviews and advance evidence-based practice in the field.

Analysis of predictive performance and reliability of classifiers for quality assessment of medical evidence revealed important variation by medical area

OBJECTIVES

A major obstacle in deployment of models for automated quality assessment is their reliability. To analyze their calibration and selective classification performance.

STUDY DESIGN AND SETTING

We examine two systems for assessing the quality of medical evidence, EvidenceGRADEr and RobotReviewer, both developed from Cochrane Database of Systematic Reviews (CDSR) to measure strength of bodies of evidence and risk of bias (RoB) of individual studies, respectively. We report their calibration error and Brier scores, present their reliability diagrams, and analyze the risk-coverage trade-off in selective classification.

RESULTS

The models are reasonably well calibrated on most quality criteria (expected calibration error [ECE] 0.04-0.09 for EvidenceGRADEr, 0.03-0.10 for RobotReviewer). However, we discover that both calibration and predictive performance vary significantly by medical area. This has ramifications for the application of such models in practice, as average performance is a poor indicator of group-level performance (e.g., health and safety at work, allergy and intolerance, and public health see much worse performance than cancer, pain, and anesthesia, and Neurology). We explore the reasons behind this disparity.

CONCLUSION

Practitioners adopting automated quality assessment should expect large fluctuations in system reliability and predictive performance depending on the medical area. Prospective indicators of such behavior should be further researched.

The Past, Present, and Future of Psychotherapy Manuals: Protocol for a Scoping Review

BACKGROUND

Psychotherapy manuals are critical to the dissemination of psychotherapy treatments. Psychotherapy manuals typically serve several purposes, including, but not limited to, establishing new psychotherapy treatments, training providers, disseminating treatments to those who deliver them, and providing guidelines to deliver treatments with fidelity. Yet, the proliferation of psychotherapy manuals has not been well-understood, and no work has aimed to assess or review the existing landscape of psychotherapy manuals. Little is known about the breadth, scope, and foci of extant psychotherapy manuals.

OBJECTIVE

This scoping review aims to identify and explore the landscape of existing book-based psychotherapy manuals. This review aims to specify the defining characteristics (ie, foci, clinical populations, clinical targets, treatment type, treatment modality, and adaptations) of existing book-based psychotherapy manuals. Further, this review will demonstrate how this information, and psychotherapy manuals more broadly, has changed over time. This project aims to make a novel contribution that will have critical implications for current methods of developing, aggregating, synthesizing, and translating knowledge about psychotherapeutic treatments.

METHODS

This scoping review will review book-based psychotherapy manuals published from 1950 to 2022.This scoping review will be informed by guidance from the Joanna Briggs Institute Scoping Review Methodology Group and prior scoping reviews. Traditional search and application programming interface-based search methods will be used with search terms defined a priori to identify relevant results using 3 large book databases: Google Books, WorldCat, and PsycINFO. This review will leverage machine learning methods to enhance and expedite the screening process. Primary screening of results will be conducted by at least 2 authors. Data will be extracted and double-coded by research assistants using an iteratively defined codebook.

RESULTS

The search process produced 78,600 results, which were then iteratively deduplicated. Following deduplication, 50,583 results remained. The scoping review is expected to identify common elements of psychotherapy manuals, establish how the foci and content of manuals have changed over time, and illustrate coverage and gaps in the landscape of psychotherapy manuals. Results from this scoping review will be critical for future work focused on developing, aggregating, synthesizing, and disseminating knowledge about psychotherapeutic treatments.

CONCLUSIONS

This review will provide knowledge about the vast landscape of psychotherapy manuals that exist. Findings from this study will inform future efforts to develop, aggregate, synthesize, and translate knowledge about psychotherapeutic treatments.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/47708.

Risk of drug use during pregnancy: master protocol for living systematic reviews and meta-analyses performed in the metaPreg project

BACKGROUND

Knowledge about the risks of drugs during pregnancy is continuously evolving due to the frequent publication of a large number of epidemiological studies. Systematic reviews and meta-analyses therefore need to be regularly updated to reflect these advances. To improve dissemination of this updated information, we developed an initiative of real-time full-scale living meta-analyses relying on an open online dissemination platform ( www.metapreg.org ).

METHOD

All living meta-analyses performed in this project will be conducted in accordance with this master protocol after adaptation of the search strategy. A systematic literature search of PubMed and Embase will be performed. All analytical studies (e.g., cohort, case-control, randomized studies) reporting original empirical findings on the association between in utero exposure to drugs and adverse pregnancy outcomes will be included. Study screening and data extraction will be performed in a semi-automation way supervised by a biocurator. A risk of bias will be assessed using the ROBINS-I tools. All clinically relevant pregnancy adverse outcomes (malformations, stillbirths, neuro-developmental disorders, pre-eclampsia, etc.) available in the included studies will be pooled through random-effects meta-analysis. Heterogeneity will be evaluated by I² statistics.

DISCUSSION

Our living systematic reviews and subsequent updates will inform the medical, regulatory, and health policy communities as the news results evolve to guide decisions on the proper use of drugs during the pregnancy.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework (OSF) registries.

[Concepts to support dynamic updating processes of guidelines and their practical implementation-Systematic literature review]

BACKGROUND

Clinical practice guidelines (CPG), which are crucial for evidence-based healthcare, should be kept up to date, especially on topics where emerging evidence could modify one of the recommendations with implications for the healthcare service; however, an updating process, which is practicable for guideline developers as well as users represents a challenge.

OBJECTIVE

This article provides an overview of the currently discussed methodological approaches for the dynamic updating of guidelines and systematic reviews.

MATERIAL AND METHODS

As part of a scoping review, a literature search was conducted in the databases MEDLINE, EMBASE (via Ovid), Scopus, Epistemonikos, medRxiv, as well as in study and guideline registers. Concepts on the dynamic updating of guidelines and systematic reviews or dynamically updated guidelines and systematic reviews or their protocols published in English or German were included.

RESULTS

The publications included most frequently described the following main processes that must be adapted in dynamic updating processes 1) the establishment of continuously working guideline groups, 2) networking between guidelines, 3) the formulation and application of prioritization criteria, 4) the adaptation of the systematic literature search and 5) the use of software tools for increased efficiency and digitalization of the guidelines.

CONCLUSION

The transformation to living guidelines requires a change in the needs for temporal, personnel and structural resources. The digitalization of guidelines and the use of software to increase efficiency are necessary instruments, but alone do not guarantee the realization of living guidelines. A process in which dissemination and implementation must also be integrated is necessary. Standardized best practice recommendations on the updating process are still lacking.

Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

BACKGROUND

Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study.

METHODS

The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD).

RESULTS

The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values.

CONCLUSIONS

Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

Assessing the Integrity of Clinical Trials Included in Evidence Syntheses

Evidence syntheses of randomized clinical trials (RCTs) offer the highest level of scientific evidence for informing clinical practice and policy. The value of evidence synthesis itself depends on the trustworthiness of the included RCTs. The rising number of retractions and expressions of concern about the authenticity of RCTs has raised awareness about the existence of problematic studies, sometimes called "zombie" trials. Research integrity, i.e., adherence to ethical and professional standards, is a multi-dimensional concept that is incompletely evaluated for the RCTs included in current evidence syntheses. Systematic reviewers tend to rely on the editorial and peer-review system established by journals as custodians of integrity of the RCTs they synthesize. It is now well established that falsified and fabricated RCTs are slipping through. Thus, RCT integrity assessment becomes a necessary step in systematic reviews going forward, in particular because RCTs with data-related integrity concerns remain available for use in evidence syntheses. There is a need for validated tools for systematic reviewers to proactively deploy in the assessment of integrity deviations without having to wait for RCTs to be retracted by journals or expressions of concern issued. This article analyzes the issues and challenges in conducting evidence syntheses where the literature contains RCTs with possible integrity deficits. The way forward in the form of formal RCT integrity assessments in systematic reviews is proposed, and implications of this new initiative are discussed. Future directions include emphasizing ethical and professional standards, providing tailored integrity-specific training, and creating systems to promote research integrity, as improvements in RCT integrity will benefit evidence syntheses.

Factors associated with successful publication for systematic review protocol registration: an analysis of 397 registered protocols

BACKGROUND

Meta-analyses are on top of the evidence-based medicine pyramid, yet many of them are not completed after they are begun. Many factors impacting the publication of meta-analysis works have been discussed, and their association with publication likelihood has been investigated. These factors include the type of systematic review, journal metrics, h-index of the corresponding author, country of the corresponding author, funding sources, and duration of publication. In our current review, we aim to investigate these various factors and their impact on the likelihood of publication. A comprehensive review of 397 registered protocols retrieved from five databases was performed to investigate the different factors that might affect the likelihood of publication. These factors include the type of systematic review, journal metrics, h-index of the corresponding author, country of the corresponding author, funding sources, and duration of publication.

RESULTS

We found that corresponding authors in developed countries and English-speaking countries had higher likelihoods of publication: 206/320 (p = 0.018) and 158/236 (p = 0.006), respectively. Factors affecting publications are the countries of corresponding author (p = 0.033), whether they are from developed countries (OR: 1.9, 95% CI: 1.2-3.1, p = 0.016), from English-speaking countries (OR: 1.8, 95% CI: 1.2-2.7, p = 0.005), update status of the protocol (OR: 1.6, 95% CI: 1.0-2.6, p = 0.033), and external funding (OR: 1.7, 95% CI: 1.1-2.7, p = 0.025). Multivariable regression retains three variables as significant predictors for the publication of a systematic review: whether it is the corresponding author from developed countries (p = 0.013), update status of the protocol (p = 0.014), and external funding (p = 0.047).

CONCLUSION

Being on top of the evidence hierarchy, systematic review and meta-analysis are the keys to informed clinical decision-making. Updating protocol status and external funding are significant influences on their publications. More attentions should be paid to the methodological quality of this type of publication.