Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey

Automation tools to support undertaking scoping reviews

OBJECTIVE

This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews.

STUDY DESIGN AND SETTING

The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group.

RESULTS

This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability.

CONCLUSION

This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.

An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review

BACKGROUND

Systematic reviews (SRs) are time-consuming and labor-intensive to perform. With the growing number of scientific publications, the SR development process becomes even more laborious. This is problematic because timely SR evidence is essential for decision-making in evidence-based healthcare and policymaking. Numerous methods and tools that accelerate SR development have recently emerged. To date, no scoping review has been conducted to provide a comprehensive summary of methods and ready-to-use tools to improve efficiency in SR production.

OBJECTIVE

To present an overview of primary studies that evaluated the use of ready-to-use applications of tools or review methods to improve efficiency in the review process.

METHODS

We conducted a scoping review. An information specialist performed a systematic literature search in four databases, supplemented with citation-based and grey literature searching. We included studies reporting the performance of methods and ready-to-use tools for improving efficiency when producing or updating a SR in the health field. We performed dual, independent title and abstract screening, full-text selection, and data extraction. The results were analyzed descriptively and presented narratively.

RESULTS

We included 103 studies: 51 studies reported on methods, 54 studies on tools, and 2 studies reported on both methods and tools to make SR production more efficient. A total of 72 studies evaluated the validity (n = 69) or usability (n = 3) of one method (n = 33) or tool (n = 39), and 31 studies performed comparative analyses of different methods (n = 15) or tools (n = 16). 20 studies conducted prospective evaluations in real-time workflows. Most studies evaluated methods or tools that aimed at screening titles and abstracts (n = 42) and literature searching (n = 24), while for other steps of the SR process, only a few studies were found. Regarding the outcomes included, most studies reported on validity outcomes (n = 84), while outcomes such as impact on results (n = 23), time-saving (n = 24), usability (n = 13), and cost-saving (n = 3) were less often evaluated.

CONCLUSION

For title and abstract screening and literature searching, various evaluated methods and tools are available that aim at improving the efficiency of SR production. However, only few studies have addressed the influence of these methods and tools in real-world workflows. Few studies exist that evaluate methods or tools supporting the remaining tasks. Additionally, while validity outcomes are frequently reported, there is a lack of evaluation regarding other outcomes.

Methodological components, structure and quality assessment tools for evidence summaries: a scoping review

OBJECTIVE

The objective of this review was to identify and map the available information related to the definition, structure, and core methodological components of evidence summaries, as well as to identify any indicators of quality.

INTRODUCTION

Evidence summaries offer a practical solution to overcoming some of the barriers present in evidence-based health care, such as lack of access to evidence at the point of care, and the knowledge and expertise to evaluate the quality and translate the evidence into clinical decision-making. However, lack of transparency in reporting and inconsistencies in the methodology of evidence summary development have previously been cited and pose problems for end-users (eg, clinicians, policymakers).

INCLUSION CRITERIA

Any English-language resource that described the methodological development or appraisal of an evidence summary was included.

METHODS

PubMed, Embase, and CINAHL (EBSCOhost) were systematically searched in November 2019, with no limits on the search. The search was updated in June 2021 and January 2023. Gray literature searches and pearling of references of included sources were also conducted at the same time as the database searches. All resources (ie, articles, papers, books, dissertations, reports, and websites) were eligible for inclusion in the review if they evaluated or described the development or appraisal of an evidence summary methodology within a point-of-care context and were published in English. Literature reviews (eg, systematic reviews, rapid reviews), including summaries of evidence on interventions or health care activities that either measure effects, a phenomena of interest, or where the objective was the development, description or evaluation of methods without a clear point-of-care target, were excluded from the review.

RESULTS

A total of 76 resources (n=56 articles from databases and n=20 reports from gray literature sources) were included in the review. The most common type/name included critically appraised topic (n=18) and evidence summary (n=17). A total of 25 resources provided a definition of an evidence summary: commonalities included a clinical question; a structured, systematic literature search; a description of literature selection; and appraisal of evidence. Of these 25, 16 included descriptors such as brief, concise, rapid, short, succinct and snapshot. The reported methodological components closely reflected the definition results, with the most reported methodological components being a systematic, multi-database search, and critical appraisal. Evidence summary examples were mostly presented as narrative summaries and usually included a reference list, background or clinical context, and recommendations or implications for practice or policy. Four quality assessment tools and a systematic review of tools were included.

CONCLUSIONS

The findings of this study highlight the wide variability in the definition, language, methodological components and structure used for point-of-care resources that met our definition of an evidence summary. This scoping review is one of the first steps aimed at improving the credibility and transparency of evidence summaries in evidence-based health care, with further research required to standardize the definitions and methodologies associated with point-of-care resources and accepted tools for quality assessment.

SUPPLEMENTAL DIGITAL CONTENT

A Chinese-language version of the abstract of this review is available at http://links.lww.com/SRX/A59, studies ineligible following full-text review http://links.lww.com/SRX/A60.

Rapid review methods series: Guidance on the use of supportive software

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RRs) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. This paper guides how to use supportive software for RRs.We strongly encourage the use of supportive software throughout RR production. Specifically, we recommend (1) using collaborative online platforms that enable working in parallel, allow for real-time project management and centralise review details; (2) using automation software to support, but not entirely replace a human reviewer and human judgement and (3) being transparent in reporting the methodology and potential risk for bias due to the use of supportive software.

Machine learning to optimize literature screening in medical guideline development

OBJECTIVES

In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels.

DESIGN

We used a mixed-methods design. Two independent clinicians' manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue.

MAIN OUTCOME MEASURES

Inter-rater reliability was assessed using Cohen's Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers.

RESULTS

Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between - 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively.

CONCLUSION

While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

BACKGROUND

The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.

METHODS

In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.

RESULTS

From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.

CONCLUSIONS

Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Investigating the Cost-Effectiveness of Telemonitoring Patients With Cardiac Implantable Electronic Devices: Systematic Review

BACKGROUND

Telemonitoring patients with cardiac implantable electronic devices (CIEDs) can improve their care management. However, the results of cost-effectiveness studies are heterogeneous. Therefore, it is still a matter of debate whether telemonitoring is worth the investment.

OBJECTIVE

This systematic review aims to investigate the cost-effectiveness of telemonitoring patients with CIEDs, focusing on its key drivers, and the impact of the varying perspectives.

METHODS

A systematic review was performed in PubMed, Web of Science, Embase, and EconLit. The search was completed on July 7, 2022. Studies were included if they fulfilled the following criteria: patients had a CIED, comparison with standard care, and inclusion of health economic evaluations (eg, cost-effectiveness analyses and cost-utility analyses). Only complete and peer-reviewed studies were included, and no year limits were applied. The exclusion criteria included studies with partial economic evaluations, systematic reviews or reports, and studies without standard care as a control group. Besides general study characteristics, the following outcome measures were extracted: impact on total cost or income, cost or income drivers, cost or income drivers per patient, cost or income drivers as a percentage of the total cost impact, incremental cost-effectiveness ratios, or cost-utility ratios. Quality was assessed using the Consensus Health Economic Criteria checklist.

RESULTS

Overall, 15 cost-effectiveness analyses were included. All studies were performed in Western countries, mainly Europe, and had primarily a male participant population. Of the 15 studies, 3 (20%) calculated the incremental cost-effectiveness ratio, 1 (7%) the cost-utility ratio, and 11 (73%) the health and cost impact of telemonitoring. In total, 73% (11/15) of the studies indicated that telemonitoring of patients with implantable cardioverter-defibrillators (ICDs) and cardiac resynchronization therapy ICDs was cost-effective and cost-saving, both from a health care and patient perspective. Cost-effectiveness results for telemonitoring of patients with pacemakers were inconclusive. The key drivers for cost reduction from a health care perspective were hospitalizations and scheduled in-office visits. Hospitalization costs were reduced by up to US $912 per patient per year. Scheduled in-office visits included up to 61% of the total cost reduction. Key drivers for cost reduction from a patient perspective were loss of income, cost for scheduled in-office visits and transport. Finally, of the 15 studies, 8 (52%) reported improved quality of life, with statistically significance in only 1 (13%) study (P=.03).

CONCLUSIONS

From a health care and patient perspective, telemonitoring of patients with an ICD or a cardiac resynchronization therapy ICD is a cost-effective and cost-saving alternative to standard care. Inconclusive results were found for patients with pacemakers. However, telemonitoring can lead to a decrease in providers' income, mainly due to a lack of reimbursement. Introducing appropriate reimbursement could make telemonitoring sustainable for providers while still being cost-effective from a health care payer perspective.

TRIAL REGISTRATION

PROSPERO CRD42022322334; https://tinyurl.com/puunapdr.

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research

Background

Identifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity.

Methods

We conducted an eight-arm cost-effectiveness analysis to assess the costs, recall and precision of semi-automated workflows, incorporating MAG with adjunctive machine learning, for continually updating our living map. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Our systematic review software, EPPI-Reviewer, was adapted to incorporate MAG and associated machine learning workflows, and also used to collect data on recall, precision, and manual screening workload.

Results

The semi-automated MAG-enabled workflow dominated conventional workflows in both the base case and sensitivity analyses. At one month our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified 469 additional, eligible articles for inclusion in our living map, and cost £3,179 GBP per week less, compared with conventional methods relying on Boolean searches of Medline and Embase.

Conclusions

We were able to increase recall and coverage of a large living map, whilst reducing its production costs. This finding is likely to be transferrable to OpenAlex, MAG's successor database platform.

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research

BACKGROUND

Identifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity.

METHODS

We conducted an eight-arm cost-effectiveness analysis to assess the costs, recall and precision of semi-automated workflows, incorporating MAG with adjunctive machine learning, for continually updating our living map. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Our systematic review software, EPPI-Reviewer, was adapted to incorporate MAG and associated machine learning workflows, and also used to collect data on recall, precision, and manual screening workload.

RESULTS

The semi-automated MAG-enabled workflow dominated conventional workflows in both the base case and sensitivity analyses. At one month our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified 469 additional, eligible articles for inclusion in our living map, and cost £3,179 GBP per week less, compared with conventional methods relying on Boolean searches of Medline and Embase.

CONCLUSIONS

We were able to increase recall and coverage of a large living map, whilst reducing its production costs. This finding is likely to be transferrable to OpenAlex, MAG's successor database platform.

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group (RRMG). Rapid reviews (RRs) use modified systematic review (SR) methods to accelerate the review process while maintaining systematic, transparent and reproducible methods to ensure integrity. This paper addresses considerations around the acceleration of study selection, data extraction and risk of bias (RoB) assessment in RRs. If a RR is being undertaken, review teams should consider using one or more of the following methodological shortcuts: screen a proportion (eg, 20%) of records dually at the title/abstract level until sufficient reviewer agreement is achieved, then proceed with single-reviewer screening; use the same approach for full-text screening; conduct single-data extraction only on the most relevant data points and conduct single-RoB assessment on the most important outcomes, with a second person verifying the data extraction and RoB assessment for completeness and correctness. Where available, extract data and RoB assessments from an existing SR that meets the eligibility criteria.

Overcoming the challenges of using automated technologies for public health evidence synthesis

Many organisations struggle to keep pace with public health evidence due to the volume of published literature and length of time it takes to conduct literature reviews. New technologies that help automate parts of the evidence synthesis process can help conduct reviews more quickly and efficiently to better provide up-to-date evidence for public health decision making. To date, automated approaches have seldom been used in public health due to significant barriers to their adoption. In this Perspective, we reflect on the findings of a study exploring experiences of adopting automated technologies to conduct evidence reviews within the public health sector. The study, funded by the European Centre for Disease Prevention and Control, consisted of a literature review and qualitative data collection from public health organisations and researchers in the field. We specifically focus on outlining the challenges associated with the adoption of automated approaches and potential solutions and actions that can be taken to mitigate these. We explore these in relation to actions that can be taken by tool developers (e.g. improving tool performance and transparency), public health organisations (e.g. developing staff skills, encouraging collaboration) and funding bodies/the wider research system (e.g. researchers, funding bodies, academic publishers and scholarly journals).

A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses

Living reviews are an increasingly popular research paradigm. The purpose of a 'living' approach is to allow rapid collation, appraisal and synthesis of evolving evidence on an important research topic, enabling timely influence on patient care and public health policy. However, living reviews are time- and resource-intensive. The accumulation of new evidence and the possibility of developments within the review's research topic can introduce unique challenges into the living review workflow. To investigate the potential of software tools to support living systematic or rapid reviews, we present a narrative review informed by an examination of tools contained on the Systematic Review Toolbox website. We identified 11 tools with relevant functionalities and discuss the important features of these tools with respect to different steps of the living review workflow. Four tools (NestedKnowledge, SWIFT-ActiveScreener, DistillerSR, EPPI-Reviewer) covered multiple, successive steps of the review process, and the remaining tools addressed specific components of the workflow, including scoping and protocol formulation, reference retrieval, automated data extraction, write-up and dissemination of data. We identify several ways in which living reviews can be made more efficient and practical. Most of these focus on general workflow management, or automation through artificial intelligence and machine-learning, in the screening process. More sophisticated uses of automation mostly target living rapid reviews to increase the speed of production or evidence maps to broaden the scope of the map. We use a case study to highlight some of the barriers and challenges to incorporating tools into the living review workflow and processes. These include increased workload, the need for organisation, ensuring timely dissemination and challenges related to the development of bespoke automation tools to facilitate the review process. We describe how current end-user tools address these challenges, and which knowledge gaps remain that could be addressed by future tool development. Dedicated web presences for automatic dissemination of in-progress evidence updates, rather than solely relying on peer-reviewed journal publications, help to make the effort of a living evidence synthesis worthwhile. Despite offering basic living review functionalities, existing end-user tools could be further developed to be interoperable with other tools to support multiple workflow steps seamlessly, to address broader automatic evidence retrieval from a larger variety of sources, and to improve dissemination of evidence between review updates.

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey

OBJECTIVE

Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.

STUDY DESIGN AND SETTING

We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.

RESULTS

162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).

CONCLUSION

Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review

OBJECTIVE

Evidence-based medicine (EBM) is a decision-making process based on the conscious and judicious use of the best available scientific evidence. However, the exponential increase in the amount of information currently available likely exceeds the capacity of human-only analysis. In this context, artificial intelligence (AI) and its branches such as machine learning (ML) can be used to facilitate human efforts in analyzing the literature to foster EBM. The present scoping review aimed to examine the use of AI in the automation of biomedical literature survey and analysis with a view to establishing the state-of-the-art and identifying knowledge gaps.

MATERIALS AND METHODS

Comprehensive searches of the main databases were performed for articles published up to June 2022 and studies were selected according to inclusion and exclusion criteria. Data were extracted from the included articles and the findings categorized.

RESULTS

The total number of records retrieved from the databases was 12,145, of which 273 were included in the review. Classification of the studies according to the use of AI in evaluating the biomedical literature revealed three main application groups, namely assembly of scientific evidence (n=127; 47%), mining the biomedical literature (n=112; 41%) and quality analysis (n=34; 12%). Most studies addressed the preparation of systematic reviews, while articles focusing on the development of guidelines and evidence synthesis were the least frequent. The biggest knowledge gap was identified within the quality analysis group, particularly regarding methods and tools that assess the strength of recommendation and consistency of evidence.

CONCLUSION

Our review shows that, despite significant progress in the automation of biomedical literature surveys and analyses in recent years, intense research is needed to fill knowledge gaps on more difficult aspects of ML, deep learning and natural language processing, and to consolidate the use of automation by end-users (biomedical researchers and healthcare professionals).

In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature

We aimed to evaluate the performance of supervised machine learning algorithms in predicting articles relevant for full-text review in a systematic review. Overall, 16,430 manually screened titles/abstracts, including 861 references identified relevant for full-text review were used for the analysis. Of these, 40% (n = 6573) were sub-divided for training (70%) and testing (30%) the algorithms. The remaining 60% (n = 9857) were used as a validation set. We evaluated down- and up-sampling methods and compared unigram, bigram, and singular value decomposition (SVD) approaches. For each approach, Naïve Bayes, Support Vector Machines (SVM), regularized logistic regressions, neural networks, random forest, Logit boost, and XGBoost were implemented using simple term frequency or Tf-Idf feature representations. Performance was evaluated using sensitivity, specificity, precision and area under the Curve. We combined predictions of the best-performing algorithms (Youden Index ≥0.3 with sensitivity/specificity≥70/60%). In a down-sample unigram approach, Naïve Bayes, SVM/quanteda text models with Tf-Idf, and linear SVM e1071 package with Tf-Idf achieved >90% sensitivity at specificity >65%. Combining the predictions of the 10 best-performing algorithms improved the performance to reach 95% sensitivity and 64% specificity in the validation set. Crude screening burden was reduced by 61% (5979) (adjusted: 80.3%) with 5% (27) false negativity rate. All the other approaches yielded relatively poorer performances. The down-sampling unigram approach achieved good performance in our data. Combining the predictions of algorithms improved sensitivity while screening burden was reduced by almost two-third. Implementing machine learning approaches in title/abstract screening should be investigated further toward refining these tools and automating their implementation.

Systematic Review Support software system: web-based solution for managing duplicates and screening eligible studies

OBJECTIVE

To describe the main functions of the "Systematic Review Support" web-based system for removing duplicate articles and aiding eligibility analysis during the process of conducting systematic review studies.

METHODS

The system was developed based on the incremental build model using the Agile methodology. The software is proprietary source code and was published on a proprietary platform. The architecture of the production environment allows the infrastructure used to increase or decrease according to demand. The system functions are presented with insertion of screenshots of the interfaces of the version for personal computers during the simulation of a systematic review.

RESULTS

After importing the files containing the abstracts retrieved from the Pubmed, Embase, and Web of Science databases, the system identifies and removes duplicates for later reading and analysis of title and abstract, a stage which can be performed by one or more reviewers independently. After unblinding of reviewers, the decisions on the eligibility of the studies are compared automatically to help the researchers reach a consensus on any disagreements. Results can be filtered and a PDF produced containing the eligible studies.

CONCLUSION

Version 1.0 of the system is available on the web (sysrev.azurewebsites.net) to assist researchers in the initial stages of systematic reviews.

Artificial intelligence in COVID-19 evidence syntheses was underutilized, but impactful: a methodological study

OBJECTIVES

A rapidly developing scenario like a pandemic requires the prompt production of high-quality systematic reviews, which can be automated using artificial intelligence (AI) techniques. We evaluated the application of AI tools in COVID-19 evidence syntheses.

STUDY DESIGN

After prospective registration of the review protocol, we automated the download of all open-access COVID-19 systematic reviews in the COVID-19 Living Overview of Evidence database, indexed them for AI-related keywords, and located those that used AI tools. We compared their journals' JCR Impact Factor, citations per month, screening workloads, completion times (from pre-registration to preprint or submission to a journal) and AMSTAR-2 methodology assessments (maximum score 13 points) with a set of publication date matched control reviews without AI.

RESULTS

Of the 3,999 COVID-19 reviews, 28 (0.7%, 95% CI 0.47-1.03%) made use of AI. On average, compared to controls (n = 64), AI reviews were published in journals with higher Impact Factors (median 8.9 vs. 3.5, P < 0.001), and screened more abstracts per author (302.2 vs. 140.3, P = 0.009) and per included study (189.0 vs. 365.8, P < 0.001) while inspecting less full texts per author (5.3 vs. 14.0, P = 0.005). No differences were found in citation counts (0.5 vs. 0.6, P = 0.600), inspected full texts per included study (3.8 vs. 3.4, P = 0.481), completion times (74.0 vs. 123.0, P = 0.205) or AMSTAR-2 (7.5 vs. 6.3, P = 0.119).

CONCLUSION

AI was an underutilized tool in COVID-19 systematic reviews. Its usage, compared to reviews without AI, was associated with more efficient screening of literature and higher publication impact. There is scope for the application of AI in automating systematic reviews.

Lessons learnt: Undertaking rapid reviews on public health and social measures during a global pandemic

Public health and social measures (PHSM) have been central to the COVID‐19 response. Consequently, there has been much pressure on decision‐makers to make evidence‐informed decisions and on researchers to synthesize the evidence regarding these measures. This article describes our experiences, responses and lessons learnt regarding key challenges when planning and conducting rapid reviews of PHSM during the COVID‐19 pandemic. Stakeholder consultations and scoping reviews to obtain an overview of the evidence inform the scope of reviews that are policy‐relevant and feasible. Multiple complementary reviews serve to examine the benefits and harms of PHSM across different populations and contexts. Conceiving reviews of effectiveness as adaptable living reviews helps to respond to evolving evidence needs and an expanding evidence base. An appropriately skilled review team and good planning, coordination and communication ensures smooth and rigorous processes and efficient use of resources. Scientific rigor, the practical implications of PHSM‐related complexity and likely time savings should be carefully weighed in deciding on methodological shortcuts. Making the best possible use of modeling studies represents a particular challenge, and methods should be carefully chosen, piloted and implemented. Our experience raises questions regarding the nature of rapid reviews and regarding how different types of evidence should be considered in making decisions about PHSM during a global pandemic. We highlight the need for readily available protocols for conducting studies on the effectiveness, unintended consequences and implementation of PHSM in a timely manner, as well as the need for rapid review standards tailored to “rapid” versus “emergency” mode reviewing.

Health literacy education programmes developed for qualified health professionals: a scoping review protocol

Introduction: Health literacy education, for health professionals, has been identified as having the potential to improve patient outcomes and has been recognized as such in policy developments. Health literacy, as a relational concept, encompasses individuals’ skills and how health information is processed in relation to the demands and complexities of the surrounding environment. Focus has been predominantly on the dimension of functional health literacy (reading, writing and numeracy), although increasing emphasis has been placed on interactive and critical domains. Such dimensions often guide the development of health professional education programmes, where the aim is to enhance the patient-practitioner relationship, and ultimately reduce the health literacy burden experienced by patients navigating health services. Currently little is known about qualified health professionals’ education in health literacy and communication skills, and development, implementation or evaluation of such interventions.

Aim: To identify and map current educational interventions to improve health literacy competencies and communication skills of qualified health professionals.

Methods: A scoping review will be conducted drawing on methods and guidance from the Joanna Briggs Institute, and will be reported according to the Preferred Reporting Items for Systematic Review and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist. This study will retrieve literature on health professional education for health literacy and communication skills through a comprehensive search strategy in the following databases: CINAHL; Medline (Ovid); the Cochrane Library; EMBASE; ERIC; UpToDate; PsycINFO. Grey literature will be searched within the references of identified articles; Lenus; ProQuest E-Thesis Portal; RIAN and OpenGrey. A data charting form will be developed with categories including: article details, demographics, intervention details, implementation and evaluation methods.

Conclusion: Little is known about the extent and nature of the current evidence base therefore a scoping review will be conducted, in order to identify programme characteristics in relation to health literacy competencies and communication skills.

Sistema “Apoio à Revisão Sistemática”: solução web para gerenciamento de duplicatas e seleção de artigos elegíveis

RESUMO Objetivo: Descrever as principais funcionalidades do sistema “Apoio à Revisão Sistemática” na identificação e exclusão de artigos duplicados e no auxílio na análise de elegibilidade durante a condução de estudo de revisão sistemática. Métodos: O sistema foi desenvolvido com base em um modelo de processo incremental, utilizando-se metodologia Ágil. É de código fechado e foi publicado em plataforma proprietária. O ambiente de produção onde o sistema foi implantado possui arquitetura que permite que a infraestrutura utilizada aumente ou diminua conforme a demanda. As funcionalidades foram apresentadas com inserção de imagens das interfaces da versão para computadores, simulando uma revisão sistemática. Resultados: Após a importação dos resumos recuperados nas bases de dados PubMed, Embase e Web of Science, o sistema permite a identificação e eliminação de duplicatas para posterior leitura e análise de título e resumo, etapa que pode ser realizada por mais de um revisor de maneira independente. Após a quebra do cegamento entre os revisores, as respostas sobre a elegibilidade dos estudos podem ser comparadas automaticamente para facilitar a resolução de divergências pelos pesquisadores. É possível filtrar os resultados e gerar um arquivo PDF com os estudos elegíveis. Conclusão: A versão 1.0 do sistema “Apoio à Revisão Sistemática” encontra-se disponível na web (sysrev.azurewebsites.net) para auxiliar pesquisadores nas etapas iniciais de um estudo de revisão sistemática.

Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews

BACKGROUND

Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews.

METHODS

We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively.

RESULTS

Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%).

CONCLUSIONS

Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

Health literacy education programmes developed for qualified health professionals: a scoping review protocol

Introduction: Health professional education for health literacy has been identified as having the potential to improve patient outcomes and has been recognized as such in policy developments. Health literacy is an emerging concept encompassing individuals’ skills and how health information is processed in relation to the demands and complexities of the surrounding environment. Focus has been predominantly on the dimension of functional health literacy (reading, writing and numeracy), although increasing emphasis has been placed on interactive and critical domains. Such dimensions can guide the development of health professional education programmes and bridge the gap in the interaction between health professionals and their patients. Currently little is known about qualified health professional’s education for health literacy, its development, implementation or evaluation. Aim: To identify and map current educational interventions to improve health literacy competencies and communication skills of qualified health professionals. Methods: A scoping review will be conducted drawing on methods and guidance from the Joanna Briggs Institute, and will be reported according to the Preferred Reporting Items for Systematic Review and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist. This study will retrieve literature on health professional education for health literacy through a comprehensive search strategy in the following databases: CINAHL; Medline (Ovid); the Cochrane Library; EMBASE; ERIC; UpToDate; PsycINFO and Central Register of Controlled Trials (CENTRAL). Grey literature will be searched within the references of identified articles: Lenus; ProQuest E-Thesis Portal; the HSE health research repository and RIAN. A data charting form will be developed with categories agreed by the research team, including: article details, demographics, intervention details, implementation and evaluation methods. Conclusion: Little is known about the extent and nature of the current evidence base therefore in order to identify programmes and consolidate their demographics and characteristics within health literacy competencies and communication skills, a scoping review is warranted.