Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis

Reimagining How We Synthesize Information to Impact Clinical Care, Policy, and Research Priorities in Real Time: Examples and Lessons Learned from COVID-19

Real-time clinical care, policy, and research decisions need real-time evidence synthesis. However, as we found during the COVID-19 pandemic, it is challenging to rapidly address key clinical and policy questions through rigorous, relevant, and usable evidence. Our objective is to present three exemplar cases of rapid evidence synthesis products from the Veterans Healthcare Administration Evidence Synthesis Program (ESP) and, in the context of these examples, outline ESP products, challenges, and lessons learned. We faced challenges in (1) balancing scientific rigor with the speed in which evidence synthesis was needed, (2) sorting through rapidly evolving large bodies of evidence, and (3) assessing the impact of evidence synthesis products on clinical care, policy, and research. We found solutions in (1) engaging stakeholders early, (2) utilizing artificial intelligence capabilities, (3) building infrastructure to establish living reviews, and (4) planning for dissemination to maximize impact.

A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making

BACKGROUND

The COVID-19 pandemic has had an unprecedented impact in the global research production and has also increased research waste. Living evidence syntheses (LESs) seek to regularly update a body of evidence addressing a specific question. During the COVID-19 pandemic, the production and dissemination of LESs emerged as a cornerstone of the evidence infrastructure. This critical interpretive synthesis answers the questions: What constitutes an LES to support decision-making?; when should one be produced, updated, and discontinued?; and how should one be disseminated?

METHODS

Searches included the Cochrane Library, EMBASE (Ovid), Health Systems Evidence, MEDLINE (Ovid), PubMed, and Web of Science up to 23 April 2024 and included articles that provide any insights on addressing the compass questions on LESs. Articles were selected and appraised, and their insights extracted. An interpretive and iterative coding process was used to identify relevant thematic categories and create a conceptual framework.

RESULTS

Among the 16,630 non-duplicate records identified, 208 publications proved eligible. Most were non-empirical articles, followed by actual LESs. Approximately one in three articles were published in response to the COVID-19 pandemic. The conceptual framework addresses six thematic categories: (1) what is an LES; (2) what methodological approaches facilitate LESs production; (3) when to produce an LES; (4) when to update an LES; (5) how to make available the findings of an LES; and (6) when to discontinue LES updates.

CONCLUSION

LESs can play a critical role in reducing research waste and ensuring alignment with advisory and decision-making processes. This critical interpretive synthesis provides relevant insights on how to better organize the global evidence architecture to support their production.

TRIAL REGISTRATION

PROSPERO registration: CRD42021241875.

An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review

BACKGROUND

Systematic reviews (SRs) are time-consuming and labor-intensive to perform. With the growing number of scientific publications, the SR development process becomes even more laborious. This is problematic because timely SR evidence is essential for decision-making in evidence-based healthcare and policymaking. Numerous methods and tools that accelerate SR development have recently emerged. To date, no scoping review has been conducted to provide a comprehensive summary of methods and ready-to-use tools to improve efficiency in SR production.

OBJECTIVE

To present an overview of primary studies that evaluated the use of ready-to-use applications of tools or review methods to improve efficiency in the review process.

METHODS

We conducted a scoping review. An information specialist performed a systematic literature search in four databases, supplemented with citation-based and grey literature searching. We included studies reporting the performance of methods and ready-to-use tools for improving efficiency when producing or updating a SR in the health field. We performed dual, independent title and abstract screening, full-text selection, and data extraction. The results were analyzed descriptively and presented narratively.

RESULTS

We included 103 studies: 51 studies reported on methods, 54 studies on tools, and 2 studies reported on both methods and tools to make SR production more efficient. A total of 72 studies evaluated the validity (n = 69) or usability (n = 3) of one method (n = 33) or tool (n = 39), and 31 studies performed comparative analyses of different methods (n = 15) or tools (n = 16). 20 studies conducted prospective evaluations in real-time workflows. Most studies evaluated methods or tools that aimed at screening titles and abstracts (n = 42) and literature searching (n = 24), while for other steps of the SR process, only a few studies were found. Regarding the outcomes included, most studies reported on validity outcomes (n = 84), while outcomes such as impact on results (n = 23), time-saving (n = 24), usability (n = 13), and cost-saving (n = 3) were less often evaluated.

CONCLUSION

For title and abstract screening and literature searching, various evaluated methods and tools are available that aim at improving the efficiency of SR production. However, only few studies have addressed the influence of these methods and tools in real-world workflows. Few studies exist that evaluate methods or tools supporting the remaining tasks. Additionally, while validity outcomes are frequently reported, there is a lack of evaluation regarding other outcomes.

Machine learning to optimize literature screening in medical guideline development

OBJECTIVES

In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels.

DESIGN

We used a mixed-methods design. Two independent clinicians' manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue.

MAIN OUTCOME MEASURES

Inter-rater reliability was assessed using Cohen's Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers.

RESULTS

Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between - 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively.

CONCLUSION

While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

BACKGROUND

The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.

METHODS

In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.

RESULTS

From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.

CONCLUSIONS

Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Cardiovascular morbidity and mortality in lean vs. non-lean MASLD: A comprehensive meta-analysis

BACKGROUND

Lean metabolic dysfunction-associated steatotic liver disease (MASLD), characterized by a BMI < 25 kg/m² (or < 23 kg/m² in Asians), presents a challenging prognosis compared to non-lean MASLD. This study examines cardiovascular outcomes in both lean and non-lean MASLD cohorts.

METHODS

In this meta-analysis, pooled odds ratios (ORs) within 95 % confidence intervals (CIs) were calculated for primary outcomes (cardiovascular mortality and major adverse cardiovascular events [MACE]) and secondary outcomes (cardiovascular disease [CVD], all-cause mortality, hypertension, and dyslipidemia). Studies comparing lean and non-lean MASLD within the same cohorts were analyzed, prioritizing those with larger sample sizes or recent publication dates.

RESULTS

Twenty-one studies were identified, encompassing lean MASLD patients (n = 7153; mean age 52.9 ± 7.4; 56 % male) and non-lean MASLD patients (n = 23,514; mean age 53.2 ± 6.8; 63 % male). Lean MASLD exhibited a 50 % increase in cardiovascular mortality odds compared to non-lean MASLD (OR: 1.5, 95 % CI 1.2-1.8; p < 0.0001). MACE odds were 10 % lower in lean MASLD (OR: 0.9, 95 % CI 0.7-1.2; p = 0.7), while CVD odds were 40 % lower (p = 0.01). All-cause mortality showed a 40 % higher odds in lean MASLD versus non-lean MASLD (p = 0.06). Lean MASLD had 30 % lower odds for both hypertension (p = 0.01) and dyslipidemia (p = 0.02) compared to non-lean MASLD.

CONCLUSION

Despite a favorable cardiometabolic profile and comparable MACE rates, lean individuals with MASLD face elevated cardiovascular mortality risk.

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.

Barriers in utilization and provisioning of obstetric care services (OCS) in India: a mixed-methods systematic review

BACKGROUND

Despite the reduction in the maternal mortality ratio, barriers in obstetric care services (OCS) remain a significant risk factor for adverse maternal and perinatal outcomes in India. This review covers the 'continuum of care' (ANC, child delivery, and PNC services) and identifies multiple barriers in provisioning as well as utilization of OCS in India. We conducted a systematic review to understand the barriers using a mixed-methods approach.

METHODS

PubMed, Scopus, Web of Science, Google Scholar, and Science Direct databases were searched from 1 January 2000 to 30 June 2022. The methodological quality of the included studies was assessed using appropriate tools. After a full-text review of 164 studies, total of 56 studies (33 quantitative, 18 qualitative, and 5 mixed-methods studies) were finally included in the review. All the barriers were classified into five major themes: (i) individual and interpersonal barriers, (ii) social and cultural barriers, (iii) structural barriers, (iv) logistical barriers, and (v) organizational barriers. A thematic synthesis approach was used to present the findings of the included studies.

RESULTS

Lack of knowledge and awareness and less family support in availing the required OCS were key individual and interpersonal barriers. Negative social and cultural practices, such as belief in traditional herbs/healers, dietary restrictions, and discarding colostrum were frequently reported barriers, especially in rural settings. Poor economic status and high health service costs were the most often cited barriers to low institutional delivery and delayed ANC services. Long distances to health facilities and poor road conditions were the most frequently reported logistical barriers. On the provisioning side, poor quality of treatment, shortage of drugs and equipment, and non-cooperative attitude of health professionals were the most significant barriers.

CONCLUSION

This review identified several important barriers ranging from individual and cultural to structural, logistical, and organizational, which are prevalent in India. To mitigate the barriers, the governments need to develop strategies at the individual and organizational levels. Innovative interventions and program implementation at the community and village levels could also be contributory steps towards improving OCS utilization in India.

Oral Pre-exposure Prophylaxis Uptake and Acceptability Among Men Who Have Sex With Men: A Scoping Review of the Literature

Despite the global effort to end the human immunodeficiency virus (HIV) and acquired immunodeficiency syndrome (AIDS) epidemic as a global threat by 2030, the rate of new HIV infections worldwide remains unacceptably high among men who have sex with men, hence the need to use pre-exposure prophylaxis (PrEP) to reduce the risk of HIV infection. This population has an increased risk of getting HIV; thus, it is imperative to assess the uptake and acceptability of PrEP. This study investigated the willingness, availability, accessibility, and knowledge and identified barriers and facilitators to using PrEP among this population. A scoping literature review search was conducted on research papers published in English and focused on men who have sex with men and their use of PrEP. These were independently screened and coded. Of about 1,202 literature sources, 55 were included in the study. Findings reported that the uptake and acceptability of PrEP were influenced by knowledge and perception of being high-risk. Generally, PrEP uptake and understanding were high in North America, Latin America, and Europe and low in Asia and Africa. Low uptake and acceptability have been largely attributed to fear of side effects, societal stigma, cost, and perception of not being at risk. Noted facilitators to PrEP use include education, availability of free pills, support groups, and friendly health care facilities. Health intervention programs to increase the use of PrEP must be backed by appropriate legal and regulatory frameworks.